TRANSCRIPT 🔗
WEBVTT
00:00:00.000 --> 00:00:05.440
We were in a three-month period of token maxing and leaderboards and now we're god I can't believe I used this word in front of 1500 people.
00:00:05.599 --> 00:00:09.359
But I said the era of token maxing is over and the era of value mogging is here.
00:00:09.679 --> 00:00:11.279
I'm gonna cringe, but I said it anyway.
00:00:11.599 --> 00:00:11.919
Again.
00:00:12.160 --> 00:00:13.599
Recorded again.
00:00:19.760 --> 00:00:22.640
I'm Brendan Fuoty and welcome to the Frontier.
00:00:22.800 --> 00:00:36.640
Today I'm really excited to welcome Jonathan Frankel, who previously created the lottery ticket hypothesis research paper, which is one of the most famous papers in AI, and then subsequently co-founded Mosaic ML, which he sold to Databricks, where he became their chief scientist.
00:00:36.799 --> 00:00:48.479
Jonathan is one of the most unique perspectives I've seen in the industry because he simultaneously is an extraordinary researcher, but is in the weeds with enterprises, understanding all the data required to make their models work.
00:00:48.640 --> 00:00:50.240
So looking forward to the conversation.
00:00:50.560 --> 00:00:52.240
Jonathan, I'm so excited to have you.
00:00:52.399 --> 00:00:53.679
Thank you so much for joining today.
00:00:53.920 --> 00:00:54.560
It's great to be here.
00:00:54.719 --> 00:00:56.000
I'm really glad the timing worked out.
00:00:56.399 --> 00:00:57.359
It should be a lot of fun.
00:00:57.520 --> 00:01:01.200
I figured we can just get started uh talking about your background.
00:01:01.359 --> 00:01:05.920
Uh I mean, obviously the lottery ticket hypothesis has made waves.
00:01:06.319 --> 00:01:07.920
I'm never going to escape that, am I?
00:01:08.319 --> 00:01:12.400
For for a long time since you uh created it during your PhD.
00:01:12.560 --> 00:01:22.239
Maybe you can share a little bit about the lottery ticket hypothesis, and then we can uh go from there to uh what brought you to Databricks and where you are today.
00:01:22.480 --> 00:01:27.519
Yeah, I will say to my team, if you're out there and you happen to see this, someone mentioned lottery ticket hypothesis, take a drink.
00:01:27.760 --> 00:01:32.079
That is the the team drinking game at this point, is you know, anybody mentions lottery tickets, got to take a drink.
00:01:32.159 --> 00:01:33.599
It was interesting work for 2018.
00:01:33.760 --> 00:01:39.280
I think, you know, if you roll back the clock to 2018 when you were like 12, you were not how old were you in 2018?
00:01:39.359 --> 00:01:39.920
I'm kind of terrified.
00:01:40.159 --> 00:01:41.359
I was 15 in 2018.
00:01:42.400 --> 00:01:43.120
Damn it.
00:01:43.680 --> 00:01:44.319
Not far off.
00:01:44.480 --> 00:01:45.599
The gray is starting to show.
00:01:45.840 --> 00:01:52.879
No, in 2018, I think you know, deep learning was early, um, or at least earlier, at least this phase of deep learning was earlier.
00:01:53.040 --> 00:01:53.920
People were excited about it.
00:01:54.079 --> 00:01:55.040
Results were starting to show up.
00:01:55.200 --> 00:02:02.480
We had Image Net results, there's like some cool, you know, AI and music stuff happening at Google, like kind of little glimpses of what we have today.
00:02:02.959 --> 00:02:08.960
But it was still very much a debate about whether we were gonna like what our understanding of deep learning was going to be.
00:02:09.199 --> 00:02:11.840
Were we going to understand this from a nice, clean theoretical model?
00:02:12.000 --> 00:02:14.560
Was there some order to the world that we were going to be able to impose on this?
00:02:14.719 --> 00:02:16.719
Or was the world just going to be messy?
00:02:16.960 --> 00:02:23.520
And as computer scientists, at least my training was you never want to just have an empirical statement that like this algorithm seems to be this fast.
00:02:23.599 --> 00:02:24.240
You want a proof.
00:02:24.319 --> 00:02:25.520
You want a worst case bound.
00:02:25.599 --> 00:02:26.400
We want to know for certain.
00:02:26.560 --> 00:02:27.919
We're a field that comes from math.
00:02:28.080 --> 00:02:33.759
And I think that kind of mathematical strain is still kind of like somewhere deep in the DNA of anyone who's trained in computer science.
00:02:33.919 --> 00:02:39.919
I mean, the lasting part of the lottery ticket work is really to be completely empirical, to just ask, what do we see in real models?
00:02:40.080 --> 00:02:42.000
At that point, real models meant CFAR 10.
00:02:42.080 --> 00:02:46.159
I think it was 200,000 parameters, maybe a million parameters on a good day.
00:02:46.319 --> 00:02:56.240
Um, you know, the data sets were 32 by 32 pixel images of like trains and buses, and we were classifying them, and that was like a hardcore expensive task on my two K80s.
00:02:56.319 --> 00:03:02.400
If you really want to go back in time and like you've probably never even encountered a K80 in your life, um, I now have one on my wall.
00:03:02.960 --> 00:03:09.919
But the idea back then was this debate like, are we gonna have a grand unified theory of everything of deep learning and like have a true understanding of it?
00:03:10.080 --> 00:03:14.400
Or are we going to just have to measure and empirically find our way through by looking at the properties of models?
00:03:14.560 --> 00:03:17.599
And we're here today to talk about evals, like it's very clear which one won.
00:03:17.680 --> 00:03:17.840
Yeah.
00:03:18.080 --> 00:03:19.759
We're not proving that the model behaves well.
00:03:19.840 --> 00:03:20.960
We're measuring its behavior.
00:03:21.199 --> 00:03:23.520
In 2018, that was a really controversial thing to say.
00:03:23.599 --> 00:03:32.960
If you go and look at the lottery ticket hypothesis papers, like iClear reviews, because they're all public, the scores were 9, 9, 5, and the five was there's no proof of this.
00:03:33.280 --> 00:03:37.759
And I remember when I was on the faculty market, this is just
Load full transcript
this is stuck with me forever.
00:03:38.000 --> 00:03:41.039
Someone during an interview asked me, You've done all this empirical work.
00:03:41.199 --> 00:03:42.479
Do you plan to do anything principled?
00:03:42.639 --> 00:03:48.319
What this person meant was principled is synonymous in computer science with theoretical and provable.
00:03:48.479 --> 00:03:51.599
I was really insulted because like empiricism's principled.
00:03:51.759 --> 00:03:54.080
We can do empirical science in a principled way.
00:03:54.319 --> 00:03:56.719
That was the state of the field in 2018.
00:03:56.879 --> 00:04:00.240
So the work itself was about understanding the training dynamics of models.
00:04:00.479 --> 00:04:05.120
And I ended up, you know, having some kind of interesting, curious findings that were popular at the time.
00:04:05.360 --> 00:04:09.520
I don't think they really stood the test of time in terms of being that useful, quite frankly.
00:04:09.680 --> 00:04:10.800
But it was cool at the time.
00:04:10.879 --> 00:04:20.160
And I think really the lasting impact was empiricism is a lens into understanding how to use neural networks that allows us to go further than theory will allow us to go.
00:04:20.399 --> 00:04:24.480
We can ask bigger questions that we just can't prove or describe in a mathematical way.
00:04:24.720 --> 00:04:27.519
And today, I think we all just take that for granted.
00:04:27.759 --> 00:04:30.720
Like, is anyone doing math about deep learning?
00:04:30.879 --> 00:04:35.120
Like any of Mercore's customers doing like proofs to try to understand if their model is good?
00:04:35.360 --> 00:04:37.040
Mercore wouldn't exist if you could prove this.
00:04:37.279 --> 00:04:39.360
Yeah, it all has come down to empirics.
00:04:39.439 --> 00:04:43.439
And that was obviously an incredibly impactful contribution to the field.
00:04:43.519 --> 00:04:53.279
And talk a little bit more about what brought you from your PhD to Mosaic and the early days of that team and why you joined, how you got started, what you worked on.
00:04:53.439 --> 00:04:53.600
Yeah.
00:04:53.680 --> 00:05:00.720
I mean, what brought me to Mosaic was a cold email from my, you know, the person who became my CEO saying, nice paper, want to do a startup on it?
00:05:00.879 --> 00:05:01.920
It's like, who the hell are you?
00:05:02.000 --> 00:05:03.199
And why are you cold emailing me?
00:05:03.279 --> 00:05:03.519
I don't know.
00:05:03.600 --> 00:05:08.959
If you have an MIT email address as a PhD student, you get a lot of weird emails from weird people asking weird things.
00:05:09.040 --> 00:05:11.360
And I was just like, okay, another crazy person.
00:05:11.519 --> 00:05:17.040
And actually, there was this really wonderful friend and lab mate, Jesse, who I happened to be running into that day.
00:05:17.120 --> 00:05:18.079
And I was like, I got this weird email.
00:05:18.240 --> 00:05:19.519
And he's like, You don't know who that is?
00:05:20.079 --> 00:05:21.519
You've
never heard of a startup before?
00:05:21.680 --> 00:05:22.720
Like, you're an idiot.
00:05:22.959 --> 00:05:24.240
Um, you should really talk to this guy.
00:05:24.399 --> 00:05:25.600
I was gonna just ignore it.
00:05:25.759 --> 00:05:27.199
So I'm really glad I opened the email.
00:05:27.360 --> 00:05:29.120
And, you know, that turned into mosaic.
00:05:29.199 --> 00:05:32.160
Um, you know, Naveen Rao reached out and said, Hey, we should do a startup.
00:05:32.240 --> 00:05:33.040
And I was like, that's dumb.
00:05:33.120 --> 00:05:36.560
The lottery ticket paper is not that useful and it's never gonna be valuable in the world.
00:05:36.639 --> 00:05:38.079
And I still think that was right.
00:05:38.160 --> 00:05:54.000
But we evolved that into something about training efficiency and trying to look for empirical ways to this is gonna sound really quaint, but changing the training algorithm to get better quality in the model more efficiently and being okay with that, even though it was a different algorithm, which today we call doing deep learning.
00:05:54.079 --> 00:05:58.800
But back then that was a very controversial thing because you're gonna get something different than you would have gotten otherwise.
00:05:59.040 --> 00:06:00.959
And people were really nervous about that.
00:06:01.199 --> 00:06:06.560
So it was a controversial thing at the time to say, yeah, we'll train your model, but it'll be trained slightly differently than you planned.
00:06:06.720 --> 00:06:09.839
We'll get you to the same quality on the eval, and that's all that matters.
00:06:10.160 --> 00:06:13.759
That was a scary thing for people, which today sounds so hilarious.
00:06:14.000 --> 00:06:22.800
Well, and I definitely feel like in the arc of the industry over the last few years, it felt like initially there was a big push towards custom models and efficiency.
00:06:22.879 --> 00:06:30.639
And then the frontier models improved so quickly that all of the inference really went towards the frontier models, I think, for most of the last 12 months.
00:06:30.800 --> 00:06:40.720
But it feels like as costs are ballooning so dramatically, we will naturally converge more towards more and more of those costs, going towards custom, smaller, more efficient models.
00:06:40.879 --> 00:06:49.040
And a lot of that uh work and the continuation of it that you're doing at Databricks will be really, really important and impactful as that transition accelerates.
00:06:49.199 --> 00:07:00.399
Um, so maybe share more about uh the acquisition and how you all got in touch with the Databricks team, what that was like and what you're now working on and spending your time on at Databricks.
00:07:00.639 --> 00:07:00.800
Yeah.
00:07:00.879 --> 00:07:09.199
I mean, Mosaic went on this crazy journey where we were trying to train custom vision models for a while because who didn't want ResNet 50 as an image classifier?
00:07:09.279 --> 00:07:11.519
That was, you know, such a hot thing back then.
00:07:11.600 --> 00:07:25.199
Um, and who didn't want a segmentation model that was, you know, a custom, like, I don't even remember the name of the model at this point, or an object detection model, like a mascar CNN, like all these things that sound like they're from a history book at this point because they're three whole years old or five years old.
00:07:25.360 --> 00:07:26.480
And we were doing a lot of that.
00:07:26.639 --> 00:07:36.160
And then someone on my team said to me at some point, maybe people are gonna want more than eight GPUs, and they're gonna want to train these things, these large language models, something that's like maybe a billion parameters.
00:07:36.319 --> 00:07:43.360
It's like, Obby, nobody's gonna need more than eight GPUs, and nobody's ever gonna need to train, you know, a model with a billion parameters.
00:07:43.439 --> 00:07:44.720
25 million is already a lot.
00:07:44.800 --> 00:07:46.480
Like you're 25 million.
00:07:46.639 --> 00:07:47.040
Wow.
00:07:47.279 --> 00:07:48.000
Yeah, those.
00:07:48.079 --> 00:07:50.639
I mean, this is again, I've this was 2022.
00:07:50.800 --> 00:07:51.040
Yeah.
00:07:51.199 --> 00:07:52.079
This wasn't that long ago.
00:07:52.160 --> 00:07:52.959
This was four years.
00:07:53.120 --> 00:07:53.439
Yeah.
00:07:53.680 --> 00:07:56.240
And thank God that Obby didn't listen to me.
00:07:56.480 --> 00:08:04.639
Um, and one of my favorite things about running a research team is the people who fail to listen to me do something, get it to the point where there's some value there, and then I'm like, okay, we'll pivot the whole team.
00:08:04.800 --> 00:08:11.839
He went and built out a multi-node training stack on his own, built out the initial version of our what became our LLM pre-training stack on its own.
00:08:12.000 --> 00:08:14.160
And this was in like summer of 22.
00:08:14.240 --> 00:08:18.240
So pre-Chat GPT, he just he felt like something was coming.
00:08:18.560 --> 00:08:23.680
And it ended up working well enough that we pivoted the whole team, and that became our product, custom LLMs.
00:08:23.839 --> 00:08:31.120
Um, and you know, less than a year later, we had trained the first open source reproduction of Llama, which again sounds really quaint.
00:08:31.279 --> 00:08:33.840
7 billion parameter model on 1 trillion tokens.
00:08:34.080 --> 00:08:35.200
Really hard thing to do back then.
00:08:35.279 --> 00:08:36.879
Now an intern project for many people.
00:08:36.960 --> 00:08:45.279
But you know, back then it was a full team ordeal on a really scary, this is gonna be sound like a huge cluster, 256 A100s.
00:08:45.840 --> 00:08:47.440
That was so expensive and so scary.
00:08:47.600 --> 00:08:48.240
Why did we get that?
00:08:48.399 --> 00:08:49.679
Why are we burning all this cash?
00:08:49.840 --> 00:08:54.080
Legend speaks that Naveen Rao and Ali God seemed that at the first Cerebral Valley.
00:08:54.159 --> 00:09:02.159
I have no idea if this is actually true
or if this is just what legend speaks of, because these things tend to accelerate, but they met, they hit it off and became part of Databricks and got to go big.
00:09:02.320 --> 00:09:07.039
Um, it turns out that, you know, when you're a custom training company, you really need data.
00:09:07.120 --> 00:09:09.759
And when you're a data company, you really need custom training.
00:09:09.919 --> 00:09:16.080
And to Databricks' credit, the annual conference for Databricks, which is why I'm even out here, is called the Data and AI Summit.
00:09:16.159 --> 00:09:18.720
And it sounds like a very 2026 pilled name.
00:09:18.879 --> 00:09:22.480
I think the name was changed from the Spark Summit to the Data and AI Summit in like 2018.
00:09:23.120 --> 00:09:28.480
There was, I think the Databricks founders understood at that time that AI is what you do with data.
00:09:28.639 --> 00:09:28.879
Yeah.
00:09:29.120 --> 00:09:31.279
And, you know, AI doesn't work without data.
00:09:31.440 --> 00:09:35.679
Now we call these environments and agents, but the two go together.
00:09:36.000 --> 00:09:49.919
Talk a little bit more about what your data day looks like, what key initiatives that Databricks you're owning, especially within the enterprise, um, as we think through what are the bottlenecks to AI becoming useful and the economy and enterprise deployment.
00:09:50.159 --> 00:09:57.120
Yeah, I feel like running a research team is such a weird job, especially if you do it at a company where you're really thinking about the customers.
00:09:57.279 --> 00:10:03.919
On the one hand, my day job is to like I frame it lots of different ways to the team depending on my mood and what we're trying to do and what I want to sell them on.
00:10:04.080 --> 00:10:06.000
One version is that we're the risk budget for the company.
00:10:06.080 --> 00:10:14.559
If you're the CEO of a company like Databricks or, you know, any reasonably sized company, you need someone who's willing to go out there and just take a lot of risks, do the crazy things.
00:10:14.799 --> 00:10:24.159
And, you know, whenever you look at just your team, you say, I need a portfolio of risks, I need to like do some conservative things so I deliver value and do some high-risk things, but we aren't a portfolio as a team.
00:10:24.240 --> 00:10:27.679
We are literally the high risk budget for a massive company.
00:10:27.919 --> 00:10:33.039
And so if we're not taking risks and trying new things, we're not serving our purpose within the organization.
00:10:33.120 --> 00:10:34.240
And so our job is to fail.
00:10:34.320 --> 00:10:39.519
And that means failing on some research projects, trying to do some reinforcement learning at scale on a task that's too hard and failing at it.
00:10:39.679 --> 00:10:44.960
That means trying to develop some new products that just don't work because the technology is not there yet.
00:10:45.039 --> 00:10:53.200
And then trying again next year and the technology is there, and this has happened a couple of times where we've taken two or three cracks at the same problem over the course of two or three years.
00:10:53.360 --> 00:10:58.159
And like there's a thing this year that started working that someone had been trying, I think since 2022.
00:10:58.320 --> 00:11:04.320
And finally the models got good enough that this project actually started working in a really big way and has been game-changing for the team and the product.
00:11:04.480 --> 00:11:08.879
Well, the other way I look at this is that my job is to spend a lot of time with our customers.
00:11:09.039 --> 00:11:17.039
Just hang out with a lot of enterprises that have AI needs, figure out what problems they're getting stuck on, and build entirely new technology to solve those problems.
00:11:17.120 --> 00:11:26.320
It's not just giving them a product or an interface or a way to do a thing like the plumbing, it is inventing a new way of solving that problem through our expertise in machine learning.
00:11:26.480 --> 00:11:27.200
And that's a lot of fun.
00:11:27.279 --> 00:11:34.879
Like there aren't that many research teams in the world where you can say, I'm a scientist and I spend, you know, a quarter of my time with customers.
00:11:35.039 --> 00:11:38.159
The way I frame it to our team at least is the customers are the evals.
00:11:38.480 --> 00:11:49.440
Like the one true measure of our success in the world is is there somebody at one of our customers who actually adopted it and actually liked it and is paying to use it and maybe got promoted for using it and we made them look really good?
00:11:49.600 --> 00:11:50.320
That's the eval.
00:11:50.559 --> 00:11:52.960
Like Suitebench is great, Suitebench does not matter at all.
00:11:53.120 --> 00:11:55.759
Birdbench is great, Birdbench doesn't matter at all.
00:11:55.840 --> 00:11:57.120
Are people using our models?
00:11:57.360 --> 00:11:57.600
Yeah.
00:11:57.679 --> 00:12:08.639
And you have a very unique vantage point in both leading a research team as well as spending a lot of time with customers and understanding what things are actually working about the models in production and what things aren't.
00:12:08.879 --> 00:12:18.559
What do you see as the largest bottleneck to 10xing enterprise adoption and making these models actually useful on the true eval of the customer?
00:12:18.879 --> 00:12:21.039
Very simply put, it's evals.
00:12:21.120 --> 00:12:21.679
It's measurement.
00:12:21.759 --> 00:12:26.080
You know, there's a lot more to say about that, but the bottom line is I think we have plenty of intelligence.
00:12:26.320 --> 00:12:33.919
I think that, you know, I the line I draw in the sand is like, I think Claude Sonnet 3.5, which also feels like a very quaint statement to make that makes me sound ancient.
00:12:34.080 --> 00:12:36.240
Some people don't remember anything before Claude 4 at this point.
00:12:36.320 --> 00:12:40.080
But back in my day, we had Sonnet 3.5, and that was an awesome
upgrade.
00:12:40.320 --> 00:12:47.919
And from then on, it's kind of felt like the models are good enough at a lot of really complex enterprise tasks and knowledge tasks much of the time.
00:12:48.159 --> 00:12:50.399
The problem isn't the model being smarter.
00:12:50.559 --> 00:12:54.879
You can take the world's smartest person in the world and sit them down at a desk and say, do work.
00:12:55.039 --> 00:12:56.559
That's an underspecified task.
00:12:56.960 --> 00:13:01.840
So I think the real challenge is describing what you want AI to do and knowing whether AI did it.
00:13:02.000 --> 00:13:03.840
That's an eval task by and large.
00:13:04.000 --> 00:13:08.080
If we get better at that, I think we will unlock huge amounts of AI value for enterprises.
00:13:08.240 --> 00:13:18.799
If we don't get better at that, you could give me a model that is 10x smarter than Mythos and the world's smartest person without knowing what they're supposed to do and what it means to do a good job is still going to fail.
00:13:19.039 --> 00:13:25.200
And of course, there's sort of two processes that the enterprise would go through first creating the eval, and then second, hill climbing it.
00:13:25.360 --> 00:13:37.039
How much do you see the hill climbing process involving automated prompt optimization with JEPA or manual prompt optimization versus custom model training to hill climb those evals?
00:13:37.279 --> 00:13:38.080
It's a good question.
00:13:38.240 --> 00:13:41.440
The way I think about it is kind of metaphorically when it comes to software.
00:13:41.600 --> 00:13:44.399
Usually, if you're gonna create a prototype, you just throw it together in Python.
00:13:44.559 --> 00:13:52.639
It's not gonna be fast, it's not gonna be great, it's not gonna be robust, but you need to demonstrate that it's the prototype is good enough that it's worth bothering to actually build a high quality version of it.
00:13:52.720 --> 00:13:53.919
Or you make a wireframe in Figma.
00:13:54.080 --> 00:13:54.639
That's where you start.
00:13:54.720 --> 00:14:00.720
You don't go and build out a whole user interface and make a perfect product, write it in Rust, and then say, like, is anybody gonna buy this?
00:14:00.799 --> 00:14:02.799
Make a quick prototype and see if anybody likes it.
00:14:02.960 --> 00:14:10.639
And I think this is true even more so with agents, where we're at the early stages of knowing what a good agent is and when an agent is worth building.
00:14:10.879 --> 00:14:20.320
First, just pull up your favorite foundation model, do a little prompt engineering, maybe build one or two quick eval examples or kind of vibe checking examples, iterate on it a bit and be like, is this even possible?
00:14:20.720 --> 00:14:26.559
If I throw the world's best model at this and spend a lot of time coaching it, can I get it through this task successfully?
00:14:26.720 --> 00:14:28.000
Like prove that it's worth it.
00:14:28.159 --> 00:14:32.960
And then from there, show that people are like people actually like this, show that you can make it good.
00:14:33.200 --> 00:14:34.960
That's where evals come in, obviously.
00:14:35.039 --> 00:14:37.440
And that's where you know actually testing it with users comes in.
00:14:37.600 --> 00:14:39.360
And it's okay if it's expensive initially.
00:14:39.519 --> 00:14:40.879
This is still a proof of concept.
00:14:41.039 --> 00:14:42.159
Time is much more valuable.
00:14:42.320 --> 00:14:45.840
If you can fail fast, if you can determine that this is working, sure, spend the money.
00:14:46.080 --> 00:14:49.200
I think Cursor is kind of in some sense a giant example of this.
00:14:49.360 --> 00:14:51.919
They built an incredible product that was very popular on that.
00:14:52.159 --> 00:14:56.240
And once you have commitment to a use case, you know that something's going to be around for a while.
00:14:56.399 --> 00:14:58.240
That's a great time to train a custom model.
00:14:58.399 --> 00:15:00.960
Where, you know, I think of this as like switching from Python to Rust.
00:15:01.039 --> 00:15:01.759
You're really committed.
00:15:01.919 --> 00:15:04.159
Nobody should write their first prototype in Rust, I think.
00:15:04.320 --> 00:15:06.799
Um I don't know, maybe some people really, really like Rust.
00:15:06.960 --> 00:15:10.639
But when you're really committed to something, yeah, you need to make it efficient and make it scalable.
00:15:10.799 --> 00:15:11.519
Write it in Rust.
00:15:11.600 --> 00:15:12.320
And it's the same thing.
00:15:12.480 --> 00:15:13.840
Do some custom RL on a model.
00:15:13.919 --> 00:15:17.279
But there are lots of stages in between where, you know, there are a lot of steps in the ladder.
00:15:17.360 --> 00:15:19.440
Custom proptimization is awesome.
00:15:19.679 --> 00:15:20.720
Run Jeppa.
00:15:20.799 --> 00:15:22.879
Use Jeppa against an eval set that you've built.
00:15:22.960 --> 00:15:27.360
That'll get you something better than you could probably tune by hand or at least faster, and you'll get some more value out of it.
00:15:27.519 --> 00:15:30.559
Figure out whether you can use a smaller model for it to make it more efficient and more scalable.
00:15:30.720 --> 00:15:37.600
But at the end of the day, once you've committed to something at large scale, you know, custom models can shave off 5x or more of your cost.
00:15:37.840 --> 00:15:40.320
And, you know, we're seeing Cursor talk about this.
00:15:40.399 --> 00:15:42.559
I think Harvey just mentioned something about this recently.
00:15:42.720 --> 00:15:44.960
And I'm sure there are lots of other companies that haven't talked about it yet.
00:15:45.120 --> 00:15:49.440
I know at our summit this week, there are a bunch of slides that had custom models showing up all over the place.
00:15:49.519 --> 00:15:57.679
And, you know, I won't say any more than is in those slides, but you can imagine, you know, that's because these use cases are so popular that we know they're going to be around
for a while and it's worth committing.
00:15:58.240 --> 00:15:58.480
Yeah.
00:15:58.639 --> 00:16:05.360
You and I are probably two of the most eval-pilled people in the world, but I imagine that a lot of enterprises aren't there yet.
00:16:05.519 --> 00:16:07.919
Uh, how are you seeing that journey develop?
00:16:08.080 --> 00:16:15.600
Like, are enterprises actually building robust evals, or do you think that's more so something that they need to do over the next 12 months?
00:16:15.919 --> 00:16:21.840
It's kind of, I think it's a bit of a bell curve, but I wouldn't underestimate how fast enterprises are getting savvy about this.
00:16:22.000 --> 00:16:26.240
I met with like the C-suite of a major healthcare company a few months ago.
00:16:26.399 --> 00:16:28.159
People were gonna ask, like, what is AI?
00:16:28.240 --> 00:16:28.960
What is an agent?
00:16:29.120 --> 00:16:31.600
They were grilling me on the finer points of evals and judges.
00:16:32.159 --> 00:16:36.639
Like the CFO and like the CEO were grilling me on the finer points of judges.
00:16:36.799 --> 00:16:38.559
Okay, this is like it's getting there.
00:16:38.799 --> 00:16:39.279
It's getting there.
00:16:39.360 --> 00:16:42.799
I think we've been through enough of these big tech transitions in the past decade or two.
00:16:42.960 --> 00:16:48.960
You know, cloud, mobile, like, I don't know, crypto, say what you will about that, but you know, felt like a tech transition for a minute.
00:16:49.120 --> 00:17:00.320
You know, even just the web, that I think any company leader who's survived this long and has grown this far knows that when you see a tech trend coming, learn as much as you can about it and prepare yourself and your company for it.
00:17:00.480 --> 00:17:02.320
So I see people understanding the concept.
00:17:02.559 --> 00:17:06.640
The commitment to really do it and do it well enough that you can build a flywheel is harder.
00:17:06.799 --> 00:17:11.680
I met with a Databricks customer this week that has actually annotated 12,000 examples.
00:17:11.920 --> 00:17:15.279
Um, they're in a medically sensitive application where they have to be really careful.
00:17:15.440 --> 00:17:18.799
They have 70-something judges that they've built and calibrated.
00:17:18.960 --> 00:17:20.640
It's insanely impressive.
00:17:20.799 --> 00:17:22.640
Like I was blown away when I heard this.
00:17:22.720 --> 00:17:24.240
And I've been chatting with them for a couple of years.
00:17:24.400 --> 00:17:27.519
Initially, it was talking to them about the importance of judges two years ago.
00:17:27.599 --> 00:17:27.759
Yeah.
00:17:27.920 --> 00:17:31.200
And now they are the most judge-pilled, eval-pilled people in the world.
00:17:31.359 --> 00:17:32.480
And it's extraordinary.
00:17:32.640 --> 00:17:37.920
They're like the level of rigor is something beyond basically anything I've ever seen anywhere, even if you paid somebody to do it.
00:17:38.079 --> 00:17:43.519
But they need to because
they're in an area where using AI related to medicine would be very risky otherwise.
00:17:43.680 --> 00:17:46.400
So I'm seeing them as a leading indicator of where things are going.
00:17:46.559 --> 00:17:49.680
And once you have that, the amount of value you start to get out of it is huge.
00:17:49.839 --> 00:17:53.599
I'm also seeing kind of this debate between online and offline evals.
00:17:53.759 --> 00:17:59.519
Like how much do you do eval that is, you know, a static set of examples that you can constantly run against, your kind of unit tests.
00:17:59.680 --> 00:18:02.160
And how much are you just inspecting traces every day?
00:18:02.319 --> 00:18:08.960
And how do you get a pipeline from inspecting traces back into your offline eval, kind of so that, you know, when you see something weird, you add a unit test to it?
00:18:09.119 --> 00:18:11.599
I don't think any of these processes have really been figured out yet.
00:18:11.680 --> 00:18:18.240
We're kind of, I know we've talked about this, like metaphorically, we're in the 1960s of software for AI and you know, we're still using punch cards.
00:18:18.319 --> 00:18:22.160
I can't wait for us to discover the object-oriented programming of what AI looks like.
00:18:22.400 --> 00:18:41.839
Well, and we've worked with so many customers that have the thumbs up, thumbs down flywheels that they try to initially use for eval, but obviously the challenge with that is even if you have thumbs down signal, that doesn't give you the full understanding of what was the problem with that, what could have been different that would have made that a correct solution, and how do you effectively hill climb that in a iterative way?
00:18:42.000 --> 00:18:54.480
And so that ties so much into how do we effectively understand the full problem space of what's possible, what's good, what's bad, and measure that in a reliable way, which ties into a lot of the importance of offline evals.
00:18:54.640 --> 00:19:06.319
But maybe going off of that, I imagine that in the enterprise, you've seen so many examples of enterprises that are sophisticated, like you're describing, but also a lot of examples of enterprises that aren't quite there yet.
00:19:06.480 --> 00:19:09.839
What have been your largest takeaways on what makes a good eval?
00:19:10.079 --> 00:19:15.839
When you use the word rigor, what does that mean in the enterprise context of these evals actually being useful?
00:19:16.079 --> 00:19:20.720
So I'll give you the like grand sweeping cosmic version of this and the highly concrete version of this.
00:19:20.880 --> 00:19:23.119
The cosmic version is like imagine Waymo cars.
00:19:23.200 --> 00:19:29.119
We let 16-year-olds on the road if they can drive in a circle in parallel park and not, you know, crash the car, basically.
00:19:29.279 --> 00:19:31.920
Would we let a Waymo on the road if it could pass that test?
00:19:32.000 --> 00:19:39.200
And like, you know, we let 16-year-olds literally go and drive
on any highway in the country if they can like drive in a circle and turn the car around properly without crashing.
00:19:39.359 --> 00:19:40.480
It's kind of crazy.
00:19:40.640 --> 00:19:45.440
We would never let a Waymo loose on public highways if it could pass that test.
00:19:45.759 --> 00:19:50.160
Um, and you know, I some people say, well, we're unfair to the AI.
00:19:50.240 --> 00:19:55.599
And I look at it the other way as like, with AI, you can scale something to an arbitrarily large scale.
00:19:55.680 --> 00:20:01.039
You know, Waymo is limited only by the number of cars they can manufacture, but otherwise, one software change scales everywhere at once.
00:20:01.200 --> 00:20:07.920
And with AI, you know, we've kind of built our entire world around human strengths and weaknesses and the fact that humans are limited.
00:20:08.000 --> 00:20:12.480
You know, we live in meat space, not cyberspace, as they would say in the 90s, but AI is not.
00:20:12.640 --> 00:20:27.680
So the way that we measure AI has to have a degree of rigor like a Waymo car, you know, where you have four different LIDAR sensors and, you know, annotation going on over the course of a decade to measure every ounce of driving in a way that would have been unimaginable a decade ago.
00:20:27.920 --> 00:20:30.799
And, you know, that's not about measuring AI driving.
00:20:30.960 --> 00:20:32.319
That's about measuring driving.
00:20:32.480 --> 00:20:44.240
And, you know, if I had a 16-year-old child someday and they were learning how to drive, if driving is still a thing at that point, I would want them to drive a Waymo type car that has all those sensors and is giving me and them constant feedback and measuring their driving.
00:20:44.400 --> 00:20:49.839
The driving test should be how do you do over the course of several days driving in a car with that many sensors?
00:20:50.000 --> 00:20:54.880
So I think with AI, the risk increases if you're going to deploy AI widely.
00:20:55.279 --> 00:21:04.480
But it also gives us this opportunity to measure work with a degree of rigor that goes beyond a six-month performance review where somebody says, Yeah, I'm feeling this way about your work.
00:21:04.640 --> 00:21:10.799
Like I don't think anyone believes a performance review is particularly rigorous at any job in the same way the driving test is not rigorous.
00:21:11.039 --> 00:21:13.759
So, what does that look like for any kind of work?
00:21:13.920 --> 00:21:15.119
It starts off with one example.
00:21:15.279 --> 00:21:29.119
A Waymo car starts off with one little driving course, probably on some Google campus somewhere, where there are some street signs and some traffic lights and some fake pedestrians that, you know, some fake people who like carefully walk in front of the car and we see if the car stops or not.
00:21:29.279 --> 00:21:30.480
What does that look like for you?
00:21:30.559 --> 00:21:43.599
It could be one example where you have one spreadsheet or one question you just keep asking, and you have a couple of golden answers, and you see if the output matches according to some mediocre LLM judge.
00:21:43.759 --> 00:21:45.680
But one example is a great place to start.
00:21:45.839 --> 00:21:49.119
And then two examples, and then three examples, whatever you would vibe test.
00:21:49.279 --> 00:21:56.960
I think whenever you see anyone use a new model, everybody's got their favorite question, whether it's the unicorn drawing or, you know, how many R's are there in Strawberry or what have you.
00:21:57.119 --> 00:21:59.599
I know that whenever a new model comes out, I have my favorite questions I ask.
00:22:00.079 --> 00:22:03.039
Internally at Databricks in our agents when the new model is there.
00:22:03.359 --> 00:22:05.119
And those become your e-vals.
00:22:05.279 --> 00:22:07.359
And then you start looking at like where do things go wrong?
00:22:07.519 --> 00:22:08.640
Where did somebody else complain?
00:22:08.799 --> 00:22:11.920
It's kind of vibe testing transforms into evals over time.
00:22:12.079 --> 00:22:14.960
And then eventually you can get systematic once you've deployed something at scale.
00:22:15.119 --> 00:22:21.359
You know, Waymo now, like, you know, as they run into these weird corner cases, they issue a recall because it's a real product.
00:22:21.519 --> 00:22:24.880
They add that to their eval suite and they collect a lot more data about that situation.
00:22:25.039 --> 00:22:26.240
I think that's how you bootstrap it.
00:22:26.480 --> 00:22:36.079
When these enterprises put a lot of time and effort into creating a robust eval, have you seen cases where they're comfortable sharing that with foundation model labs?
00:22:36.240 --> 00:22:39.440
Or in most cases, are you finding that they're extremely sensitive to that?
00:22:39.759 --> 00:22:41.279
I think they're extremely sensitive to that.
00:22:41.359 --> 00:22:44.000
And I'm gonna choose my words carefully here.
00:22:44.160 --> 00:22:48.160
But if you're a company, your data is basically your lifeblood.
00:22:48.400 --> 00:22:55.680
Like it's some sense, you know, and data is both the literal data you have and kind of the human capital, the data that's invested in that human capital.
00:22:55.839 --> 00:23:02.240
If you're building an LLM, data is also your lifeblood, but it's your lifeblood in the sense that you would like more of it.
00:23:02.559 --> 00:23:12.880
And so it's a very scary thing if you're an enterprise where you have your crown jewels, yeah, and you know that there's a company that both really wants to help you, but also really, really needs more data.
00:23:13.039 --> 00:23:15.119
That's a complicated proposition.
00:23:15.359 --> 00:23:35.119
So I don't know, being at Databricks, it's been nice to kind of be Switzerland in this conversation, where in some sense we're the place where the data lives, we're the place where the models arrive, and we can kind of intermediate a little bit to make it
possible to measure agents and give some feedback without, you know, having this weird relationship where we would like the data for ourselves.
00:23:35.279 --> 00:23:40.160
And we're finding that, of course, this is the source of the real-to-sim gap in so many ways.
00:23:40.319 --> 00:23:46.559
If every enterprise had robust evals that they could share with the labs, we would solve the real-to-sim gap very quickly.
00:23:46.720 --> 00:23:56.799
But because they don't, there needs to be this huge workforce of contractors that we scale out to help Frontier Labs approximate that distribution throughout the entire economy.
00:23:56.960 --> 00:24:04.160
And of course, there is a gap on public benchmarks even today, as much as uh we're trying to make a lot of effort to resolve that.
00:24:04.319 --> 00:24:07.039
How do you think about public benchmarks today?
00:24:07.359 --> 00:24:10.559
Which parts of them are useful, which parts of them aren't?
00:24:10.640 --> 00:24:15.200
And what are what are the largest gaps that uh we as an industry need to solve?
00:24:15.519 --> 00:24:18.079
I mean, it's easy to bash public benchmarks.
00:24:18.240 --> 00:24:20.400
Nobody has ever liked a public benchmark.
00:24:20.559 --> 00:24:26.880
Like I think I've never, in my, you know, very long decade of having been an AI, which I guess makes me ancient at this point.
00:24:27.039 --> 00:24:31.920
I've never seen a public benchmark where someone said, Oh, that benchmark is amazing and clearly measures the real world.
00:24:32.079 --> 00:24:34.559
And people definitely aren't overfitting to that benchmark.
00:24:34.720 --> 00:24:41.599
I mean, CFAR 10, everybody hated, but that's also 32 by 32 images, where if you look at those images, you can't tell that that's an airplane or not.
00:24:41.759 --> 00:24:46.559
Um, ImageNet has like, I think it's 10% of the categories in ImageNet are just dog breeds.
00:24:46.799 --> 00:24:48.559
And then there's one category for cats.
00:24:48.799 --> 00:24:53.119
Like you can tell the people who created ImageNet and how they feel about that, but it's not a realistic benchmark.
00:24:53.279 --> 00:25:00.240
But the ImageNet benchmark alone, I think, is one of the most important contributions to the fact that we have all of this, you know, AI today.
00:25:00.480 --> 00:25:06.079
That benchmark showed that deep learning was valuable and led to a huge amount of investment over five years.
00:25:06.319 --> 00:25:11.440
And even though we totally beat the crap out of that benchmark to the point of uselessness, it's hugely advanced the field.
00:25:11.519 --> 00:25:17.920
So I think we can live in this in-between where new benchmarks challenge us and new public benchmarks are important because they give us a shared measuring stick.
00:25:18.079 --> 00:25:20.480
And we tend to bend them out of shape pretty quickly these days.
00:25:20.640 --> 00:25:26.319
And it's easy to wonder whether people are, you know, bench maxing or people are, you know, literally training on the benchmark data.
00:25:26.400 --> 00:25:27.279
And that's a risk.
00:25:27.440 --> 00:25:30.400
So they're useful to the field, they're useful to advancing science.
00:25:30.559 --> 00:25:38.640
That doesn't mean they're perfect, and they tend to have bias built into them of first of all, what is possible to build in the academic world.
00:25:38.799 --> 00:25:42.319
And second, what the academic world thinks is important for AI.
00:25:42.559 --> 00:25:45.359
For example, there's a lot of emphasis on coding right now, which is understandable.
00:25:45.440 --> 00:25:46.480
Coding is really popular.
00:25:46.640 --> 00:25:51.759
I don't think we're putting enough emphasis on kind of knowledge-related tasks or kind of knowledge synthesis tasks.
00:25:51.839 --> 00:25:53.279
That's very Databricks pilled.
00:25:53.359 --> 00:25:55.359
So take that, Freddy Will, and very enterprise pilled.
00:25:55.440 --> 00:25:56.559
You know, I think that's true.
00:25:56.720 --> 00:25:59.920
I mean, like software engineering is something like 1% of the economy, right?
00:26:00.160 --> 00:26:02.240
And in Silicon Valley, we feel like it's everything.
00:26:02.559 --> 00:26:03.519
Well, come to New York.
00:26:03.680 --> 00:26:06.319
Come get a bagel with me and you can see the rest of the economy.
00:26:06.480 --> 00:26:10.240
And I'm sure we can look out this window right now and we can see only tech companies.
00:26:10.400 --> 00:26:15.680
But I think there's also like even a bias in the RL environment world toward coding-related environments.
00:26:15.759 --> 00:26:21.519
The idea that a Docker container is sufficient to contain an environment is a very coding-pilled way to look at the world.
00:26:21.680 --> 00:26:27.440
When I look at the world from a Databricks or enterprise pill perspective, many of the environments I look at are tens of petabytes.
00:26:27.599 --> 00:26:28.960
Go put that in a Docker container.
00:26:29.039 --> 00:26:29.279
Yeah.
00:26:29.519 --> 00:26:32.240
And go make that something that you can easily modify and reset.
00:26:32.400 --> 00:26:35.920
Like that's actually a new systems problem that needs to be solved at scale.
00:26:36.160 --> 00:26:43.440
How do you allow an AI agent to work in a 10 petabyte environment, make some changes, and immediately revert that so it doesn't break prod?
00:26:43.759 --> 00:26:49.279
And how do you do that without literally breaking prod because the environment you're working in is also your production environment?
00:26:49.359 --> 00:26:52.480
That's something very different than I think the academic world is focused on.
00:26:52.640 --> 00:26:53.599
And it's not a bad thing.
00:26:53.759 --> 00:26:55.759
These are just different perspectives, different problems.
00:26:56.000 --> 00:27:02.480
But I think the thing that's always missing in any academic task, you know, going back to time immemorial in computer science is scale.
00:27:02.640 --> 00:27:09.440
The story
goes that when Google came into existence, it basically killed the field of information retrieval in academia for a decade.
00:27:09.519 --> 00:27:14.720
Because why would you do information retrieval on a nice synthetic task in academia when you literally have the whole internet?
00:27:14.960 --> 00:27:16.240
It's just a different scale of problem.
00:27:16.319 --> 00:27:17.920
And we're watching that happen in academia again.
00:27:18.079 --> 00:27:36.319
Like look at the number of faculty, yours truly included in some sense, who either never arrived in academia, sorry to everybody at Harvard, or who, you know, have left academia recently because I'm sorry, but working on 16 H100s is just not the same as working at cursor training incredible models or working at Databricks training models or what have you.
00:27:36.480 --> 00:27:37.759
It's the same thing with benchmarks.
00:27:37.920 --> 00:27:39.279
We're missing that scale.
00:27:39.440 --> 00:27:43.839
There's no academic benchmark that works on a tens of petabytes data set, nor can there be.
00:27:44.160 --> 00:27:46.720
What are the public benchmarks that you find most useful today?
00:27:46.960 --> 00:27:47.839
That's a good question.
00:27:48.000 --> 00:27:52.640
I think when like increasingly at Databricks, we're not really working off of any public benchmarks.
00:27:52.720 --> 00:27:59.359
It's all in-house annotated benchmarks and usually on our own data because our data is real and messy.
00:27:59.440 --> 00:28:01.759
And we can at least, it's data we have full access to.
00:28:01.839 --> 00:28:03.279
We don't have to ask anybody's permission.
00:28:03.359 --> 00:28:05.039
We don't have to create a synthetic data set.
00:28:05.200 --> 00:28:07.759
Honestly, I mean, things like Browse Comp Plus have been really useful.
00:28:07.839 --> 00:28:09.119
Things like Stark have been really useful.
00:28:09.200 --> 00:28:11.200
Like a lot of retrieval benchmarks are good.
00:28:11.359 --> 00:28:12.319
They're contrived.
00:28:12.480 --> 00:28:19.359
Like you can read the questions in Browse Comp Plus, and you're not going to say this is a real search anybody would do, but they do challenge some capability of a model in a meaningful way.
00:28:19.599 --> 00:28:19.839
Totally.
00:28:20.079 --> 00:28:21.680
We've released our own benchmarks at this point.
00:28:21.759 --> 00:28:26.000
Like the Office QA benchmark that we've worked really hard on is a parsing and reasoning benchmark.
00:28:26.079 --> 00:28:29.599
Because it turns out that PDF parsing is still an unsolved problem, even in this day and age.
00:28:29.839 --> 00:28:30.160
Shocking.
00:28:30.319 --> 00:28:35.599
And you know, the the fun part, we actually just released it this week, was that it's a benchmark on treasury documents.
00:28:35.680 --> 00:28:42.480
And we actually worked with the Treasury and got, you know, Treasury documents going back to 1789 to try to parse.
00:28:42.880 --> 00:28:46.160
And good frickin' luck parsing
benchmarks in 1789.
00:28:46.240 --> 00:28:49.599
We have a copy of like Alexander Hamilton's paycheck, which is just hilarious.
00:28:50.079 --> 00:28:52.319
Jonathan, is that actually in the user distribution?
00:28:52.799 --> 00:28:57.519
Um, some Databricks customers are actually hundreds of years old and actually have some data like this.
00:28:57.759 --> 00:28:57.920
Okay.
00:28:58.240 --> 00:28:59.119
It's really cool.
00:28:59.279 --> 00:29:07.039
But like I was chatting with a customer that has jet engine schematics going back to before there were jet engines because they developed some of the first ones.
00:29:07.200 --> 00:29:08.319
And that data is there.
00:29:08.559 --> 00:29:09.920
And people want to ask questions about it.
00:29:10.000 --> 00:29:17.680
So, again, not a realistic benchmark in a sense that nobody, at least I don't think any Databricks customers are trying to figure out how much Alexander Hamilton was getting paid in 1790.
00:29:18.079 --> 00:29:21.519
But it's a really challenging benchmark that pushes on some important capabilities.
00:29:21.599 --> 00:29:26.880
And it's one we could package cleanly enough that like now OpenAI includes it in every new model release as a benchmark that they use.
00:29:26.960 --> 00:29:29.359
And we just released a new version of it and we're continuing to iterate.
00:29:29.519 --> 00:29:38.640
Even that is tiny compared to the stuff we're doing internally, where we do have tens of petabytes of workspaces and the environments are dynamic and the answer may actually change day to day.
00:29:38.880 --> 00:29:51.039
Another part of benchmarking that we've talked about is governance, which is obviously becoming extremely important to enterprises as they trust the models with more tools, more data, more access.
00:29:51.200 --> 00:29:53.200
How are you seeing those evals evolve?
00:29:53.359 --> 00:30:00.480
Do most enterprises also have evals for governance yet alongside these evals that they're curating for accuracy?
00:30:00.720 --> 00:30:06.000
Or uh is that still an earlier process that many of them are just beginning to embark on?
00:30:06.160 --> 00:30:08.079
I don't think anyone's even figured out how to measure this yet.
00:30:08.160 --> 00:30:15.680
I think it's worth defining governance because it's a like if you want to talk about an enterprise-pilled word that the average AI person doesn't even think about.
00:30:15.920 --> 00:30:17.359
What access should a model have?
00:30:17.519 --> 00:30:18.880
What access should anyone have?
00:30:19.119 --> 00:30:21.440
And this feels like a really simple question initially.
00:30:21.519 --> 00:30:24.240
Like, you should have access to this code, but not that code.
00:30:24.400 --> 00:30:30.799
You should have access to this information, but not that financial information, or these Google Docs you have permission to, but not those Google Docs.
00:30:30.960 --> 00:30:37.440
And then when you get to a company with a thousand people, let alone 10,000 people, the permissions get so complicated.
00:30:38.079 --> 00:30:53.440
And then also think about I think for anyone out there, the number of times you've made a Google Doc world readable to everybody at your company and have just sent it to a couple of people, but you didn't want to have to keep updating the permissions list, or you've made a document, worst of all, world editable because you didn't want to have to keep updating the list.
00:30:53.680 --> 00:31:00.319
This is the kind of thing that like you probably have access to hundreds of Google Docs that you have never looked at and never thought about.
00:31:00.559 --> 00:31:02.160
Agents tend to find that stuff.
00:31:02.400 --> 00:31:08.960
And so the question of what permissions an agent should have, agents do very creative, very strange things when you give them permissions.
00:31:09.119 --> 00:31:14.720
Things that can make a ton of sense when you really think about it, but are kind of inexplicable in the moment that you wouldn't have predicted.
00:31:14.960 --> 00:31:22.640
And so when you think about an enterprise, there are probably hundreds of thousands or millions of individual permissions that you need to keep track of.
00:31:22.799 --> 00:31:24.640
Even just doing that for humans is complicated.
00:31:24.720 --> 00:31:25.759
And there are all sorts of solutions.
00:31:25.839 --> 00:31:33.039
We have our own at Databricks, but lots of governance solutions just to track that globally across all of your different kinds of data, both within Databricks and beyond.
00:31:33.200 --> 00:31:34.400
Now apply that to agents.
00:31:34.480 --> 00:31:36.480
You're going to give an agent permission to go and pursue a task.
00:31:36.640 --> 00:31:40.960
A lot of people say, like, oh, just give the agents your permissions and you're responsible for the agent's actions.
00:31:41.119 --> 00:31:44.079
Um, the agent is gonna find that doc that you would never have found.
00:31:44.240 --> 00:31:46.480
Agents work at scale in ways that humans don't.
00:31:46.559 --> 00:31:49.519
You don't have time to go looking through all the Google Docs you technically have permission to.
00:31:49.599 --> 00:31:56.160
An agent does and will find something it should not have found, HR-related, legal-related, what have you, that may actually cause some trouble for your business.
00:31:56.240 --> 00:32:00.400
Like the permissions today are probably not very rigorous as it stands.
00:32:00.559 --> 00:32:07.839
And then you as a human need to understand what permissions did I give my agent and what could an agent do with those permissions that I would never have thought of.
00:32:08.000 --> 00:32:13.279
I had a fun example at Databricks where you know I gave an agent access to Jira and a Google Doc.
00:32:13.680 --> 00:32:15.519
It's like, oh, what could go wrong there?
00:32:15.680 --> 00:32:25.279
And actually narrowly avoided a really complicated scenario because I didn't think about what data was in
both places and what would happen if you combine those two permissions together.
00:32:25.440 --> 00:32:29.440
Now multiply that across Slack, you know, glean, what have you.
00:32:29.519 --> 00:32:33.359
And that's just services, not individual Google Docs that you might or might not have access to.
00:32:33.759 --> 00:32:35.759
Things get messy really quickly.
00:32:36.000 --> 00:32:42.480
And I think the evolution for evals is not just about did the task get completed, but what was the process?
00:32:42.640 --> 00:32:45.039
And was that process a good and reasonable process?
00:32:45.279 --> 00:32:46.559
What permissions did the agent need?
00:32:46.720 --> 00:32:47.519
What did it access?
00:32:47.680 --> 00:32:48.640
Should it have accessed that?
00:32:48.799 --> 00:32:50.240
Does it need internet access?
00:32:50.480 --> 00:32:51.680
I gave it internet access.
00:32:51.839 --> 00:32:54.319
There's no like you just have permission to view this one website.
00:32:54.400 --> 00:32:56.079
There's no read and write permissions for the internet.
00:32:56.319 --> 00:32:58.799
There's just get requests and post requests.
00:32:59.039 --> 00:33:03.839
You can write data to the internet really easily without having permission to do that and weird stuff happens.
00:33:04.079 --> 00:33:14.720
It feels like so much of this has come from the fact that agents have improved so much over the last 12 months that enterprises trust them to do more or at least want to.
00:33:14.880 --> 00:33:25.839
So as we give agents more autonomy to interact with access context, to use tools, that also comes with the risk associated with them and how we build the right guardrails.
00:33:26.000 --> 00:33:31.599
I'm curious, how has progress of models and agents over the last 12 months compared to your expectations?
00:33:31.839 --> 00:33:35.839
I think it's been ahead of my expectations in general, but I tend to always be a little more pessimistic.
00:33:35.920 --> 00:33:37.359
You ask my friends three years ago.
00:33:37.440 --> 00:33:47.839
I have a friend who's now at Anthropic where I think he told me two years ago that he thought that by 2026 or 2027, AI was going to be able to address like 50% of all GitHub issues out there.
00:33:47.920 --> 00:33:49.119
It's like, you're an idiot.
00:33:49.279 --> 00:33:50.319
It's going to be like 5%.
00:33:50.720 --> 00:33:52.000
These are really hard tasks.
00:33:52.160 --> 00:33:52.799
He's totally right.
00:33:52.960 --> 00:33:54.079
I was a little pessimistic.
00:33:54.240 --> 00:33:59.920
The way I look at it is more of, you know, this is going to sound dumb, but like roll back the clock to December of 2025.
00:34:00.079 --> 00:34:07.200
I know it's a very long time ago at this point, and ask yourself, what are the like, what did all the VCs say the big trends in AI were going to be in 2026?
00:34:07.359 --> 00:34:13.679
And it was things like we're going to deploy agents widely for work, memory is going to be a really important
topic, open claw was a hot topic.
00:34:13.760 --> 00:34:15.679
Like, we're really going to delegate our whole lives to agents.
00:34:15.840 --> 00:34:17.679
Coding agents were going to be really important.
00:34:18.239 --> 00:34:22.559
It was just this unbridled optimism about everything exciting that was going to happen.
00:34:22.719 --> 00:34:30.639
And I was sitting there, and you were sitting there saying the same thing of like quality is going to matter and measuring quality is going to matter, like evals, evals, evals, and evals were going to be the big trend.
00:34:30.960 --> 00:34:37.039
When I talked to my customers today, I would say the three big trends I hear most or the questions they ask me first are cost.
00:34:37.280 --> 00:34:39.679
We were in a three-month period of token maxing and leaderboards.
00:34:39.760 --> 00:34:42.719
And now we're, God, I can't believe I used this word in front of 1,500 people.
00:34:42.880 --> 00:34:46.960
But I said the era of token maxing is over and the era of value mogging is here.
00:34:47.199 --> 00:34:50.079
And I'm I'm going to cringe, but I said it anyway.
00:34:50.239 --> 00:34:51.920
Again, recorded again.
00:34:52.159 --> 00:34:54.079
Um, the second one is governance.
00:34:54.239 --> 00:34:55.599
What should an agent have access to?
00:34:55.760 --> 00:34:56.239
What shouldn't it?
00:34:56.320 --> 00:35:01.199
And how do you avoid an agent doing really stupid, inexplicable things that actually cause you a lot of risk?
00:35:01.360 --> 00:35:02.400
And the third is security.
00:35:02.559 --> 00:35:07.519
I mean, the fact that now we don't even know what code is being written and an agent can pip install anything.
00:35:07.679 --> 00:35:14.320
And we've seen the number of compromises in pip installations, like that's actually really scaring enterprises.
00:35:14.400 --> 00:35:20.320
And I'm seeing it cause them to shy away from even using software from startups because you don't even know what to do.
00:35:20.480 --> 00:35:28.400
So the all this is to say that like it's very interesting that we've gone from this very optimistic posture, kind of as a field, as an industry, and as business in general.
00:35:28.559 --> 00:35:32.960
I feel like everybody's kind of like battening down the hatches and getting very afraid.
00:35:33.119 --> 00:35:34.880
And I hate that in some sense.
00:35:34.960 --> 00:35:38.320
Like I feel like there's an in-between we can live in where it's been an interesting spring.
00:35:38.480 --> 00:35:39.519
Like some stuff has happened.
00:35:39.599 --> 00:35:42.559
It's been hard and complicated and we've all made some mistakes.
00:35:42.800 --> 00:35:47.360
Yeah, I definitely feel like in the first four or five months of this year, it was all about token maxing.
00:35:47.440 --> 00:35:54.559
It was this first phase of deployment of how do we just plug the models into the context and the tools and see what they can do.
00:35:54.719 --> 00:36:01.360
And then as Athropic and OpenAI revenue skyrocketed, naturally the question became how do we optimize this?
00:36:01.440 --> 00:36:05.360
How do we find the proto-frontier of cost and performance?
00:36:05.519 --> 00:36:10.719
And how do we understand whether we actually need to be using the largest model to accomplish this task?
00:36:10.880 --> 00:36:17.039
And obviously, the foundation for all of that optimization is the Eval, which a lot of it is flowing back to.
00:36:17.119 --> 00:36:18.559
So that makes a lot of sense.
00:36:18.800 --> 00:36:27.039
But I think as models improve so much in the economy, it's also begging this important question of what role will humans play.
00:36:27.199 --> 00:36:34.159
I'm curious how you think about that and the role that humans will play in the enterprise uh over the next five to 10 years.
00:36:34.400 --> 00:36:46.880
Yeah, I think this is one of those things where people have these really oversimplified views of like all the humans will be replaced, or none of the humans will be replaced, or that, you know, entire job functions will go away.
00:36:47.039 --> 00:36:49.119
We won't have any more, I don't know, I'm picking on lawyers.
00:36:49.199 --> 00:36:50.639
I can say this because my parents are both lawyers.
00:36:50.800 --> 00:36:52.079
We'll get rid of all the lawyers or something.
00:36:52.320 --> 00:36:59.360
Not only is that like an oversimplified view that's wrong, but it's also not helpful to tell that story.
00:36:59.519 --> 00:37:03.360
Like the idea that we're going around telling that story is wrong.
00:37:03.519 --> 00:37:11.280
And like I can understand if someone says, yes, we're going to replace all the lawyers, I can understand why a lawyer might say, well, let's not have any more data centers.
00:37:11.440 --> 00:37:17.360
I think this is, you know, if you take a very extreme oversimplified position, I can understand why someone would oppose that position.
00:37:17.599 --> 00:37:22.400
I think we're actually headed for a nice Goldilocks scenario where work is complicated.
00:37:22.639 --> 00:37:27.840
The more time I spend with my customers, the more I learn that like there's no such thing as AI for finance.
00:37:28.239 --> 00:37:43.039
Finance is thousands of different individual little tasks, each of which is both complicated and like nuanced and also different at every single organization because you have all the legacy of how your business operates and how it's structured.
00:37:43.199 --> 00:37:47.679
And there may be three humans in the world, if that, who understand the details of any one task.
00:37:47.840 --> 00:37:50.480
The phrase I like to use is like the world is fractal in complexity.
00:37:50.639 --> 00:37:56.239
You look at any problem and you see the same level of complexity as you look more closely and more closely and more closely.
00:37:56.400 --> 00:38:07.440
So the idea of a finance benchmark or a biology benchmark or a medical benchmark is just to me so absurd and honestly
disrespectful to the folks who are actually doing that work every day.
00:38:07.599 --> 00:38:17.519
And I think what's likely to happen is as with digital computers, as with calculators, as with the internet, work is going to change because there's a lot of stuff we don't need to do anymore.
00:38:17.599 --> 00:38:22.400
You no longer need to pull out your typewriter and typeset something, copy, edit it, and then retype it.
00:38:22.559 --> 00:38:22.960
That's gone.
00:38:23.119 --> 00:38:26.639
We no longer need to have someone who listens to audio tapes and transcribes them.
00:38:26.800 --> 00:38:27.440
That's gone.
00:38:27.599 --> 00:38:30.159
That's really boring work that nobody wants to do.
00:38:30.320 --> 00:38:38.159
And instead, the task of doing law no longer requires a typewriter and document preparation in the like word processing sense is no longer an issue.
00:38:38.320 --> 00:38:39.519
Instead, people are doing more law.
00:38:39.679 --> 00:38:44.480
So I think it's going to push humans both up a level, but it's also going to push humans to the boundary of tasks.
00:38:44.800 --> 00:38:56.239
If the really simple, obvious stuff in any domain is relatively automatable, that doesn't mean we no longer need you any more than I think you were just mentioning to me, like, cool, we've like agriculture has become efficient.
00:38:56.480 --> 00:39:00.559
So now 95% of humans are out of work because they used to all work on agriculture.
00:39:00.719 --> 00:39:03.679
No, we've pushed to the boundaries of more interesting, more sophisticated things.
00:39:03.840 --> 00:39:08.800
A lawyer who can spend their time on the most nuanced, challenging issues, or who can help a lot more people.
00:39:08.960 --> 00:39:12.000
We're not in a world where we have an abundance of legal counsel.
00:39:12.159 --> 00:39:15.280
How many people in the world, do you have an estate plan right now?
00:39:15.519 --> 00:39:16.000
Not yet.
00:39:16.239 --> 00:39:20.639
And you're the kind of person who should definitely have an estate plan, but it takes a while.
00:39:20.800 --> 00:39:20.960
Yeah.
00:39:21.199 --> 00:39:26.400
Like what percent of people have an estate plan, and yet we're all going to need an estate plan someday.
00:39:27.440 --> 00:39:27.599
Yeah.
00:39:27.760 --> 00:39:29.280
That is one universal human thing.
00:39:29.360 --> 00:39:29.519
Yeah.
00:39:29.679 --> 00:39:31.440
We're all going to need an estate plan someday.
00:39:31.599 --> 00:39:34.639
And I say that my dad's an estate lawyer, so this is you know near and dear to my heart.
00:39:35.039 --> 00:39:38.559
But every single person is going to need one and it's expensive and complicated.
00:39:38.719 --> 00:39:40.400
Wouldn't it be great if everyone could have one?
00:39:40.639 --> 00:39:51.440
If we were to itemize all of your customers' problems and even the broader economy into tasks, those tasks obviously require reasoning, but more importantly, they also require context.
00:39:51.599 --> 00:39:54.960
Contextual to that business, contextual to that problem, et cetera.
00:39:55.199 --> 00:40:00.639
What percent of those tasks do you think that the models have actually automated so far?
00:40:00.960 --> 00:40:02.400
Very, very, very little.
00:40:02.639 --> 00:40:06.719
I would say we're in the single-digit percents and probably the very low single-digit percents.
00:40:06.800 --> 00:40:10.239
I think some of it may be very difficult to automate for a really long time.
00:40:10.480 --> 00:40:16.079
The fundamental breakthrough we need, in my view, is not more intelligent models, but more cost-effective e-file development.
00:40:16.239 --> 00:40:25.920
And cost effective, not in the sense of like you spend less money, but that it is for the three humans who know how to do this one tiny enterprise task, it's not miserable for them to do this.
00:40:26.079 --> 00:40:28.000
And they feel like maybe it's even delightful.
00:40:28.079 --> 00:40:36.239
And maybe it's not just an extractive task where they feel like they are getting the knowledge pulled out of them into an agent, but they are growing and learning and they become more sophisticated, what they want to do.
00:40:36.480 --> 00:40:42.079
This is kind of, you know, in some sense, the EM algorithm in AI, you know, you, you know, expectation maximization.
00:40:42.159 --> 00:40:50.239
It's a back and forth process where things get better and where, you know, the idea of building an eval is also a learning process if you structure it that way, if you can make it delightful.
00:40:50.400 --> 00:40:53.440
And to me, that is the ultimate place where we need a breakthrough right now.
00:40:53.599 --> 00:40:57.519
If we can do that, we can unleash AI and unleash humans.
00:40:57.679 --> 00:40:57.920
Yeah.
00:40:58.079 --> 00:41:05.760
I don't think this is mutually exclusive any more than, you know, to what you told me earlier, like making agriculture efficient did unleash humans.
00:41:06.000 --> 00:41:09.599
We had more food, we didn't need as many people to do manual labor.
00:41:09.760 --> 00:41:11.760
And instead, we built our modern world.
00:41:12.000 --> 00:41:14.400
What do you see that work looking like in five years?
00:41:14.559 --> 00:41:23.840
Like, do you think that people spend a lot of time building evals, updating the context of agents, uh just operating at higher levels of abstraction, managing those agents?
00:41:24.079 --> 00:41:29.519
What will that look like as we think about this category for training agents and how it develops?
00:41:29.840 --> 00:41:31.119
God, I wish I knew.
00:41:31.199 --> 00:41:32.639
Um, and I can't wait to find out.
00:41:32.719 --> 00:41:41.039
I can tell you my best guess right now, knowing that we'll laugh at this in three years, just as much as we can laugh that I said you'd never need more than eight GPUs back in 2022.
00:41:41.199 --> 00:41:44.960
But I think, you know, there will be a little bit of like all humans
will become more like managers a little bit.
00:41:45.119 --> 00:41:55.519
We're all gonna have to learn the skills of like how to declaratively describe a task and then check whether the task was done well selectively, know which questions to ask, kind of almost context-free to figure out whether something was done well.
00:41:55.679 --> 00:41:57.119
We're all gonna be writing a lot of process.
00:41:57.199 --> 00:41:59.119
We're all gonna become kind of program managers in a sense.
00:41:59.199 --> 00:42:02.239
We're gonna write process and figure out how to build pipelines of agents.
00:42:02.400 --> 00:42:05.360
So we're all gonna kind of like become our own architects of teams.
00:42:05.519 --> 00:42:10.880
But I think we're all really gonna have to become critical thinkers and get pushed to the boundary of what we do.
00:42:11.119 --> 00:42:18.559
Nothing is gonna be wrote anymore because if it's wrote, we can have an agent do it, which on the one hand is a little bit scary because some of us are very comfortable with our rote task.
00:42:18.639 --> 00:42:20.559
I know how to do performance reviews really well at this point.
00:42:20.719 --> 00:42:22.559
I know how to be a good research manager.
00:42:22.719 --> 00:42:25.840
Um, and that's scary for me because I'm gonna have to be more creative every day.
00:42:26.000 --> 00:42:32.000
But it's also gonna force each of us to like really get to the hard and interesting stuff that makes our work worthwhile.
00:42:32.239 --> 00:42:39.760
In the same way that, like, if you don't have to sit down with a typewriter and typeset, or you don't have to figure out the punch card and walk to the mainframe and plug it in.
00:42:39.920 --> 00:42:42.559
You can think about building software that's really creative.
00:42:42.800 --> 00:42:50.880
I was meeting with, you know, someone who used to be high up at a bank who used to work on mergers in the 80s, and he was telling me what they were like, because I'd mentioned the punch card example.
00:42:51.039 --> 00:42:53.519
He was like, Yeah, a merger used to be a stack of punch cards.
00:42:53.760 --> 00:42:56.320
And I asked him, so how many mergers did you do a year?
00:42:56.480 --> 00:42:56.960
Three.
00:42:57.199 --> 00:43:00.559
And I asked him how sophisticated were the mergers, and he was like, they were all the same structure.
00:43:00.719 --> 00:43:01.280
What about today?
00:43:01.360 --> 00:43:05.599
And he was like, Yeah, an associate can do three mergers, and every merger is creative and interesting.
00:43:05.760 --> 00:43:14.639
And it's led to more dynamism in the economy because we can now conceive of transactions that wouldn't have made sense because we can handle the scale and complexity of it because we have computers.
00:43:14.800 --> 00:43:17.840
And I think that's what we're going to see is just the sophistication of the tasks.
00:43:18.000 --> 00:43:30.719
An example I love is like in the insurance world, the idea that, like, okay, in California, there are wildfires, so we will cancel all insurance policies in California for homeowners is such a 2026 thing to do.
00:43:30.960 --> 00:43:40.239
In 2030, I would hope that with AI, every single home is being like evaluated by a drone that's taking footage and looking at the environment and analyzing it multiple times a year.
00:43:40.400 --> 00:43:42.559
And every policy is custom written.
00:43:42.800 --> 00:43:46.880
And we would never cancel all the policies in California because there was a wildfire in LA.
00:43:47.039 --> 00:43:49.199
We would look at the risk specific to each home.
00:43:49.440 --> 00:43:54.000
An insurance adjuster is going to be thinking about not like rubber stamp this policy.
00:43:54.079 --> 00:43:55.840
Oh, this policy doesn't meet this one criteria.
00:43:55.920 --> 00:43:56.880
Let me follow the process.
00:43:57.039 --> 00:43:58.159
Cool, AI can do that.
00:43:58.320 --> 00:43:59.679
They'll be thinking about how do I ensure.
00:44:00.239 --> 00:44:02.559
Person's home that I could never have ensured before.
00:44:02.639 --> 00:44:02.880
Yeah.
00:44:03.039 --> 00:44:04.480
We'll push to the boundaries of work.
00:44:04.639 --> 00:44:12.719
And I don't know, to me, that's I hate using the word abundance in like I don't mean it in the political sense, and I don't mean it in the um, you know, AGI sense.
00:44:12.800 --> 00:44:19.440
I mean it in the sense of like a lot of things will be easier to create and demand will go up for them.
00:44:19.519 --> 00:44:20.960
We'll need more people to work on them.
00:44:21.119 --> 00:44:26.639
The work will be more interesting, and more people will have access to things like an estate plan at a really low cost.
00:44:26.800 --> 00:44:34.480
We'll all have access to better tax preparation, like these very boring things that we actually spend a lot of time on, but the life of an accountant will also get a lot more interesting.
00:44:34.639 --> 00:44:36.239
And I'm really excited about that world.
00:44:36.559 --> 00:44:41.599
Right now, creating Evals or training agents is a small fraction of a percent of the economy.
00:44:41.760 --> 00:44:45.360
How many multiples larger do you expect that to be in five years?
00:44:45.599 --> 00:44:49.199
I mean, you know, not to flatter you, but I think it's gonna get pretty big.
00:44:49.440 --> 00:44:51.360
Um, it's hard for me to say off the bat.
00:44:51.519 --> 00:44:55.199
I think these things can grow beyond, way beyond our imagination.
00:44:55.440 --> 00:44:59.119
As with everything, the way it's going to grow is that we have to reduce the cost.
00:44:59.199 --> 00:45:01.679
We have to make it easier, more accessible for everyone.
00:45:01.840 --> 00:45:02.960
The market size will grow.
00:45:03.119 --> 00:45:06.400
The cost per eval, I hope, will go down by several orders of magnitude.
00:45:06.559 --> 00:45:11.280
And the market size will go up by several more orders of magnitude, just as with any other good.
00:45:11.360 --> 00:45:16.559
The fact that digital computers have come down so much in cost has meant we have them in our pockets and our watches and now our smoke detectors.
00:45:16.719 --> 00:45:18.559
I hope that evals feel like that.
00:45:18.800 --> 00:45:27.280
Where in five years, you don't need to go to Mercore and say, here's tens of millions of dollars, please give me a really good eval for this task.
00:45:27.360 --> 00:45:37.280
You go to Mercore and you say, here's$1,000 for a license to some amazing product that I'm giving to this one person, but I've actually given it to every single employee at my company.
00:45:37.440 --> 00:45:46.400
And now I'm spending hundreds of millions or billions of dollars on Mercore and the market has grown, the technology has changed and become democratized, just as any digital technology does.
00:45:46.559 --> 00:45:48.159
I hope we get there the way that software has.
00:45:48.320 --> 00:46:04.480
It used to be that you needed to be at a university or a lab and know the right person so you could get access to the mainframe, learn how to use punch cards, and get your time on the computer, potentially in the middle of the night, because these things ran 24-7, so that you could get your program to run, find out what happened, and come back the next day to try again.
00:46:04.639 --> 00:46:06.239
That's what evals feel like today.
00:46:06.400 --> 00:46:10.079
I hope we get to the point where it feels like clawed code for building an eval.
00:46:10.400 --> 00:46:10.559
Yeah.
00:46:10.719 --> 00:46:14.159
The demand for software has gone way up because it's become really accessible.
00:46:14.239 --> 00:46:21.039
And I hope we get there with evals where the demand for evals looks like the demand for software, the market size gets big, and anyone can do this.
00:46:21.280 --> 00:46:28.880
If you had one final parting piece of advice for enterprises that are looking to adopt AI, what would it be?
00:46:29.119 --> 00:46:30.239
I think I would have two.
00:46:30.400 --> 00:46:50.800
One is to like try to like live in almost the dialectic of containing both all the possibilities of AI and the perils and risks as we roll out this new technology that we still don't really understand fully, and contain both, like be able to keep this middle perspective that there are both incredible opportunities and some risks you need to be cognizant of.
00:46:50.960 --> 00:46:52.719
It's so easy to fall into one bucket or the other.
00:46:52.800 --> 00:46:56.079
Like this is dangerous, no AI, or this is incredible.
00:46:56.159 --> 00:46:57.760
We're gonna use AI for everything.
00:46:58.000 --> 00:47:01.039
No more workers, no more anything, just tokens to the moon.
00:47:01.199 --> 00:47:09.280
We have to live in this in-between where the internet changed the world, but also, you know, didn't change a lot of things.
00:47:09.519 --> 00:47:16.800
The other piece I always say to enterprises and to the world at large is be willing to experiment, like controlled sandboxes where you can experiment a little bit and try things.
00:47:17.039 --> 00:47:19.440
Let people bottom up try to use this technology.
00:47:19.599 --> 00:47:26.719
I think, in some sense, if we look back at the internet, like the internet was kind of built from first principles, or the web at least was built from first principles in a democratized way.
00:47:26.800 --> 00:47:28.079
Anyone can put a website online.
00:47:28.239 --> 00:47:32.880
That meant a lot of people did some really stupid things, some really bad things, and some really silly things.
00:47:32.960 --> 00:47:43.440
Like we ended up with like all the like Adobe Flash videos from the late 90s and early 2000s that you weren't even around when they existed, but were hilarious at the time.
00:47:43.519 --> 00:47:45.039
And you know, we got the beginning of memes.
00:47:45.199 --> 00:47:48.480
Open source software, software that anyone can see the code to and edit.
00:47:48.639 --> 00:47:52.000
Nobody's gonna make money off of that, nobody's gonna bother on that, changed the world.
00:47:52.239 --> 00:48:06.559
So the idea that within your business or within the world, like this technology is just available to you and you can try to create something, most things you try to create are gonna be either useless, kind of silly, or maybe even a little dumb and risky, but a handful are going to change your business or change the world.
00:48:06.719 --> 00:48:09.760
And creating the space for folks to do that, I think is really important.
00:48:09.840 --> 00:48:14.079
I mean, the best piece of advice I've gotten about computing is you never know who's going to change the world.
00:48:14.159 --> 00:48:17.039
And so you should try to give as many opportunities to as many people as possible.
00:48:17.199 --> 00:48:19.199
It's like, what would the world look like without Wikipedia?
00:48:19.280 --> 00:48:20.239
It's kind of hard to imagine.
00:48:20.320 --> 00:48:25.920
And that sounded like a really stupid idea where if I was someone's manager, I would tell them, don't bother.
00:48:26.079 --> 00:48:30.079
Like, why are you wasting your time on this like utopian, stupid project?
00:48:30.320 --> 00:48:30.960
I was wrong.
00:48:31.039 --> 00:48:33.440
And I tend to be wrong when I say these things to people on my team.
00:48:33.599 --> 00:48:37.760
So I try to give them some unstructured room to get a prototype out and prove me wrong.
00:48:38.000 --> 00:48:41.199
That led to LLMs at Mosaic and us getting acquired.
00:48:41.280 --> 00:48:49.039
That's led to a bunch of other really key things we built at Databricks where I said no initially, and I'm really glad the person didn't listen to me.
00:48:49.280 --> 00:48:51.440
So, you know, be open minded, I guess.
00:48:51.679 --> 00:48:51.920
Totally.
00:48:52.159 --> 00:48:53.599
Jonathan, an incredible conversation.
00:48:53.679 --> 00:48:54.559
Thank you so much for joining.
00:48:54.800 --> 00:48:55.360
Thank you for having me.
00:48:55.440 --> 00:48:56.639
I'm glad this worked out.