Chris Van Pelt: Machine Learning Tooling, Weights and Biases, Entrepreneurship | Learning from Machine Learning #9

1
00:00:00,000 --> 00:00:02,240
These models are going to get better.
2
00:00:02,240 --> 00:00:03,600
They're going to do more amazing things.
3
00:00:03,600 --> 00:00:06,720
It's an exciting time for us to be in.
4
00:00:06,720 --> 00:00:09,920
But as these models get generally better,
5
00:00:09,920 --> 00:00:13,260
this problem of like, all right, well, when it fails,
6
00:00:13,260 --> 00:00:16,120
knowing how it fails and doing everything we can
7
00:00:16,120 --> 00:00:17,960
to like inform the user and protect against it
8
00:00:17,960 --> 00:00:20,520
is going to become even bigger
9
00:00:20,520 --> 00:00:22,960
because we're going to start trusting these things more.
10
00:00:22,960 --> 00:00:28,460
How did the best machine learning practitioners
11
00:00:28,460 --> 00:00:30,440
get involved in the field?
12
00:00:30,440 --> 00:00:32,920
What challenges have they faced?
13
00:00:32,920 --> 00:00:35,080
What has helped them flourish?
14
00:00:35,080 --> 00:00:36,880
Let's ask them.
15
00:00:36,880 --> 00:00:39,320
Welcome to Learning from Machine Learning.
16
00:00:39,320 --> 00:00:41,480
I'm your host, Seth Levine.
17
00:00:41,480 --> 00:00:44,320
Hello and welcome to Learning from Machine Learning.
18
00:00:44,320 --> 00:00:47,080
On this episode, we have a very special guest,
19
00:00:47,080 --> 00:00:51,200
Chris Van Pelt, the co-founder of Weights and Biases,
20
00:00:51,200 --> 00:00:53,920
the co-founder of CrowdFlower and Figure 8,
21
00:00:53,920 --> 00:00:57,460
and somebody who's dedicated his career optimizing
22
00:00:57,460 --> 00:01:00,480
ML workflows and teaching ML practitioners,
23
00:01:00,480 --> 00:01:03,000
making machine learning more accessible to all.
24
00:01:03,000 --> 00:01:06,720
Chris, it is an absolute pleasure to have you on the show.
25
00:01:06,720 --> 00:01:09,400
It's a pleasure to be here.
26
00:01:09,400 --> 00:01:12,840
Why don't you start us off with what attracted you
27
00:01:12,840 --> 00:01:13,800
to machine learning?
28
00:01:16,680 --> 00:01:20,520
Yeah, this was quite a while ago.
29
00:01:20,520 --> 00:01:25,400
But I remember all the way back in college,
30
00:01:25,400 --> 00:01:30,480
studying computer science in the early 2000s,
31
00:01:30,480 --> 00:01:32,920
talking about machine learning.
32
00:01:32,920 --> 00:01:35,240
But in my college years, it wasn't something
33
00:01:35,240 --> 00:01:39,440
that I immersed myself that deeply into.
34
00:01:39,440 --> 00:01:43,600
It wasn't until a little later, early in my career,
35
00:01:43,600 --> 00:01:47,480
I moved to the Bay Area in 2006 to work
36
00:01:47,480 --> 00:01:50,440
at a startup called PowerSet.
37
00:01:50,440 --> 00:01:53,680
And PowerSet was a startup that was really ahead of its time.
38
00:01:53,680 --> 00:01:56,400
And that was where I first got immersed
39
00:01:56,400 --> 00:01:59,000
in the world of machine learning.
40
00:01:59,000 --> 00:02:01,720
So at PowerSet, we were oddly enough
41
00:02:01,720 --> 00:02:04,240
doing a lot of natural language processing, which
42
00:02:04,240 --> 00:02:06,280
is a hot topic these days.
43
00:02:06,280 --> 00:02:10,520
But we were using a very different approach,
44
00:02:10,520 --> 00:02:15,240
a rules-based, heuristic approach to language modeling.
45
00:02:15,240 --> 00:02:18,200
And we had licensed technology from Xerox Park
46
00:02:18,200 --> 00:02:26,240
and brought on a lot of these very learned professionals,
47
00:02:26,240 --> 00:02:30,560
PhDs in the field, tackling very hard problems
48
00:02:30,560 --> 00:02:32,320
around language understanding and how
49
00:02:32,320 --> 00:02:36,040
we could apply that to search and make a better search product.
50
00:02:36,040 --> 00:02:39,760
It was at that company I also met Lucas B. Walt, who I've now
51
00:02:39,760 --> 00:02:42,200
founded two companies with.
52
00:02:42,200 --> 00:02:48,320
And that is what really launched my career in AI and ML.
53
00:02:48,320 --> 00:02:51,760
My co-founder, Lucas, actually studied machine learning
54
00:02:51,760 --> 00:02:56,840
and had been working with models throughout college
55
00:02:56,840 --> 00:02:58,240
and in his career.
56
00:02:58,240 --> 00:03:01,160
And I'm the full-stack web developer
57
00:03:01,160 --> 00:03:04,160
that landed in this hot and exciting space
58
00:03:04,160 --> 00:03:07,800
that has had the blessing of being
59
00:03:07,800 --> 00:03:12,640
able to create tools for who I consider some of the most
60
00:03:12,640 --> 00:03:16,520
impactful and interesting engineers out there building
61
00:03:16,520 --> 00:03:19,120
the next generation of products and solutions
62
00:03:19,120 --> 00:03:20,000
on top of this stuff.
63
00:03:20,000 --> 00:03:21,440
So exciting times for sure.
64
00:03:21,440 --> 00:03:25,840
I'm glad I landed at that startup in the early 2000s.
65
00:03:25,840 --> 00:03:26,840
Awesome.
66
00:03:26,840 --> 00:03:30,040
How would you say that your background as a full-stack
67
00:03:30,040 --> 00:03:32,960
engineer sort of prepared you for the machine learning world?
68
00:03:32,960 --> 00:03:39,440
Well, I mean, I think the core thing, what I consider to be
69
00:03:39,440 --> 00:03:42,240
like most important when you're an engineer building
70
00:03:42,240 --> 00:03:45,680
products is thinking about the end user experience.
71
00:03:45,680 --> 00:03:49,120
Like how is the world going to interact with this thing?
72
00:03:49,120 --> 00:03:53,040
And I think the same is true and often a lot trickier
73
00:03:53,040 --> 00:03:55,040
with machine learning models.
74
00:03:55,040 --> 00:03:56,880
Like the second you introduce one of these models,
75
00:03:56,880 --> 00:03:59,000
you suddenly have this thing that's
76
00:03:59,000 --> 00:04:03,080
like right some percentage of the time.
77
00:04:03,080 --> 00:04:05,720
And by design, it's going to be wrong.
78
00:04:05,720 --> 00:04:11,480
So thinking about how end users are going to experience that
79
00:04:11,480 --> 00:04:16,040
or ways in which you could potentially make the end user
80
00:04:16,040 --> 00:04:19,640
experience better when they need to get involved and kind
81
00:04:19,640 --> 00:04:24,360
of handle those cases where the model is wrong,
82
00:04:24,360 --> 00:04:27,240
I think has made me hopefully more
83
00:04:27,240 --> 00:04:33,240
than maybe hopefully a better engineer and developer
84
00:04:33,240 --> 00:04:36,360
when it comes to actually bringing these machine learning
85
00:04:36,360 --> 00:04:38,520
models into the real world.
86
00:04:38,520 --> 00:04:39,520
Nice.
87
00:04:39,520 --> 00:04:41,080
Yeah, as a machine learning practitioner,
88
00:04:41,080 --> 00:04:43,720
I get to use my favorite quote like once a week,
89
00:04:43,720 --> 00:04:47,520
all models are wrong, some are useful.
90
00:04:47,520 --> 00:04:49,080
George Box, he was a statistician.
91
00:04:49,080 --> 00:04:50,400
I don't know if he was necessarily talking
92
00:04:50,400 --> 00:04:52,600
about machine learning, but it's still a fun one to get to say.
93
00:04:52,600 --> 00:04:55,480
I'm glad I got to just say it also.
94
00:04:55,480 --> 00:04:58,880
Moving forward a little bit to weights and biases,
95
00:04:58,880 --> 00:05:02,720
which is just an absolutely incredible tool.
96
00:05:02,720 --> 00:05:06,520
I've been using it for a better part of like five years
97
00:05:06,520 --> 00:05:11,400
for every part of my machine learning life cycle
98
00:05:11,400 --> 00:05:12,200
for my projects.
99
00:05:12,200 --> 00:05:13,920
I use it for a bunch of personal projects.
100
00:05:13,920 --> 00:05:16,400
I'm now using it in industry.
101
00:05:16,400 --> 00:05:20,160
Why don't you tell us in your own words as a co-founder,
102
00:05:20,160 --> 00:05:22,120
what is weights and biases?
103
00:05:22,120 --> 00:05:22,720
Yeah, you bet.
104
00:05:22,720 --> 00:05:26,080
So our mission at weights and biases
105
00:05:26,080 --> 00:05:30,080
is to build the world's best developer tools for machine
106
00:05:30,080 --> 00:05:30,960
learning engineers.
107
00:05:30,960 --> 00:05:36,320
So we're really interested in building really good tool.
108
00:05:36,320 --> 00:05:38,400
I've always been a fan of tools.
109
00:05:38,400 --> 00:05:42,160
To have a good tool, to have the right tool for the job
110
00:05:42,160 --> 00:05:47,120
in the real world is there's nothing better.
111
00:05:47,120 --> 00:05:50,600
I don't do a lot of handy work, but going to Home Depot
112
00:05:50,600 --> 00:05:52,840
and looking at the different actual tools
113
00:05:52,840 --> 00:05:55,040
is quite exhilarating for me.
114
00:05:55,040 --> 00:05:56,360
I've always enjoyed that.
115
00:05:56,360 --> 00:06:00,240
So weights and biases is building tools
116
00:06:00,240 --> 00:06:02,600
for machine learning engineers.
117
00:06:02,600 --> 00:06:07,640
So the kinds of tools that a machine learning engineer needs,
118
00:06:07,640 --> 00:06:10,560
it was pretty obvious in the early days.
119
00:06:10,560 --> 00:06:15,280
And as we've grown, it's become more nuanced.
120
00:06:15,280 --> 00:06:18,000
There's like little pockets of the problem space
121
00:06:18,000 --> 00:06:19,560
that we're always kind of going, hey,
122
00:06:19,560 --> 00:06:21,000
is there a better way to do this?
123
00:06:21,000 --> 00:06:23,200
How can create a better tool?
124
00:06:23,200 --> 00:06:24,600
Cluster problems are like, well, you
125
00:06:24,600 --> 00:06:27,280
need to keep track of what data you're training on.
126
00:06:27,280 --> 00:06:31,120
It's always when you're modeling, the data is king.
127
00:06:31,120 --> 00:06:34,640
So we created a number of tools to just have
128
00:06:34,640 --> 00:06:37,640
a solid understanding of data lineage, data versioning,
129
00:06:37,640 --> 00:06:41,720
being able to dive in and visualize, understand the data.
130
00:06:41,720 --> 00:06:43,520
And then there's a lot of experimentation
131
00:06:43,520 --> 00:06:44,280
in machine learning.
132
00:06:44,280 --> 00:06:46,280
So when you're training a machine learning model,
133
00:06:46,280 --> 00:06:48,040
it's not just the source code.
134
00:06:48,040 --> 00:06:50,320
As a traditional software developer, it's like,
135
00:06:50,320 --> 00:06:51,640
all right, I've got GitHub.
136
00:06:51,640 --> 00:06:56,480
I know what the truth is, and I have CI CD running.
137
00:06:56,480 --> 00:06:57,720
It's going to be the data, and it's
138
00:06:57,720 --> 00:07:01,000
going to be some hyperparameters, some command line arguments
139
00:07:01,000 --> 00:07:03,120
that you passed into the program that you're running.
140
00:07:03,120 --> 00:07:05,240
And then ultimately, the weights and the biases
141
00:07:05,240 --> 00:07:08,160
that you're creating when you've trained a model.
142
00:07:08,160 --> 00:07:12,160
So weights and biases is an end-to-end ML ops platform
143
00:07:12,160 --> 00:07:14,840
that helps engineers keep track of all of these things
144
00:07:14,840 --> 00:07:18,640
and then conserve as a system of record for their day-to-day
145
00:07:18,640 --> 00:07:21,680
development and understanding of how these models are
146
00:07:21,680 --> 00:07:25,720
performing and how they can make them perform better.
147
00:07:25,720 --> 00:07:27,600
Very cool.
148
00:07:27,600 --> 00:07:29,520
Yeah, the amazing thing for me is I've
149
00:07:29,520 --> 00:07:33,080
gotten to see how weights and biases has expanded over time
150
00:07:33,080 --> 00:07:35,960
and my usage of it also.
151
00:07:35,960 --> 00:07:39,680
I started out using it really just to keep track of things,
152
00:07:39,680 --> 00:07:41,840
keep track of experiment results.
153
00:07:41,840 --> 00:07:45,240
I think seeing loss curves was very illuminating for me.
154
00:07:45,240 --> 00:07:47,080
I guess I had seen it in fast AI,
155
00:07:47,080 --> 00:07:50,920
but there was something about seeing multiple runs all
156
00:07:50,920 --> 00:07:53,920
in one place, which was really nice.
157
00:07:53,920 --> 00:07:57,840
Towing around with sweeps and creating reports, all of it.
158
00:07:57,840 --> 00:07:59,560
Tables is incredible.
159
00:07:59,560 --> 00:08:03,320
Haven't gotten into Weave yet, but I'm looking forward to it.
160
00:08:03,320 --> 00:08:07,720
And ML prompts also, which is really nice.
161
00:08:07,720 --> 00:08:11,360
And speaking of, yeah, nothing better than a good tool, right?
162
00:08:11,360 --> 00:08:13,600
I mean, the right tool for the job.
163
00:08:13,600 --> 00:08:16,560
It's amazing how seamless it can be.
164
00:08:16,560 --> 00:08:19,400
And also, when you don't have the right tool,
165
00:08:19,400 --> 00:08:21,160
how frustrating it can be when you're
166
00:08:21,160 --> 00:08:27,600
trying to do something, don't try to hammer in something.
167
00:08:27,600 --> 00:08:30,320
You get a hammer, right?
168
00:08:30,320 --> 00:08:33,200
From your perspective, how have the goals of weights
169
00:08:33,200 --> 00:08:38,520
and biases changed since the onset?
170
00:08:38,520 --> 00:08:40,800
Yeah, in the beginning, right?
171
00:08:40,800 --> 00:08:44,400
This is like 2016.
172
00:08:44,400 --> 00:08:49,240
This is a time, I'm sure, much of your audience
173
00:08:49,240 --> 00:08:53,120
could remember, or maybe they were in the space at this point.
174
00:08:53,120 --> 00:08:58,000
But TensorFlow was really the main player
175
00:08:58,000 --> 00:08:59,600
in the framework space.
176
00:08:59,600 --> 00:09:02,560
PyTorch really wasn't a thing.
177
00:09:02,560 --> 00:09:06,400
Computer vision was the use case that everyone
178
00:09:06,400 --> 00:09:08,000
was talking about and excited about.
179
00:09:08,000 --> 00:09:14,320
This was a time when self-driving cars was the primary topic
180
00:09:14,320 --> 00:09:16,760
around AI or applying ML.
181
00:09:19,560 --> 00:09:22,000
The core problem that we set it to solve in the early days
182
00:09:22,000 --> 00:09:25,440
was just keeping track of your experiments.
183
00:09:25,440 --> 00:09:28,920
So state of the art at that time for just keeping
184
00:09:28,920 --> 00:09:31,800
track of your modeling effort was like a Google spreadsheet
185
00:09:31,800 --> 00:09:33,760
or an Excel spreadsheet.
186
00:09:33,760 --> 00:09:37,520
So that was a pretty low bar.
187
00:09:37,520 --> 00:09:40,480
And we just set out to make a tool that
188
00:09:40,480 --> 00:09:43,920
was really easy to keep track of the experiments
189
00:09:43,920 --> 00:09:44,960
that you're doing.
190
00:09:44,960 --> 00:09:46,640
Like originally in the very beginning,
191
00:09:46,640 --> 00:09:48,880
we didn't think putting a whole bunch of charts
192
00:09:48,880 --> 00:09:52,160
into the product was necessarily needed.
193
00:09:52,160 --> 00:09:53,400
Like the main problem we were solving
194
00:09:53,400 --> 00:09:56,400
was just keep track of the actual experiments
195
00:09:56,400 --> 00:09:58,880
and maybe what the final loss value was
196
00:09:58,880 --> 00:10:02,720
or the final accuracy value was.
197
00:10:02,720 --> 00:10:05,760
And then as we added more rich visualization features,
198
00:10:05,760 --> 00:10:10,360
we saw users love it, so we really doubled down there.
199
00:10:10,360 --> 00:10:12,880
The ways in which we've expanded was we
200
00:10:12,880 --> 00:10:15,080
finally found we convinced ourselves
201
00:10:15,080 --> 00:10:17,640
we had product market fit that we had created something that
202
00:10:17,640 --> 00:10:21,280
was useful when we got teams like OpenAI to actually use
203
00:10:21,280 --> 00:10:23,360
the product for work that they were doing
204
00:10:23,360 --> 00:10:27,440
or to a research institute on a lot of the robotics
205
00:10:27,440 --> 00:10:30,120
and autonomous vehicle work.
206
00:10:30,120 --> 00:10:32,280
So then it became like, all right, well,
207
00:10:32,280 --> 00:10:33,680
what other problems are there?
208
00:10:33,680 --> 00:10:35,080
And this was literally just going out
209
00:10:35,080 --> 00:10:37,160
talking to our customers or users
210
00:10:37,160 --> 00:10:41,720
and hearing where their pain points were.
211
00:10:41,720 --> 00:10:45,240
So the sweeps offering inside of Waste Advices
212
00:10:45,240 --> 00:10:48,720
where we make it really easy to run a hyperparameter search,
213
00:10:48,720 --> 00:10:50,160
initially it wasn't obvious.
214
00:10:50,160 --> 00:10:51,840
It was kind of like, well, there's
215
00:10:51,840 --> 00:10:54,760
good tools on the market that do that.
216
00:10:54,760 --> 00:10:57,480
We don't think we're going to magically come out
217
00:10:57,480 --> 00:11:00,120
and have the greatest hyperparameter algorithm that's
218
00:11:00,120 --> 00:11:01,240
going to save everybody money.
219
00:11:01,240 --> 00:11:04,400
It was just like, let's just make it as easy and as pleasant
220
00:11:04,400 --> 00:11:08,480
as possible to run a hyperparameter search.
221
00:11:08,480 --> 00:11:13,640
And since features like our model registry reports
222
00:11:13,640 --> 00:11:15,520
was an interesting set of features
223
00:11:15,520 --> 00:11:17,920
that came out of the reality of like, all right, well,
224
00:11:17,920 --> 00:11:22,240
everyone's report or the end result
225
00:11:22,240 --> 00:11:23,160
is very different.
226
00:11:23,160 --> 00:11:25,560
The kinds of things you want to understand and know about
227
00:11:25,560 --> 00:11:27,160
when you're doing computer vision, very different
228
00:11:27,160 --> 00:11:29,720
than if you're making a financial prediction model
229
00:11:29,720 --> 00:11:33,200
or some arbitrary classifier.
230
00:11:33,200 --> 00:11:36,640
So we built this very flexible platform
231
00:11:36,640 --> 00:11:41,920
to actually communicate these graphs and charts and results
232
00:11:41,920 --> 00:11:43,000
around to customers.
233
00:11:43,000 --> 00:11:44,880
And the product continues to evolve,
234
00:11:44,880 --> 00:11:46,800
I think most recently.
235
00:11:46,800 --> 00:11:51,360
The move from what when we started and for the past five
236
00:11:51,360 --> 00:11:53,960
years, it was always like, OK, you build a model.
237
00:11:53,960 --> 00:11:59,840
Maybe you take ResNet or some existing base model,
238
00:11:59,840 --> 00:12:01,800
but you're going to fine tune it and you're
239
00:12:01,800 --> 00:12:03,280
going to do all of this stuff in-house.
240
00:12:03,280 --> 00:12:05,480
Now it's often, we'll just call out,
241
00:12:05,480 --> 00:12:08,560
so like opening eyes API or some other API
242
00:12:08,560 --> 00:12:12,680
and the kinds of problems and things you need to be concerned
243
00:12:12,680 --> 00:12:13,360
about are different.
244
00:12:13,360 --> 00:12:15,720
But they're similar in a lot of ways as well, right?
245
00:12:15,720 --> 00:12:19,480
These are all machine systems that
246
00:12:19,480 --> 00:12:22,480
have this probabilistic nature that are going to be wrong.
247
00:12:22,480 --> 00:12:26,680
How do we evaluate and how do we try to make the user
248
00:12:26,680 --> 00:12:30,720
experience as good as possible across it?
249
00:12:30,720 --> 00:12:33,040
Yeah, absolutely.
250
00:12:33,040 --> 00:12:34,600
Yeah, there's a certain flexibility
251
00:12:34,600 --> 00:12:37,520
that's really nice with weights and biases
252
00:12:37,520 --> 00:12:43,400
that you can use it for many different use cases.
253
00:12:43,400 --> 00:12:46,440
Speaking of creating tools and sometimes you
254
00:12:46,440 --> 00:12:49,160
have the intended use for tools, what's
255
00:12:49,160 --> 00:12:51,840
a really unique use of weights and biases
256
00:12:51,840 --> 00:12:53,200
that I guess when you were creating it,
257
00:12:53,200 --> 00:12:55,960
you never really thought that it would be used for it?
258
00:12:58,400 --> 00:13:00,520
Yeah, I mean, the weights and biases platform itself,
259
00:13:00,520 --> 00:13:06,160
it's pretty versatile in terms of the core.
260
00:13:06,160 --> 00:13:08,440
As you're building a product, you're like, all right,
261
00:13:08,440 --> 00:13:12,120
well, what are the atoms of this thing?
262
00:13:12,120 --> 00:13:18,960
And I remember we built this feature a few years ago
263
00:13:18,960 --> 00:13:23,200
where we let people completely define their own visualization.
264
00:13:23,200 --> 00:13:27,080
So we built it on top of Vega, which there's
265
00:13:27,080 --> 00:13:30,920
like an altair is the Python framework that works with this
266
00:13:30,920 --> 00:13:31,880
visualization framework.
267
00:13:31,880 --> 00:13:33,640
Under the hood, it's all like D3, which
268
00:13:33,640 --> 00:13:34,920
is a very cool technology.
269
00:13:34,920 --> 00:13:38,320
But we wired up Vega such that users
270
00:13:38,320 --> 00:13:42,000
could define their own custom visualizations.
271
00:13:42,000 --> 00:13:46,160
And they could wire that up to any of the atoms
272
00:13:46,160 --> 00:13:50,640
in the weights and biases API, these units of data.
273
00:13:50,640 --> 00:13:54,440
And one of our engineers actually wired things up
274
00:13:54,440 --> 00:13:57,600
and defined a custom visualization that was actually
275
00:13:57,600 --> 00:14:02,200
like a role playing game, which I thought was awesome.
276
00:14:02,200 --> 00:14:07,440
A complete misuse of both the core Vega spec
277
00:14:07,440 --> 00:14:12,200
and the underlying data model, but a very cool demo,
278
00:14:12,200 --> 00:14:13,600
nonetheless.
279
00:14:13,600 --> 00:14:17,440
I think in the actual use cases of weights and biases,
280
00:14:17,440 --> 00:14:21,440
I've been able to see some very cool use cases of machine
281
00:14:21,440 --> 00:14:22,760
learning over the years.
282
00:14:22,760 --> 00:14:27,640
One of my favorite examples is technology around agriculture.
283
00:14:27,640 --> 00:14:30,320
So putting computer vision models
284
00:14:30,320 --> 00:14:34,360
onto big tractors and combines and reducing
285
00:14:34,360 --> 00:14:38,400
the amount of pesticides or chemicals
286
00:14:38,400 --> 00:14:44,280
that need to be applied to control weeds in a field
287
00:14:44,280 --> 00:14:46,840
has a massive impact on the environment.
288
00:14:46,840 --> 00:14:48,000
It's really cool tech.
289
00:14:48,000 --> 00:14:49,520
Like I went and saw one of the tractors,
290
00:14:49,520 --> 00:14:52,200
and they have little NVIDIA boxes
291
00:14:52,200 --> 00:14:54,680
like on the combine doing it.
292
00:14:54,680 --> 00:14:57,280
And also not one, it's not the first place your mind goes,
293
00:14:57,280 --> 00:14:59,720
where you're like, how could we use AI or ML
294
00:14:59,720 --> 00:15:02,400
to make some impact in the world?
295
00:15:02,400 --> 00:15:05,640
But yeah, the work we've done with John Deere and Blue
296
00:15:05,640 --> 00:15:08,440
River around that has been really cool to see.
297
00:15:08,440 --> 00:15:11,400
Very cool.
298
00:15:11,400 --> 00:15:13,000
In terms of all of the things that
299
00:15:13,000 --> 00:15:14,840
have been accomplished from weights and biases,
300
00:15:14,840 --> 00:15:18,520
I'm sure that you guys have a nice roadmap ahead.
301
00:15:18,520 --> 00:15:20,600
What are some of the things that you're most excited about
302
00:15:20,600 --> 00:15:24,560
for the future for weights and biases?
303
00:15:24,560 --> 00:15:26,920
Yeah, so I think the most exciting thing
304
00:15:26,920 --> 00:15:29,400
is this next generation of tooling
305
00:15:29,400 --> 00:15:34,360
for really the next generation of AI and ML engineers.
306
00:15:34,360 --> 00:15:38,800
What's happened in the last year, year and a half
307
00:15:38,800 --> 00:15:42,520
with the explosion of chat GPT, and now every data science
308
00:15:42,520 --> 00:15:44,640
conference you go to is definitely
309
00:15:44,640 --> 00:15:48,840
going to have the words like LLM or Gen AI
310
00:15:48,840 --> 00:15:50,560
somewhere on a poster.
311
00:15:50,560 --> 00:15:54,440
It's been just wild to see the whole industry
312
00:15:54,440 --> 00:16:02,120
shift to this excitement around these large models.
313
00:16:02,120 --> 00:16:03,800
The team is working on, all right,
314
00:16:03,800 --> 00:16:06,960
well, what does a product look like where you're not
315
00:16:06,960 --> 00:16:09,560
necessarily doing a lot of modeling in-house.
316
00:16:09,560 --> 00:16:12,520
You're leveraging these tools, doing more prompt engineering,
317
00:16:12,520 --> 00:16:16,960
doing more like the retrieval augmented generation space,
318
00:16:16,960 --> 00:16:19,360
kind of hooking these tools together
319
00:16:19,360 --> 00:16:27,240
and with agents and these more general purpose uses of LLM.
320
00:16:27,240 --> 00:16:30,640
It's like, what would the world's best tooling
321
00:16:30,640 --> 00:16:33,080
look like for that new world?
322
00:16:33,080 --> 00:16:40,920
That's what the team's been working on over the last year.
323
00:16:40,920 --> 00:16:46,680
And we're excited to finally release that in the next couple
324
00:16:46,680 --> 00:16:50,200
of months here and continue to iterate on it.
325
00:16:50,200 --> 00:16:53,440
As we found with our existing product,
326
00:16:53,440 --> 00:16:57,800
it's like we make a swing, we try to make something as good
327
00:16:57,800 --> 00:17:01,360
as we think it can be, and then through actually having people
328
00:17:01,360 --> 00:17:05,720
use it and solve problems, we can iterate and make
329
00:17:05,720 --> 00:17:06,840
it great and delightful.
330
00:17:06,840 --> 00:17:10,720
So that's really the area we're focusing a lot on.
331
00:17:10,720 --> 00:17:12,560
I think one of the big shifts there
332
00:17:12,560 --> 00:17:15,240
is that from the start of the company,
333
00:17:15,240 --> 00:17:18,600
we're selling a product to machine learning engineers.
334
00:17:18,600 --> 00:17:21,080
These are people that understand the underlying math.
335
00:17:21,080 --> 00:17:24,920
They understand probabilities and what
336
00:17:24,920 --> 00:17:29,520
that means from an operational standpoint.
337
00:17:29,520 --> 00:17:31,880
In this new world, we have a lot of just traditional software
338
00:17:31,880 --> 00:17:34,840
developers that are now consuming these APIs
339
00:17:34,840 --> 00:17:36,920
and building products on top of them.
340
00:17:36,920 --> 00:17:39,160
So one of the challenges is, how do we
341
00:17:39,160 --> 00:17:47,160
convey these core ideas to this new audience in a way that
342
00:17:47,160 --> 00:17:53,960
enables them to build better products without a lot of the.
343
00:17:53,960 --> 00:17:55,640
There's a lot of new, it's tricky.
344
00:17:55,640 --> 00:17:57,160
There's going to potentially be bias.
345
00:17:57,160 --> 00:17:59,080
You need to really think carefully about, OK,
346
00:17:59,080 --> 00:18:01,520
when this thing fails, how's it going to fail?
347
00:18:01,520 --> 00:18:06,240
You want to fail in a way that's least disruptive
348
00:18:06,240 --> 00:18:08,800
to the end user.
349
00:18:08,800 --> 00:18:11,840
So being able to build tools for this space,
350
00:18:11,840 --> 00:18:13,040
it's really exciting.
351
00:18:13,040 --> 00:18:15,920
And there's hundreds of other companies doing the same thing.
352
00:18:15,920 --> 00:18:18,280
So we've got a lot of work to do, and we need to do quickly.
353
00:18:18,280 --> 00:18:20,720
But it's an exciting space to be in.
354
00:18:20,720 --> 00:18:23,880
Yeah, that's definitely one of the most challenging things
355
00:18:23,880 --> 00:18:26,600
with machine learning versus, like, say,
356
00:18:26,600 --> 00:18:27,920
traditional programming.
357
00:18:27,920 --> 00:18:29,760
If it doesn't work for traditional programming,
358
00:18:29,760 --> 00:18:31,080
you just get an error, right?
359
00:18:31,080 --> 00:18:32,320
I mean, usually, most of the time,
360
00:18:32,320 --> 00:18:34,120
unless it's something really weird.
361
00:18:34,120 --> 00:18:37,640
But machine learning, you'll get an answer.
362
00:18:37,640 --> 00:18:40,160
But it won't be right.
363
00:18:40,160 --> 00:18:45,640
And with an API call, you will generate text,
364
00:18:45,640 --> 00:18:47,600
you will generate some image, but will it
365
00:18:47,600 --> 00:18:52,120
be useful for what you're actually trying to do?
366
00:18:52,120 --> 00:18:57,080
And understanding the, I guess, the responsibility
367
00:18:57,080 --> 00:18:59,480
that people have when they're creating things like that,
368
00:18:59,480 --> 00:19:02,200
it's a real transition.
369
00:19:02,200 --> 00:19:03,400
And it's tempting.
370
00:19:03,400 --> 00:19:06,280
You can make a cool demo today.
371
00:19:06,280 --> 00:19:09,160
Like, it's been so fun as an engineer having access
372
00:19:09,160 --> 00:19:12,880
to this technology and to delight myself when I make something
373
00:19:12,880 --> 00:19:13,960
and I'm like, whoa, it did that.
374
00:19:13,960 --> 00:19:16,680
I can't believe it did it.
375
00:19:16,680 --> 00:19:19,440
But that demo, where you then kind of script it
376
00:19:19,440 --> 00:19:23,640
and you're showing your friends this cool thing you made,
377
00:19:23,640 --> 00:19:28,280
it does not account for all of the weird edge cases
378
00:19:28,280 --> 00:19:30,320
and things you haven't thought about in ways in which
379
00:19:30,320 --> 00:19:33,600
another user is going to interact with this thing.
380
00:19:33,600 --> 00:19:35,280
And if you just throw that out there,
381
00:19:35,280 --> 00:19:38,080
you're not even going to know really if it's working or not.
382
00:19:38,080 --> 00:19:40,280
The closest proxy you'll have is like,
383
00:19:40,280 --> 00:19:42,640
are people sharing it and more people using it.
384
00:19:42,640 --> 00:19:45,400
But even thinking about, all right, well,
385
00:19:45,400 --> 00:19:49,040
how do I get user feedback?
386
00:19:49,040 --> 00:19:53,400
I mean, this goes back to that first job I had here
387
00:19:53,400 --> 00:19:56,680
in the valley, getting into machine learning.
388
00:19:56,680 --> 00:19:58,440
When it's a search engine, how do you
389
00:19:58,440 --> 00:20:01,800
know if you're a machine learning algorithm that return
390
00:20:01,800 --> 00:20:03,520
results is any good?
391
00:20:03,520 --> 00:20:08,800
Well, a good proxy is like, are people clicking on the results?
392
00:20:08,800 --> 00:20:12,480
But it's a subtle gnarly problem.
393
00:20:12,480 --> 00:20:14,440
And you need to really think about it
394
00:20:14,440 --> 00:20:17,320
and have rigorous ways to evaluate and understand
395
00:20:17,320 --> 00:20:19,000
if you're getting better or worse, because you're
396
00:20:19,000 --> 00:20:20,200
going to have to change the prompt.
397
00:20:20,200 --> 00:20:21,400
You're going to upgrade the model.
398
00:20:21,400 --> 00:20:23,360
You're going to change things about your product.
399
00:20:23,360 --> 00:20:26,240
And you need a way to actually measure,
400
00:20:26,240 --> 00:20:30,320
is this thing good or bad without just sending it out
401
00:20:30,320 --> 00:20:33,200
to your users and making them kind of yell, hey, what the heck?
402
00:20:33,200 --> 00:20:34,880
This sucks.
403
00:20:34,880 --> 00:20:36,200
Yeah, for sure.
404
00:20:36,200 --> 00:20:40,280
And speaking of the ability to create demos,
405
00:20:40,280 --> 00:20:42,120
maybe I'm not sure if it's over said or anything,
406
00:20:42,120 --> 00:20:43,880
but something I've been finding myself saying,
407
00:20:43,880 --> 00:20:45,880
it's easy to create a demo.
408
00:20:45,880 --> 00:20:48,640
It's hard to create something for production.
409
00:20:48,640 --> 00:20:50,920
And it's even harder to create something at scale.
410
00:20:50,920 --> 00:20:52,600
Something can work a dozen times.
411
00:20:52,600 --> 00:20:54,440
But is it going to work a thousand times?
412
00:20:54,440 --> 00:20:56,040
How's it going to work a million times?
413
00:20:56,040 --> 00:20:57,960
How's it going to work when there's multiple users
414
00:20:57,960 --> 00:20:58,840
at the same time?
415
00:20:58,840 --> 00:21:01,600
How's it going to work on all of these edge cases?
416
00:21:01,600 --> 00:21:04,600
And I think that what we're seeing
417
00:21:04,600 --> 00:21:06,920
is that especially with this generative AI,
418
00:21:06,920 --> 00:21:09,280
you can't even test all of these things.
419
00:21:09,280 --> 00:21:12,720
You can't even fully check it for prompt injection,
420
00:21:12,720 --> 00:21:15,520
let's say, because until it's out there
421
00:21:15,520 --> 00:21:19,000
and people are starting to use it for these unintended uses,
422
00:21:19,000 --> 00:21:21,880
that's when you start to see all these crazy things come out.
423
00:21:21,880 --> 00:21:23,640
But it's already kind of too late,
424
00:21:23,640 --> 00:21:25,360
because it's in production.
425
00:21:25,360 --> 00:21:26,160
Someone is using it.
426
00:21:26,160 --> 00:21:30,240
They have, it's already exposed.
427
00:21:30,240 --> 00:21:31,400
It's already out.
428
00:21:31,400 --> 00:21:33,120
We're seeing lots of things like that happen
429
00:21:33,120 --> 00:21:35,440
where people are putting out generative chatbots
430
00:21:35,440 --> 00:21:37,120
for their customer service.
431
00:21:37,120 --> 00:21:39,880
And it's just like, that's a terrible idea
432
00:21:40,880 --> 00:21:43,920
to do that fully, to just be fully relying on that.
433
00:21:43,920 --> 00:21:46,040
And there's obviously other examples too.
434
00:21:47,640 --> 00:21:49,880
But yeah, speaking of evaluation,
435
00:21:49,880 --> 00:21:51,280
it's really hard.
436
00:21:51,280 --> 00:21:55,640
How do you know if your product is working correctly?
437
00:21:55,640 --> 00:21:59,720
So yeah, something like search, it's very difficult, right?
438
00:21:59,720 --> 00:22:02,000
You might want, you could quickly get results,
439
00:22:02,000 --> 00:22:03,600
but are they the right results?
440
00:22:04,880 --> 00:22:06,440
Recommendation engines, right?
441
00:22:06,440 --> 00:22:07,800
You can quickly get results,
442
00:22:07,800 --> 00:22:09,680
but are they the right results?
443
00:22:09,680 --> 00:22:12,560
I think evaluation will always remain a problem,
444
00:22:14,320 --> 00:22:18,040
especially because I think people put too much weight
445
00:22:18,040 --> 00:22:19,680
on benchmarks as well.
446
00:22:21,120 --> 00:22:23,400
I don't know what you're feeling is on that.
447
00:22:23,400 --> 00:22:24,240
You think about that one?
448
00:22:24,240 --> 00:22:25,320
The benchmarks are very generic, right?
449
00:22:25,320 --> 00:22:27,480
So then, some will make an announcement and say,
450
00:22:27,480 --> 00:22:32,480
hey, we're better than GPT-4 in like MMLU or,
451
00:22:34,680 --> 00:22:37,080
I'm not even sure if that's one of the correct acronyms
452
00:22:37,080 --> 00:22:40,720
of the 30 core tests that people are throwing out there.
453
00:22:42,760 --> 00:22:45,680
And there's not, those are important.
454
00:22:45,680 --> 00:22:50,680
It's good to have some general set of benchmarks
455
00:22:50,720 --> 00:22:54,400
for different things that we're testing,
456
00:22:54,400 --> 00:22:56,840
but they're very general and they're never gonna tell you
457
00:22:56,840 --> 00:22:59,800
how good is this thing gonna be for my specific use case.
458
00:22:59,800 --> 00:23:03,560
You're the only one who can answer that question.
459
00:23:03,560 --> 00:23:05,600
And it could be hard to answer it.
460
00:23:05,600 --> 00:23:09,920
So like going back to the search engine ranking algorithm,
461
00:23:11,240 --> 00:23:12,080
well, how do you do that?
462
00:23:12,080 --> 00:23:16,400
Well, it turns out you hire a bunch of people
463
00:23:16,400 --> 00:23:19,400
who are trained often with a big manual
464
00:23:19,400 --> 00:23:23,160
on here's how we define relevance,
465
00:23:23,160 --> 00:23:25,120
which is already a pretty fuzzy subject,
466
00:23:25,120 --> 00:23:28,880
like how relevant is something to a given user's query.
467
00:23:28,880 --> 00:23:30,400
And then you have them label the data.
468
00:23:30,400 --> 00:23:32,720
You look at a whole bunch of queries and results
469
00:23:32,720 --> 00:23:35,000
and you have them on a scale of like one to four
470
00:23:35,000 --> 00:23:39,080
or one to five, say how relevant a given result is
471
00:23:39,080 --> 00:23:41,600
for a query and even then you're like, okay, well,
472
00:23:41,600 --> 00:23:44,880
you have to, the query could be ambiguous.
473
00:23:44,880 --> 00:23:47,360
It's hard to understand what a user's intent is
474
00:23:47,360 --> 00:23:48,720
when they query.
475
00:23:48,720 --> 00:23:53,000
These problems are very similar in the chat space
476
00:23:53,000 --> 00:23:55,960
or having a user ask for something.
477
00:23:55,960 --> 00:23:58,440
And then when the ultimate result comes out,
478
00:23:58,440 --> 00:24:01,040
you have to, you need some way to measure,
479
00:24:01,040 --> 00:24:05,600
well, okay, did this satisfy the user's question?
480
00:24:05,600 --> 00:24:07,800
We see it in chat GPT itself.
481
00:24:07,800 --> 00:24:11,400
We can give a little thumbs up or thumbs down.
482
00:24:11,400 --> 00:24:15,360
Most people probably don't interact in that way.
483
00:24:15,360 --> 00:24:18,040
When a user does, it's a really strong signal.
484
00:24:18,040 --> 00:24:20,120
Right, so you should probably incorporate that data
485
00:24:20,120 --> 00:24:25,120
back into your process and use it to make the model better.
486
00:24:26,360 --> 00:24:29,080
But yeah, I mean, the good news is companies
487
00:24:29,080 --> 00:24:33,240
have been working on this problem for 20 years.
488
00:24:34,560 --> 00:24:38,840
The bad news is every individual has like a slightly
489
00:24:38,840 --> 00:24:42,440
different definition of good for whatever they're doing.
490
00:24:42,440 --> 00:24:45,480
So there isn't just this magic, I can buy this product
491
00:24:45,480 --> 00:24:47,880
and it's gonna like solve this problem for me.
492
00:24:47,880 --> 00:24:51,200
What you need, you need like really good tools to help you
493
00:24:51,200 --> 00:24:53,120
ask the question and solve the problem,
494
00:24:53,120 --> 00:24:55,400
which is why we built weights and biases
495
00:24:55,400 --> 00:24:58,280
and hope we can really help a lot of people
496
00:24:59,440 --> 00:25:02,320
put this rigorous process in place to be able to build
497
00:25:02,320 --> 00:25:05,640
a robust data science machine learning function.
498
00:25:05,640 --> 00:25:06,920
Yeah, absolutely.
499
00:25:07,880 --> 00:25:10,800
One of the things that weights and biases has helped me,
500
00:25:10,800 --> 00:25:14,120
it's like you try to get this one metric, right?
501
00:25:14,120 --> 00:25:15,720
Like you try to get like, oh, okay,
502
00:25:15,720 --> 00:25:17,640
F1 score is above 0.8, right?
503
00:25:17,640 --> 00:25:20,120
But it doesn't really matter that much.
504
00:25:20,120 --> 00:25:21,880
It's sometimes it's about how it's performing
505
00:25:21,880 --> 00:25:24,520
on different segments of your data.
506
00:25:24,520 --> 00:25:26,960
And I found that tables has helped me a lot.
507
00:25:26,960 --> 00:25:30,240
I've been able to look at different probability distributions
508
00:25:30,240 --> 00:25:34,000
for different classes and also just to see where there's,
509
00:25:34,000 --> 00:25:37,120
to see where there's errors and to sort of segment the data
510
00:25:37,120 --> 00:25:40,480
and then see, okay, in this particular type of conversation
511
00:25:40,480 --> 00:25:44,880
that I'm analyzing, you know, this is what I wanna be looking
512
00:25:44,880 --> 00:25:46,800
for, okay, I need to, these are like,
513
00:25:46,800 --> 00:25:48,960
it helps me with error analysis basically
514
00:25:48,960 --> 00:25:51,880
and to zoom in on those problems.
515
00:25:51,880 --> 00:25:55,760
Because often, yeah, it's not just about one accuracy metric
516
00:25:55,760 --> 00:25:59,880
or one particular thing, you have to sort of have this ability
517
00:25:59,880 --> 00:26:01,520
to zoom in and zoom out.
518
00:26:01,520 --> 00:26:03,440
And that's one thing like weights and biases
519
00:26:03,440 --> 00:26:05,400
has really helped me with.
520
00:26:05,400 --> 00:26:06,760
Yeah, I mean, this idea is like, you know,
521
00:26:06,760 --> 00:26:11,360
a confusion matrix of like, I'm making a model to predict
522
00:26:11,360 --> 00:26:13,400
like whether or not you have COVID.
523
00:26:13,400 --> 00:26:17,680
Like if it, like false positives versus false negative,
524
00:26:17,680 --> 00:26:19,760
it's like different for the use case.
525
00:26:19,760 --> 00:26:22,320
Like I would, if I tell someone they have COVID
526
00:26:22,320 --> 00:26:25,160
and they don't actually have it, probably not as bad
527
00:26:25,160 --> 00:26:27,120
as me telling someone they don't have COVID
528
00:26:27,120 --> 00:26:28,520
when they actually have it.
529
00:26:28,520 --> 00:26:29,360
Right.
530
00:26:29,360 --> 00:26:31,880
So how do you wanna optimize your model for these cases?
531
00:26:31,880 --> 00:26:35,440
What can you do to really prevent that?
532
00:26:35,440 --> 00:26:38,720
Like the case you don't want, right?
533
00:26:38,720 --> 00:26:42,720
These broad like F1 score 80%, yeah, it means nothing.
534
00:26:42,720 --> 00:26:45,960
How many times am I gonna be like lying to my user
535
00:26:45,960 --> 00:26:48,240
about this thing that's really important?
536
00:26:48,240 --> 00:26:49,160
Right.
537
00:26:49,160 --> 00:26:50,000
Yeah.
538
00:26:50,000 --> 00:26:52,960
It's whenever the cost of errors aren't equal
539
00:26:52,960 --> 00:26:55,720
and it's always that case, right?
540
00:26:55,720 --> 00:26:58,760
Cause the cost of errors are never the same.
541
00:26:59,840 --> 00:27:04,200
So therefore the metric can't just be this overall metric
542
00:27:04,200 --> 00:27:06,720
where you're treating true positives and false positives
543
00:27:06,720 --> 00:27:09,520
or whatever, you know, true negatives and whatever.
544
00:27:09,520 --> 00:27:13,120
All of your combinations in your confusion matrix,
545
00:27:13,120 --> 00:27:15,480
each box matters differently
546
00:27:15,480 --> 00:27:17,920
and you have to be able to somehow incorporate that.
547
00:27:17,920 --> 00:27:19,440
And the only way you can really do that
548
00:27:19,440 --> 00:27:21,960
is by, you know, segmenting it.
549
00:27:21,960 --> 00:27:24,280
Especially when you're iterating, like, you know,
550
00:27:24,280 --> 00:27:28,080
maybe I moved F1 score from 80% to 90%.
551
00:27:29,280 --> 00:27:31,240
That's a no, of course, let's ship that model.
552
00:27:31,240 --> 00:27:33,960
Well, wait, like look at those cases.
553
00:27:33,960 --> 00:27:35,760
Did the cases get better or worse?
554
00:27:35,760 --> 00:27:37,280
Cause maybe overall you got better,
555
00:27:37,280 --> 00:27:39,800
but now you're like way worse on the false positive
556
00:27:39,800 --> 00:27:40,800
or whatever case.
557
00:27:40,800 --> 00:27:42,600
And that's really important to know.
558
00:27:42,600 --> 00:27:43,440
Right.
559
00:27:43,440 --> 00:27:45,720
Or you'll just get better at the majority class
560
00:27:45,720 --> 00:27:48,680
and then you won't even ever detect the rare class
561
00:27:48,680 --> 00:27:52,000
and you'll think, oh, okay, yeah, my model's better.
562
00:27:52,000 --> 00:27:54,080
I know people just wanna know is this,
563
00:27:54,080 --> 00:27:55,600
is model A better than model B,
564
00:27:55,600 --> 00:27:57,640
but there's always some trade off.
565
00:27:57,640 --> 00:28:00,160
It's never, very rarely do you ever get it
566
00:28:00,160 --> 00:28:02,880
like across the board that one thing is better,
567
00:28:02,880 --> 00:28:07,240
you know, categorically better than another model.
568
00:28:07,240 --> 00:28:10,160
You know, I mean, like these models are gonna get better.
569
00:28:10,160 --> 00:28:11,520
They're gonna do more amazing things.
570
00:28:11,520 --> 00:28:14,640
It's an exciting time for us to be in.
571
00:28:14,640 --> 00:28:17,840
But as these models get generally better,
572
00:28:17,840 --> 00:28:21,160
this problem of like, all right, well, when it fails,
573
00:28:21,160 --> 00:28:24,040
knowing how it fails and doing everything we can
574
00:28:24,040 --> 00:28:25,880
to like inform the user and protect against it,
575
00:28:25,880 --> 00:28:28,440
it's gonna become even bigger.
576
00:28:28,440 --> 00:28:31,040
Cause we're gonna start trusting these things more.
577
00:28:31,040 --> 00:28:33,360
Like I bet we'll never get rid of hallucination
578
00:28:33,360 --> 00:28:37,560
because by definition of the way these things work,
579
00:28:37,560 --> 00:28:41,080
there's some weird corner case or something weird
580
00:28:41,080 --> 00:28:44,120
with the data that's gonna like be really bad.
581
00:28:44,120 --> 00:28:45,600
It's very important to understand that
582
00:28:45,600 --> 00:28:48,040
and do it we can to prevent users
583
00:28:48,040 --> 00:28:52,200
from having a bad experience because of it.
584
00:28:52,200 --> 00:28:54,760
Yeah, 100%.
585
00:28:54,760 --> 00:28:57,360
Yeah, I know, I always find it so funny
586
00:28:57,360 --> 00:29:02,000
like companies say we have eliminated hallucinations.
587
00:29:02,000 --> 00:29:05,400
If you've said that, then don't trust that company
588
00:29:05,400 --> 00:29:06,720
because they don't know what they're talking about.
589
00:29:06,720 --> 00:29:08,240
It's like eliminating bias.
590
00:29:08,240 --> 00:29:10,400
It's like, no, you have not eliminated bias.
591
00:29:10,400 --> 00:29:13,400
You can try to minimize it, but you cannot eliminate it.
592
00:29:13,400 --> 00:29:15,280
And if you think that you have,
593
00:29:15,280 --> 00:29:18,040
then you didn't really fully think through your problem.
594
00:29:18,960 --> 00:29:19,800
Yeah.
595
00:29:20,880 --> 00:29:24,160
So just like looking at this space and, you know,
596
00:29:24,160 --> 00:29:26,160
obviously like the last year and a half
597
00:29:26,160 --> 00:29:29,000
has been this hype cycle, right?
598
00:29:29,000 --> 00:29:31,040
But you've been in this industry, you know,
599
00:29:31,040 --> 00:29:35,000
since like 2007, were there any other like big
600
00:29:36,400 --> 00:29:39,720
revolutionary like step function things like this
601
00:29:39,720 --> 00:29:41,960
that really created such hype?
602
00:29:41,960 --> 00:29:43,720
Have you ever seen something like this
603
00:29:43,720 --> 00:29:45,400
like chatGPT has created?
604
00:29:47,240 --> 00:29:48,200
Not to this level.
605
00:29:48,200 --> 00:29:52,760
I mean, this is astronomical hype and it like continues.
606
00:29:52,760 --> 00:29:55,400
I kind of thought like, all right, people will chill.
607
00:29:56,800 --> 00:29:59,120
But there's still like every conference I go to,
608
00:29:59,120 --> 00:30:03,760
every company I talk to, they're, you know,
609
00:30:03,760 --> 00:30:05,680
deploying a lot of resources to figure out
610
00:30:05,680 --> 00:30:10,000
how generative AI is gonna change how they function,
611
00:30:10,000 --> 00:30:10,960
how the world functions.
612
00:30:10,960 --> 00:30:15,960
So this is definitely unlike anything I've ever experienced.
613
00:30:16,200 --> 00:30:21,200
The closest is maybe the, yeah,
614
00:30:22,480 --> 00:30:24,480
the hype around autonomous vehicles.
615
00:30:24,480 --> 00:30:27,520
Really like when we first started weights and biases,
616
00:30:27,520 --> 00:30:29,440
it was clear that, okay, deep learning was really
617
00:30:29,440 --> 00:30:31,440
starting to work.
618
00:30:31,440 --> 00:30:35,320
Like the things that the demos I was seeing,
619
00:30:35,320 --> 00:30:40,240
how good these models were getting at just taking in pixels
620
00:30:40,240 --> 00:30:43,840
and spitting out like what everything in that image was
621
00:30:43,840 --> 00:30:47,360
or putting bounding boxes around important objects was,
622
00:30:47,360 --> 00:30:49,360
I remember seeing examples of it being like, wow,
623
00:30:49,360 --> 00:30:52,160
I did not think we'd be able to do this
624
00:30:52,160 --> 00:30:54,000
when we were able to do it.
625
00:30:54,000 --> 00:30:54,840
Right.
626
00:30:54,840 --> 00:30:59,440
And I think you saw, you know, a ton of money go into
627
00:30:59,440 --> 00:31:01,480
a ton of different companies trying to make
628
00:31:01,480 --> 00:31:05,720
self-driven cars and predictions of having a self-driven car
629
00:31:05,720 --> 00:31:10,200
before, you know, well before we actually were able to have it.
630
00:31:10,200 --> 00:31:13,400
But, you know, now I'm going around streets of San Francisco
631
00:31:13,400 --> 00:31:17,400
and seeing the ways cars drive by without someone in them
632
00:31:17,400 --> 00:31:19,240
or taking rides in them, which is trippy.
633
00:31:19,240 --> 00:31:20,600
Like you've been in it.
634
00:31:20,600 --> 00:31:21,440
It's here.
635
00:31:21,440 --> 00:31:26,120
It's a little bit longer than any of us had hoped,
636
00:31:26,120 --> 00:31:28,240
but it's here.
637
00:31:28,240 --> 00:31:29,080
You've taken one?
638
00:31:29,080 --> 00:31:32,000
I think, yeah, yeah, a couple of times.
639
00:31:32,000 --> 00:31:33,000
It's cool.
640
00:31:33,000 --> 00:31:33,840
Creepy?
641
00:31:33,840 --> 00:31:35,000
It's very cool.
642
00:31:35,000 --> 00:31:36,600
Yeah, definitely a little creepy.
643
00:31:39,320 --> 00:31:42,320
And I've seen it's gotten into, I love like writing in ways
644
00:31:42,320 --> 00:31:44,520
because you like see some situation and you'll be like,
645
00:31:44,520 --> 00:31:46,320
I want to get like a bag of popcorn and be like,
646
00:31:46,320 --> 00:31:47,800
what's it going to do here?
647
00:31:47,800 --> 00:31:50,440
We've got like construction codes.
648
00:31:50,440 --> 00:31:52,560
Homeless person doing something crazy.
649
00:31:52,560 --> 00:31:53,760
Let's like see.
650
00:31:53,760 --> 00:31:56,680
I've always been pleasantly surprised.
651
00:31:56,680 --> 00:31:58,480
Right, yeah.
652
00:31:58,480 --> 00:32:01,320
Yeah, I don't know.
653
00:32:01,320 --> 00:32:03,280
It's creepy.
654
00:32:03,280 --> 00:32:05,800
Are there steering wheels or there's no steering wheel?
655
00:32:05,800 --> 00:32:07,040
Yeah, there's a steering wheel.
656
00:32:07,040 --> 00:32:09,160
You can even sit in the driver's seat.
657
00:32:09,160 --> 00:32:12,200
Apparently you have to keep your hands off of the,
658
00:32:12,200 --> 00:32:13,880
I haven't done that.
659
00:32:13,880 --> 00:32:14,880
Yeah.
660
00:32:14,880 --> 00:32:16,560
I get in usually in the back seat or something
661
00:32:16,560 --> 00:32:18,960
and I'll take like a video because I'm still, you know,
662
00:32:18,960 --> 00:32:21,920
when you see the wheel turning and it's going.
663
00:32:21,920 --> 00:32:24,080
Yeah, it's pretty cool.
664
00:32:24,080 --> 00:32:25,440
I guess it works.
665
00:32:25,440 --> 00:32:28,360
It needs to stay within a certain area though, right?
666
00:32:28,360 --> 00:32:30,280
It can't go outside of a certain area.
667
00:32:30,280 --> 00:32:31,320
Is that how it is?
668
00:32:32,880 --> 00:32:34,960
It takes some weird routes.
669
00:32:34,960 --> 00:32:35,800
Oh, okay.
670
00:32:35,800 --> 00:32:36,920
Like it's definitely like its route planner
671
00:32:36,920 --> 00:32:39,640
is not just like Google Maps.
672
00:32:39,640 --> 00:32:40,840
Yeah.
673
00:32:40,840 --> 00:32:43,480
But yeah, I don't know how they license it with the city
674
00:32:43,480 --> 00:32:46,280
or if there's certain like no go zones.
675
00:32:46,280 --> 00:32:50,200
But they also like the tech on those things is nuts.
676
00:32:50,200 --> 00:32:53,480
That is not a cheap vehicle to operate
677
00:32:53,480 --> 00:32:56,280
and there's lots of light ours and all these things
678
00:32:56,280 --> 00:32:58,320
that Elon doesn't like.
679
00:32:58,320 --> 00:32:59,960
But you know, it turns out it makes the problem
680
00:32:59,960 --> 00:33:02,480
a lot more doable.
681
00:33:02,480 --> 00:33:03,720
But yeah.
682
00:33:03,720 --> 00:33:05,880
Take in whatever senses you need to take in
683
00:33:05,880 --> 00:33:06,840
to get that done.
684
00:33:06,840 --> 00:33:10,680
You don't have to have it be some all knowing
685
00:33:10,680 --> 00:33:12,640
omniscient sort of model.
686
00:33:12,640 --> 00:33:17,520
It can take in multiple senses. Yeah, that's cool.
687
00:33:17,520 --> 00:33:19,680
I need to look into it even more.
688
00:33:19,680 --> 00:33:21,680
I don't know if I would take it or not.
689
00:33:21,680 --> 00:33:24,000
I guess eventually that'll become commonplace.
690
00:33:24,000 --> 00:33:25,960
You do it enough, you'll be exposed to it.
691
00:33:25,960 --> 00:33:29,640
You'll be, you'll stop taking, you know, stop taking videos.
692
00:33:29,640 --> 00:33:31,000
Come on, you know, it's exciting man.
693
00:33:31,000 --> 00:33:33,320
It's, you should take it.
694
00:33:33,320 --> 00:33:34,160
Yeah.
695
00:33:34,160 --> 00:33:35,720
I'll come to San Francisco.
696
00:33:35,720 --> 00:33:37,240
I'll get you a ride in one.
697
00:33:37,240 --> 00:33:38,080
I appreciate it.
698
00:33:38,080 --> 00:33:42,880
I would take a ride with you in a driverless car.
699
00:33:42,880 --> 00:33:45,480
I would do it.
700
00:33:45,480 --> 00:33:46,600
Very cool.
701
00:33:46,600 --> 00:33:50,560
So with all of this hype and everything
702
00:33:50,560 --> 00:33:53,480
that's happening in, you know, let's say natural language
703
00:33:53,480 --> 00:33:56,560
processing, but really just like the machine learning world,
704
00:33:56,560 --> 00:34:01,280
how do you view the gap between the hype and the reality?
705
00:34:01,280 --> 00:34:03,920
So like what the promise is of all of this stuff
706
00:34:03,920 --> 00:34:08,040
and then like where we actually are?
707
00:34:08,040 --> 00:34:08,880
Yeah.
708
00:34:08,880 --> 00:34:13,480
Well, like I said, I'm surprised that the, like,
709
00:34:13,480 --> 00:34:16,720
where we're still like peak hype from what I can see.
710
00:34:16,720 --> 00:34:18,520
So, you know, we're going to reach,
711
00:34:18,520 --> 00:34:21,080
we're going to hit the trough of disillusionment
712
00:34:21,080 --> 00:34:21,720
at some point.
713
00:34:21,720 --> 00:34:23,400
This is the, you know, the Gartner hype cycle.
714
00:34:26,240 --> 00:34:35,680
I think, you know, a big gap, like this space moves so fast.
715
00:34:35,680 --> 00:34:37,640
You know, waste and biases has been around five years.
716
00:34:37,640 --> 00:34:41,040
The amount of change, you know, the transformer architecture,
717
00:34:41,040 --> 00:34:43,520
for instance, like wasn't a thing until 2017.
718
00:34:43,520 --> 00:34:47,840
And now that's basically the most popular architecture used
719
00:34:47,840 --> 00:34:51,240
in everything from the self-driven cars
720
00:34:51,240 --> 00:34:55,280
to these language models.
721
00:34:55,280 --> 00:34:58,760
And, you know, I'm sure there'll be another architecture
722
00:34:58,760 --> 00:35:00,640
or changes to this architecture that
723
00:35:00,640 --> 00:35:03,160
proved to be even more fruitful.
724
00:35:03,160 --> 00:35:12,160
So, the, yeah, well, I think the speed is jarring.
725
00:35:12,160 --> 00:35:16,480
And then when you get these big enterprise companies
726
00:35:16,480 --> 00:35:19,840
figuring out how to use this new thing, they're slow.
727
00:35:19,840 --> 00:35:23,160
Like they're still, you know, very much being cautious
728
00:35:23,160 --> 00:35:26,400
and figuring it out.
729
00:35:26,400 --> 00:35:28,120
And, you know, we're just sitting,
730
00:35:28,120 --> 00:35:30,840
we're waiting for the number of transistors
731
00:35:30,840 --> 00:35:35,600
that NVIDIA can pack into their gyps to go up, which it will.
732
00:35:35,600 --> 00:35:38,000
And then these models will get better.
733
00:35:38,000 --> 00:35:43,840
And I saw, there was like an interview with Sam Altman,
734
00:35:43,840 --> 00:35:45,960
saying a lot of people think, like, oh, we'll get this,
735
00:35:45,960 --> 00:35:50,040
like, AGI or even the couple weeks after chat GPT blew up,
736
00:35:50,040 --> 00:35:51,760
everyone was like, oh, my god, this is going to, like,
737
00:35:51,760 --> 00:35:53,520
change everything now.
738
00:35:53,520 --> 00:35:55,360
It takes time.
739
00:35:55,360 --> 00:35:59,360
It is the actual process of finding the killer use cases
740
00:35:59,360 --> 00:36:05,720
for this and making it a core part of what you're doing.
741
00:36:05,720 --> 00:36:07,600
It will take time.
742
00:36:07,600 --> 00:36:10,560
I think, well, you look at, like, why
743
00:36:10,560 --> 00:36:13,400
Combinator and the startups coming out of that now,
744
00:36:13,400 --> 00:36:19,600
like, the majority are somehow connected to this space.
745
00:36:24,160 --> 00:36:26,720
What was the original question?
746
00:36:26,720 --> 00:36:28,080
What are the challenges going to be?
747
00:36:28,080 --> 00:36:29,840
Yeah.
748
00:36:29,840 --> 00:36:34,600
No, the gap between the hype and the reality.
749
00:36:34,600 --> 00:36:38,160
Yeah, I mean, I think this is self-serving.
750
00:36:38,160 --> 00:36:40,120
One of the big gaps is just better tooling,
751
00:36:40,120 --> 00:36:44,640
like, having visibility into how these things are performing
752
00:36:44,640 --> 00:36:46,400
and actually operationalizing it.
753
00:36:49,040 --> 00:36:51,360
You know, I think that's the thing that's happened is,
754
00:36:51,360 --> 00:36:52,600
you can use like GPT-4.
755
00:36:52,600 --> 00:36:53,960
It does these amazing things.
756
00:36:53,960 --> 00:36:57,600
But it's slow, and it's expensive at scale.
757
00:36:57,600 --> 00:36:59,200
So then people are, all right, well, yeah,
758
00:36:59,200 --> 00:37:02,480
we'll take Lama 2 and find, well, now you
759
00:37:02,480 --> 00:37:05,800
need to have a robust like MLOps process and practice
760
00:37:05,800 --> 00:37:08,560
to iterate on that model and understand its shortcomings
761
00:37:08,560 --> 00:37:13,600
and prevent all of these safety-related issues.
762
00:37:13,600 --> 00:37:16,040
So I think the gap now is that, yeah, there
763
00:37:16,040 --> 00:37:18,760
aren't a lot of push-button-managed solutions
764
00:37:18,760 --> 00:37:20,080
out there.
765
00:37:20,080 --> 00:37:21,680
Often, the use cases of these things
766
00:37:21,680 --> 00:37:23,760
are so specialized and unique that you kind of need
767
00:37:23,760 --> 00:37:26,480
to build out some internal expertise.
768
00:37:26,480 --> 00:37:30,240
And everyone's just kind of figuring that out now.
769
00:37:30,240 --> 00:37:34,720
So I guess I'd expect all of this to get better.
770
00:37:37,440 --> 00:37:42,800
But yeah, I guess I can't offer a win as soon as possible.
771
00:37:42,800 --> 00:37:46,080
It's definitely what we're working on.
772
00:37:46,080 --> 00:37:48,240
But it's clear this is not going anywhere.
773
00:37:48,240 --> 00:37:50,200
And there's a ton of potential.
774
00:37:50,200 --> 00:37:55,040
Like, I'm delighted by just like chat GPT on a daily basis
775
00:37:55,040 --> 00:37:58,160
and thinking of ideas for how this could be applied
776
00:37:58,160 --> 00:38:02,680
to different processes within organizations.
777
00:38:02,680 --> 00:38:03,920
Yeah, 100%.
778
00:38:03,920 --> 00:38:06,600
It's a really good brainstorm partner.
779
00:38:06,600 --> 00:38:07,760
You could give it some ideas.
780
00:38:07,760 --> 00:38:09,400
It could really, really helps out.
781
00:38:09,400 --> 00:38:11,200
And you can have a nice little back and forth.
782
00:38:11,200 --> 00:38:14,800
It generates very interesting ideas.
783
00:38:14,800 --> 00:38:17,720
And then you were touching upon another interesting thing,
784
00:38:17,720 --> 00:38:23,160
which was like the hardware that's involved with these systems.
785
00:38:23,160 --> 00:38:27,000
And obviously, there's an NVIDIA, which is a huge player.
786
00:38:27,000 --> 00:38:29,520
And then Google has their TPUs.
787
00:38:29,520 --> 00:38:32,440
And then there's this new thing like LPU.
788
00:38:32,440 --> 00:38:36,200
It's very interesting to think that now there's hardware
789
00:38:36,200 --> 00:38:41,040
that's going to be designed specifically for these use cases.
790
00:38:41,040 --> 00:38:43,080
So yeah, it'll be interesting to see
791
00:38:43,080 --> 00:38:45,840
can we get whoever, those companies,
792
00:38:45,840 --> 00:38:51,200
get the latency down to a point where
793
00:38:51,200 --> 00:38:53,840
you can actually make an API call, let's say.
794
00:38:53,840 --> 00:38:57,160
I guess there'll still be some challenges there,
795
00:38:57,160 --> 00:39:00,440
no matter what, as long as there's an API call involved.
796
00:39:00,440 --> 00:39:03,840
But if you're doing it locally, you also
797
00:39:03,840 --> 00:39:05,880
made another really good point.
798
00:39:05,880 --> 00:39:10,240
I think people tend to, it's like a new idea,
799
00:39:10,240 --> 00:39:12,240
like a maximum viable product.
800
00:39:12,240 --> 00:39:16,560
They'll use chat GPT to get a really good version of something.
801
00:39:16,560 --> 00:39:18,960
Then thinking, oh, then when we scale,
802
00:39:18,960 --> 00:39:23,960
we'll substitute it for Maestro or Llama or some other model.
803
00:39:23,960 --> 00:39:25,280
But it's not that simple.
804
00:39:25,280 --> 00:39:31,680
It's not really as simple as a plug and play.
805
00:39:31,680 --> 00:39:35,800
Yeah, so I guess along the same vein,
806
00:39:35,800 --> 00:39:38,200
what's an important question that you believe
807
00:39:38,200 --> 00:39:42,640
remains unanswered in machine learning?
808
00:39:42,640 --> 00:39:46,160
We've been in the space long enough to see what's happening
809
00:39:46,160 --> 00:39:46,920
here.
810
00:39:46,920 --> 00:39:51,800
We played with GPT-2, we played with GPT-3.
811
00:39:51,800 --> 00:39:55,480
We thought these were cool.
812
00:39:55,480 --> 00:39:58,000
We were telling our friends and family about it
813
00:39:58,000 --> 00:39:59,840
and having them try it.
814
00:39:59,840 --> 00:40:02,560
Right.
815
00:40:02,560 --> 00:40:06,680
It wasn't until the really instruction fine-tuned and chat
816
00:40:06,680 --> 00:40:08,880
GPT-stick stuff came out where it was like,
817
00:40:08,880 --> 00:40:10,200
whoa, this is really cool.
818
00:40:10,200 --> 00:40:16,880
But also, the models had gotten better at that point.
819
00:40:16,880 --> 00:40:24,760
So you just plot that stuff out on a graph.
820
00:40:24,760 --> 00:40:28,400
Like year thing was made and how good it was.
821
00:40:28,400 --> 00:40:32,680
Like the main limiting factor is the speed and cost
822
00:40:32,680 --> 00:40:35,040
of the chips running these things.
823
00:40:35,040 --> 00:40:37,120
And all indications are they get better
824
00:40:37,120 --> 00:40:41,920
if we're able to throw more computing power at them.
825
00:40:41,920 --> 00:40:44,440
So it's a weighted game.
826
00:40:44,440 --> 00:40:47,800
We're just waiting, essentially, for Moore's law,
827
00:40:47,800 --> 00:40:53,120
which happens to be an exponentially increasing
828
00:40:53,120 --> 00:40:57,360
phenomenon for these models to get better.
829
00:40:57,360 --> 00:41:02,160
So the question to me is, all right, well, when does that
830
00:41:02,160 --> 00:41:03,560
just mean we get AGI?
831
00:41:03,560 --> 00:41:06,480
I mean, this is a big question for open AI.
832
00:41:06,480 --> 00:41:08,760
Can we just continue to scale this thing up?
833
00:41:08,760 --> 00:41:11,720
And we have a model that's generally, however we
834
00:41:11,720 --> 00:41:21,080
want to define generally more capable than humanity.
835
00:41:21,080 --> 00:41:22,920
That's a big unquestioned answer for me.
836
00:41:22,920 --> 00:41:27,320
It's something I think about a lot.
837
00:41:27,320 --> 00:41:30,480
I think what's been really interesting in terms
838
00:41:30,480 --> 00:41:33,040
of unanswered or what I think will probably
839
00:41:33,040 --> 00:41:34,800
be some of the most interesting stuff
840
00:41:34,800 --> 00:41:36,000
in the next couple of years.
841
00:41:36,000 --> 00:41:39,960
Is all the multimodal work that's happening.
842
00:41:39,960 --> 00:41:44,000
So Gemini released their million token contact length,
843
00:41:44,000 --> 00:41:46,200
which means now we can just throw videos in there.
844
00:41:46,200 --> 00:41:48,400
And the stuff you can do with video is pretty cool.
845
00:41:51,360 --> 00:41:53,880
Just in my own personal usage of chat GPT,
846
00:41:53,880 --> 00:41:56,120
the image stuff has been amazing.
847
00:41:56,120 --> 00:41:58,000
Like I can take a picture of something
848
00:41:58,000 --> 00:42:03,120
I need transcribed or translated or I
849
00:42:03,120 --> 00:42:06,480
want you to count calories in my refrigerator.
850
00:42:06,480 --> 00:42:09,760
Like it's very cool what you can do just by adding imagery.
851
00:42:09,760 --> 00:42:14,160
And then if we throw audio and video, the use cases,
852
00:42:14,160 --> 00:42:18,120
and then if we make it faster to get input and output
853
00:42:18,120 --> 00:42:22,720
into that thing, the use cases are boundless.
854
00:42:22,720 --> 00:42:26,960
So I think that's a long winded way of saying
855
00:42:26,960 --> 00:42:31,960
the main problem here is just like more compute that's cheaper.
856
00:42:31,960 --> 00:42:38,160
And this is why NVIDIA stock is going to the moon.
857
00:42:38,160 --> 00:42:38,920
Through the roof.
858
00:42:38,920 --> 00:42:40,840
Yeah, absolutely.
859
00:42:40,840 --> 00:42:42,320
And it's like I saw it too.
860
00:42:42,320 --> 00:42:45,400
It's like I knew it was going to happen.
861
00:42:45,400 --> 00:42:46,760
Should have gotten deeper into that.
862
00:42:46,760 --> 00:42:55,920
Anyway, speaking of AGI, I think everyone
863
00:42:55,920 --> 00:42:57,800
has a different definition for it.
864
00:42:57,800 --> 00:43:00,960
Like slightly, I think.
865
00:43:00,960 --> 00:43:03,600
Do you feel like you have a good definition for AGI?
866
00:43:03,600 --> 00:43:04,120
Or so?
867
00:43:04,120 --> 00:43:05,920
No, I don't have a good definition.
868
00:43:05,920 --> 00:43:09,360
Well, I want to solve real science.
869
00:43:09,360 --> 00:43:13,880
Like solve some hairy problems that our best scientists
870
00:43:13,880 --> 00:43:14,920
can't solve.
871
00:43:14,920 --> 00:43:17,800
Then it's like, all right.
872
00:43:17,800 --> 00:43:19,080
Right.
873
00:43:19,080 --> 00:43:21,240
It's achievement unlocked.
874
00:43:21,240 --> 00:43:22,240
It can do it.
875
00:43:22,240 --> 00:43:23,560
So that's like what?
876
00:43:23,560 --> 00:43:27,840
Some unsolved math problems, some new protein thing.
877
00:43:27,840 --> 00:43:28,360
Well, yeah.
878
00:43:28,360 --> 00:43:33,120
People, they recently had a model like solve a proof
879
00:43:33,120 --> 00:43:34,120
that none of us could solve.
880
00:43:34,120 --> 00:43:35,680
So maybe it's here.
881
00:43:35,680 --> 00:43:39,680
Yeah, but if you look into it, they had it do it like 1,000
882
00:43:39,680 --> 00:43:40,360
times.
883
00:43:40,360 --> 00:43:42,080
And then they had mathematicians review it.
884
00:43:42,080 --> 00:43:43,240
And they found like, oh, OK.
885
00:43:43,240 --> 00:43:45,600
A handful of times this actually worked.
886
00:43:45,600 --> 00:43:46,200
I think that.
887
00:43:46,200 --> 00:43:46,600
I don't know.
888
00:43:46,600 --> 00:43:48,400
That's what I was reading about.
889
00:43:48,400 --> 00:43:49,720
But yes, it's possible.
890
00:43:49,720 --> 00:43:50,640
It's possible.
891
00:43:50,640 --> 00:43:53,200
Now, I think it's really, yeah.
892
00:43:53,200 --> 00:43:55,800
I mean, a lot of people way smarter than me.
893
00:43:55,800 --> 00:43:58,400
I've spent a lot of time trying to define this.
894
00:43:58,400 --> 00:44:00,960
So I'm not going to even attempt it.
895
00:44:00,960 --> 00:44:06,400
But it's one of those things where you probably
896
00:44:06,400 --> 00:44:08,960
know it when you see it.
897
00:44:08,960 --> 00:44:09,920
I don't know.
898
00:44:09,920 --> 00:44:15,960
I think it's going to be remarkable and scary.
899
00:44:15,960 --> 00:44:21,080
But it seems like, I'll also say,
900
00:44:21,080 --> 00:44:25,360
there's a long history of the machine learning
901
00:44:25,360 --> 00:44:27,880
world kind of over-promising and under-delivering
902
00:44:27,880 --> 00:44:29,080
when it comes to this stuff.
903
00:44:29,080 --> 00:44:33,920
So I would not be surprised if it takes us longer
904
00:44:33,920 --> 00:44:37,360
than the next generation of GPT here.
905
00:44:37,360 --> 00:44:44,720
But I do think there's a reasonable likelihood
906
00:44:44,720 --> 00:44:50,080
that in my lifetime, I get to see this, which is awesome.
907
00:44:50,080 --> 00:44:51,360
Scary.
908
00:44:51,360 --> 00:44:52,520
But I mean, like, wow.
909
00:44:52,520 --> 00:44:57,720
Like, I managed to be put on this Earth during a time
910
00:44:57,720 --> 00:45:03,960
when this evolved ape created this other thing that somehow
911
00:45:03,960 --> 00:45:04,480
surpassed.
912
00:45:04,480 --> 00:45:09,320
But it's just a very special time to be alive
913
00:45:09,320 --> 00:45:11,760
and to have the privilege to be a part of the space
914
00:45:11,760 --> 00:45:15,280
and kind of see it happen is pretty remarkable.
915
00:45:15,280 --> 00:45:16,560
Yeah, absolutely.
916
00:45:16,560 --> 00:45:22,240
It's like the most exciting time to be in machine learning.
917
00:45:22,240 --> 00:45:24,960
Changing gears a tiny bit.
918
00:45:24,960 --> 00:45:31,120
So you've been involved in two successful machine learning
919
00:45:31,120 --> 00:45:32,480
companies.
920
00:45:32,480 --> 00:45:37,080
What does it take to sort of take part
921
00:45:37,080 --> 00:45:39,640
in something like entrepreneurship in a field
922
00:45:39,640 --> 00:45:42,600
like machine learning where there's so much uncertainty?
923
00:45:42,600 --> 00:45:45,080
What are some of the lessons that you've learned?
924
00:45:45,080 --> 00:45:50,040
Well, I think lesson number one, you
925
00:45:50,040 --> 00:45:54,440
have to love what you're doing.
926
00:45:54,440 --> 00:45:56,440
And specifically with a start, it's like, well,
927
00:45:56,440 --> 00:46:02,520
you need to love the people that you're selling software to,
928
00:46:02,520 --> 00:46:06,880
the people you're solving problems for.
929
00:46:06,880 --> 00:46:14,680
And for me, machine learning, the intelligence,
930
00:46:14,680 --> 00:46:16,920
the thoughtfulness, the kinds of problems
931
00:46:16,920 --> 00:46:20,920
that can be solved with it just made it something
932
00:46:20,920 --> 00:46:22,720
that I could get very passionate about and put
933
00:46:22,720 --> 00:46:24,800
a ton of energy into.
934
00:46:24,800 --> 00:46:29,160
There's a lot of no one cares, especially in the beginning.
935
00:46:29,160 --> 00:46:30,280
Like you're building this thing.
936
00:46:30,280 --> 00:46:31,000
You think it's cool.
937
00:46:31,000 --> 00:46:31,760
You care a lot.
938
00:46:31,760 --> 00:46:32,400
You go out.
939
00:46:32,400 --> 00:46:33,480
You share it with people.
940
00:46:33,480 --> 00:46:40,000
And most people really do not care.
941
00:46:40,000 --> 00:46:45,640
So you need to have grit to push through that,
942
00:46:45,640 --> 00:46:51,880
to stay positive, to continue putting one foot in front
943
00:46:51,880 --> 00:46:53,480
of the other every day.
944
00:46:53,480 --> 00:46:58,640
I think others have given that advice just around persistence
945
00:46:58,640 --> 00:47:02,520
and being able to keep trying.
946
00:47:05,720 --> 00:47:08,840
But yeah, I guess for me, it's just like the main thing
947
00:47:08,840 --> 00:47:11,600
is you can go to a conference with your users
948
00:47:11,600 --> 00:47:12,760
and be energized.
949
00:47:12,760 --> 00:47:16,480
That would be the main piece of advice.
950
00:47:16,480 --> 00:47:18,640
Because if you don't have that, it's
951
00:47:18,640 --> 00:47:25,680
going to be really hard to keep going when you haven't necessarily
952
00:47:25,680 --> 00:47:29,800
found that product market fit or success in the space.
953
00:47:29,800 --> 00:47:30,840
Right.
954
00:47:30,840 --> 00:47:36,320
How did you know when you hit product market fit?
955
00:47:36,320 --> 00:47:37,240
Is it a feeling?
956
00:47:37,240 --> 00:47:41,080
Is it was there something that clicked where you had it,
957
00:47:41,080 --> 00:47:43,640
or was just about having a certain number of users,
958
00:47:43,640 --> 00:47:45,960
certain value that users were getting?
959
00:47:45,960 --> 00:47:48,000
I feel like that's something that's very hard.
960
00:47:48,000 --> 00:47:50,680
Like a lot of startup struggle with understanding,
961
00:47:50,680 --> 00:47:54,760
like have I reached product market fit?
962
00:47:54,760 --> 00:47:55,880
Yeah.
963
00:47:55,880 --> 00:47:58,560
Well, there's like first, just getting users.
964
00:47:58,560 --> 00:48:01,840
So that's big.
965
00:48:01,840 --> 00:48:05,960
But there's a lot of things you could do on the internet,
966
00:48:05,960 --> 00:48:10,480
especially if you have millions of VC dollars that give you
967
00:48:10,480 --> 00:48:12,920
a bunch of users that aren't necessarily
968
00:48:12,920 --> 00:48:16,480
ones that will stick around or be all that valuable.
969
00:48:16,480 --> 00:48:17,120
Right.
970
00:48:17,120 --> 00:48:24,360
And Lucas and I have always approached entrepreneurship
971
00:48:24,360 --> 00:48:27,920
like as a small business that really
972
00:48:27,920 --> 00:48:32,600
needs to earn every dollar and just make it work.
973
00:48:32,600 --> 00:48:37,840
So early on for us, it was those initial conversations
974
00:48:37,840 --> 00:48:40,840
with your very first customers where you're going to go,
975
00:48:40,840 --> 00:48:44,520
all right, we want to charge you for this software.
976
00:48:44,520 --> 00:48:47,160
You've got to come up with a price.
977
00:48:47,160 --> 00:48:51,080
It's kind of a harrowing process.
978
00:48:51,080 --> 00:48:53,920
But then to see customers actually say, yes,
979
00:48:53,920 --> 00:48:55,120
we want to pay you this.
980
00:48:55,120 --> 00:48:56,520
This is valuable.
981
00:48:56,520 --> 00:48:59,720
And seeing them continue to engage with the product.
982
00:48:59,720 --> 00:49:03,760
And it was probably like after a year of having paying
983
00:49:03,760 --> 00:49:06,680
customers and seeing that they actually renewed.
984
00:49:06,680 --> 00:49:10,400
All right, well, there's clearly something here.
985
00:49:10,400 --> 00:49:13,200
But even after getting those first couple of customers,
986
00:49:13,200 --> 00:49:16,360
it's like, we spent a lot of time with them.
987
00:49:16,360 --> 00:49:18,040
We held their hands a ton.
988
00:49:18,040 --> 00:49:19,120
Is this scalable?
989
00:49:19,120 --> 00:49:22,480
Are we going to be able to find broader market fit here?
990
00:49:22,480 --> 00:49:25,280
There's a lot of doubt in those early days.
991
00:49:25,280 --> 00:49:26,480
Right.
992
00:49:26,480 --> 00:49:28,040
Yeah.
993
00:49:28,040 --> 00:49:30,920
Yeah, so I guess it's not just about users.
994
00:49:30,920 --> 00:49:36,040
If you're creating a software product that anyone can use.
995
00:49:36,040 --> 00:49:39,920
Because users can be, you can do anything, anyone
996
00:49:39,920 --> 00:49:44,840
that's seen Silicon Valley.
997
00:49:44,840 --> 00:49:46,240
Have you watched Silicon Valley?
998
00:49:46,240 --> 00:49:47,240
Mm-hmm.
999
00:49:47,240 --> 00:49:49,840
Yeah.
1000
00:49:49,840 --> 00:49:51,800
But it's not just about getting users.
1001
00:49:51,800 --> 00:49:53,760
It's about retention.
1002
00:49:53,760 --> 00:49:55,840
And actually have them continue to use it.
1003
00:49:55,840 --> 00:49:59,360
And being able to continue to see how they're using it.
1004
00:49:59,360 --> 00:50:02,440
And yeah, pricing is always very tricky.
1005
00:50:02,440 --> 00:50:06,640
Because it can't just be like, however much they're willing
1006
00:50:06,640 --> 00:50:10,960
to pay, you actually have to equate that value to something.
1007
00:50:10,960 --> 00:50:13,120
So yeah, that must be very tricky.
1008
00:50:13,120 --> 00:50:14,120
Any other lessons from that?
1009
00:50:14,120 --> 00:50:17,440
Well, in the beginning, though, it is kind of an exercise of like,
1010
00:50:17,440 --> 00:50:19,680
how much do you want to pay?
1011
00:50:19,680 --> 00:50:20,160
Right.
1012
00:50:20,160 --> 00:50:22,000
I mean, you're trying to price this product that
1013
00:50:22,000 --> 00:50:27,160
has no precedent in the market.
1014
00:50:27,160 --> 00:50:29,040
Yeah, it's wild.
1015
00:50:29,040 --> 00:50:29,600
But it is.
1016
00:50:29,600 --> 00:50:33,200
You're kind of pulling numbers out of a hat.
1017
00:50:33,200 --> 00:50:35,160
Right.
1018
00:50:35,160 --> 00:50:36,560
I see the other piece on users.
1019
00:50:36,560 --> 00:50:40,080
Like an example, with both Waste and Biasis and CrowdFlight
1020
00:50:40,080 --> 00:50:46,320
Figure 8, we engaged a lot with the academic community.
1021
00:50:46,320 --> 00:50:50,600
And you're not monetizing that community.
1022
00:50:50,600 --> 00:50:53,200
There's like no, like you might be
1023
00:50:53,200 --> 00:50:56,200
able to get a university to pay a little bit for the software.
1024
00:50:56,200 --> 00:50:58,480
But the amount of work and pain you're
1025
00:50:58,480 --> 00:51:03,280
going to have to go through to get that done is a lot and not
1026
00:51:03,280 --> 00:51:04,400
worth it.
1027
00:51:04,400 --> 00:51:09,560
And then you might be able to get a handful of the academics
1028
00:51:09,560 --> 00:51:10,680
to pay for the software.
1029
00:51:10,680 --> 00:51:12,520
But the dollars are going to be really small.
1030
00:51:12,520 --> 00:51:17,080
And they have pretty tight budgets
1031
00:51:17,080 --> 00:51:22,040
and don't generally want to pay for software.
1032
00:51:22,040 --> 00:51:24,800
But we always would invest in that community
1033
00:51:24,800 --> 00:51:31,680
because we knew that if you're doing this work in academia,
1034
00:51:31,680 --> 00:51:34,280
eventually you're going to get a job in industry.
1035
00:51:34,280 --> 00:51:36,640
And you'll want to use the tools that
1036
00:51:36,640 --> 00:51:38,280
help you do your best work in academia
1037
00:51:38,280 --> 00:51:41,760
and hopefully bring us along.
1038
00:51:41,760 --> 00:51:44,440
But the end goal of the business is always
1039
00:51:44,440 --> 00:51:47,520
to close those larger deals with the various enterprises.
1040
00:51:47,520 --> 00:51:53,040
So you've got to be really smart about how you do that.
1041
00:51:53,040 --> 00:51:55,360
And there is some tension between,
1042
00:51:55,360 --> 00:51:58,160
all right, let's give as much of this away for free,
1043
00:51:58,160 --> 00:52:02,560
while also being able to monetize for industry.
1044
00:52:02,560 --> 00:52:05,200
Right, because the value that you get from people using
1045
00:52:05,200 --> 00:52:09,160
your software, figuring out what breaks, what doesn't break,
1046
00:52:09,160 --> 00:52:14,400
what people are getting value from, that's invaluable.
1047
00:52:14,400 --> 00:52:17,320
But you also don't want to just, you can't just give it away
1048
00:52:17,320 --> 00:52:18,160
forever.
1049
00:52:18,160 --> 00:52:20,280
At some point, it's a business.
1050
00:52:20,280 --> 00:52:24,080
There's a certain bottom line that you have to start collecting
1051
00:52:24,080 --> 00:52:26,640
some sort of fee.
1052
00:52:26,640 --> 00:52:27,680
But it's very interesting.
1053
00:52:27,680 --> 00:52:29,800
You mentioned in the beginning you
1054
00:52:29,800 --> 00:52:32,680
were doing things that were more almost consultative.
1055
00:52:32,680 --> 00:52:34,560
So when you were small, you were doing things
1056
00:52:34,560 --> 00:52:36,960
that didn't necessarily scale.
1057
00:52:36,960 --> 00:52:41,200
But did you know at the time that that was the case
1058
00:52:41,200 --> 00:52:44,760
and that in hopes that one day you'd be able to reach a point
1059
00:52:44,760 --> 00:52:47,800
where it would?
1060
00:52:47,800 --> 00:52:48,960
I mean, in the beginning, it's just
1061
00:52:48,960 --> 00:52:51,520
like you're trying to get anyone who will engage
1062
00:52:51,520 --> 00:52:52,760
to continue engaging.
1063
00:52:52,760 --> 00:52:54,120
So that was priceless.
1064
00:52:54,120 --> 00:52:56,600
Like, yes, the founders will drive down
1065
00:52:56,600 --> 00:53:02,680
to Mountain View every week to meet with the team at Toyota.
1066
00:53:02,680 --> 00:53:06,560
That's invaluable.
1067
00:53:06,560 --> 00:53:08,920
Now, we can't keep doing that forever.
1068
00:53:08,920 --> 00:53:10,880
But it was right for us to do it.
1069
00:53:10,880 --> 00:53:13,040
And I think of it less as like consultative
1070
00:53:13,040 --> 00:53:17,840
is that's something as an entrepreneur you always
1071
00:53:17,840 --> 00:53:19,360
need to be really careful with.
1072
00:53:19,360 --> 00:53:21,760
Because you don't want to make a consulting company that's
1073
00:53:21,760 --> 00:53:25,800
building bespoke things for different people
1074
00:53:25,800 --> 00:53:28,800
where there isn't a central platform or service that
1075
00:53:28,800 --> 00:53:34,640
can have the benefits of scale across many, many, many
1076
00:53:34,640 --> 00:53:35,400
different customers.
1077
00:53:35,400 --> 00:53:39,960
So we were working very closely and addressing
1078
00:53:39,960 --> 00:53:41,360
specific problems that they were having,
1079
00:53:41,360 --> 00:53:43,280
but always stepping back and saying, like, hey,
1080
00:53:43,280 --> 00:53:44,760
is this generally useful?
1081
00:53:44,760 --> 00:53:47,880
Will this also be something that someone working
1082
00:53:47,880 --> 00:53:50,320
in this other space could benefit from when
1083
00:53:50,320 --> 00:53:52,720
deciding whether or not we actually productized it
1084
00:53:52,720 --> 00:53:56,280
and put it into the product?
1085
00:53:56,280 --> 00:53:58,680
At my previous company, CrowdFlight Figure 8,
1086
00:53:58,680 --> 00:54:02,440
that was helping customers generate labeled data sets
1087
00:54:02,440 --> 00:54:04,840
for their machine learning model efforts.
1088
00:54:04,840 --> 00:54:07,800
That would often turn into actual consulting,
1089
00:54:07,800 --> 00:54:09,560
which was really hard.
1090
00:54:09,560 --> 00:54:11,320
Like, we're using our own software
1091
00:54:11,320 --> 00:54:13,760
on behalf of the customer, or we're
1092
00:54:13,760 --> 00:54:15,680
going deep into their specific use
1093
00:54:15,680 --> 00:54:17,600
cakes and helping them design.
1094
00:54:17,600 --> 00:54:22,760
And that makes for a very different business dynamic
1095
00:54:22,760 --> 00:54:25,520
than just selling a software license.
1096
00:54:25,520 --> 00:54:29,840
Yeah, I think that's because getting annotated data
1097
00:54:29,840 --> 00:54:32,360
is so much harder than people think it is.
1098
00:54:32,360 --> 00:54:36,560
Because it's not just like, oh, get good data.
1099
00:54:36,560 --> 00:54:39,280
Like what you were saying earlier, like, what does good mean?
1100
00:54:39,280 --> 00:54:41,800
Right, you need to create a set of annotation instructions.
1101
00:54:41,800 --> 00:54:43,920
You need to create the tooling around it.
1102
00:54:43,920 --> 00:54:45,760
And you actually have to know somehow
1103
00:54:45,760 --> 00:54:47,320
if you're collecting it.
1104
00:54:47,320 --> 00:54:51,800
And then often, this task, it won't even be objective.
1105
00:54:51,800 --> 00:54:53,520
There'll be some subjective nature to it,
1106
00:54:53,520 --> 00:54:56,880
and there'll be this low inter-annotator agreement.
1107
00:54:56,880 --> 00:54:59,320
So how do you even measure if you're getting good data?
1108
00:54:59,320 --> 00:55:02,520
So I'm sure there were so many challenges there.
1109
00:55:02,520 --> 00:55:04,800
But yet, such an important problem,
1110
00:55:04,800 --> 00:55:08,040
such an important thing to try to solve.
1111
00:55:08,040 --> 00:55:11,720
And way ahead of the game.
1112
00:55:11,720 --> 00:55:15,080
Like that was back in 2007, 2008.
1113
00:55:15,080 --> 00:55:19,000
I mean, thinking about the data-centric movement that's
1114
00:55:19,000 --> 00:55:20,640
taken place over the last few years,
1115
00:55:20,640 --> 00:55:24,960
like you knew that a long time ago.
1116
00:55:24,960 --> 00:55:27,440
Yeah, we were definitely too early to market
1117
00:55:27,440 --> 00:55:28,600
with that first company.
1118
00:55:28,600 --> 00:55:33,720
But we learned a ton and got to work
1119
00:55:33,720 --> 00:55:36,800
with a ton of really impressive machine learning teams
1120
00:55:36,800 --> 00:55:37,640
over the years.
1121
00:55:37,640 --> 00:55:39,800
I wouldn't take it back.
1122
00:55:39,800 --> 00:55:40,320
That's good.
1123
00:55:40,320 --> 00:55:43,000
Yeah, I mean, you get to learn about some of the problems.
1124
00:55:43,000 --> 00:55:48,080
I see how well scale has done, which started 10 years
1125
00:55:48,080 --> 00:55:51,040
after we started and think like, oh, if we had just
1126
00:55:51,040 --> 00:55:54,120
timed our go-to-market a little differently,
1127
00:55:54,120 --> 00:55:56,040
but now they're awesome.
1128
00:55:56,040 --> 00:55:58,000
They've executed it amazingly.
1129
00:55:58,000 --> 00:56:03,240
Yeah, the timing of things, there is a certain,
1130
00:56:03,240 --> 00:56:05,160
I never like using the term luck,
1131
00:56:05,160 --> 00:56:07,680
but there is a certain luck to timing,
1132
00:56:07,680 --> 00:56:09,360
especially for entrepreneurship.
1133
00:56:09,360 --> 00:56:12,960
You have to be excited in developing this thing
1134
00:56:12,960 --> 00:56:17,040
at the right time when other people are,
1135
00:56:17,040 --> 00:56:20,120
where some amount of people are ready for it at least.
1136
00:56:20,120 --> 00:56:23,000
You need to have some customer base.
1137
00:56:23,000 --> 00:56:24,960
I think that when it comes to creating
1138
00:56:24,960 --> 00:56:30,120
SaaS and a tech company, you have a team filled
1139
00:56:30,120 --> 00:56:32,440
with forward thinkers.
1140
00:56:32,440 --> 00:56:36,960
And that's not necessarily who the buyer is at a company.
1141
00:56:36,960 --> 00:56:40,240
It might not necessarily be the most forward thinker.
1142
00:56:40,240 --> 00:56:43,080
They might be a little bit more on the conservative side,
1143
00:56:43,080 --> 00:56:45,200
not willing to take certain risks.
1144
00:56:45,200 --> 00:56:47,800
And then you have to try to show them value,
1145
00:56:47,800 --> 00:56:49,120
which can be really tough.
1146
00:56:52,160 --> 00:56:54,280
Yeah, just thinking about some things
1147
00:56:54,280 --> 00:56:55,840
and the challenges of entrepreneurship.
1148
00:56:55,840 --> 00:56:57,600
But also, that's what makes it fun.
1149
00:56:57,600 --> 00:56:59,320
And then you combine it with machine learning.
1150
00:56:59,320 --> 00:57:00,720
It makes it even more fun.
1151
00:57:00,720 --> 00:57:02,040
There you go.
1152
00:57:02,040 --> 00:57:03,280
Yeah.
1153
00:57:03,280 --> 00:57:09,600
So in your career, well, first off,
1154
00:57:09,600 --> 00:57:13,480
you've had some of the best titles, I have to say.
1155
00:57:13,480 --> 00:57:16,600
Chief Awesome Officer at one point.
1156
00:57:16,600 --> 00:57:19,400
Just your name or your initials at another point.
1157
00:57:19,400 --> 00:57:21,680
Pretty cool.
1158
00:57:21,680 --> 00:57:23,080
Any other really cool ones?
1159
00:57:25,560 --> 00:57:28,280
Yeah, I think Chief Awesome Officer, I just put on LinkedIn
1160
00:57:28,280 --> 00:57:28,800
for fun.
1161
00:57:28,800 --> 00:57:31,320
Oh, OK.
1162
00:57:31,320 --> 00:57:34,360
But CAO, it's got a nice ring to it.
1163
00:57:34,360 --> 00:57:35,600
It does have a nice ring to it.
1164
00:57:35,600 --> 00:57:39,840
CVP, that's my personal favorite because it's my initials.
1165
00:57:39,840 --> 00:57:41,680
It could also be corporate vice president.
1166
00:57:41,680 --> 00:57:44,160
Yeah.
1167
00:57:44,160 --> 00:57:48,000
Yeah, titles are there.
1168
00:57:48,000 --> 00:57:48,680
It's the title.
1169
00:57:48,680 --> 00:57:54,320
I suppose I like having a C title, but it doesn't.
1170
00:57:54,320 --> 00:57:57,800
My titles co-founder really at the end of the day.
1171
00:57:57,800 --> 00:58:00,720
And that's one of the things I love most about the job
1172
00:58:00,720 --> 00:58:07,320
is that I'll get kind of brought into anything at any time
1173
00:58:07,320 --> 00:58:13,600
and can be really versatile and just try to solve problems
1174
00:58:13,600 --> 00:58:15,960
pragmatically.
1175
00:58:15,960 --> 00:58:17,520
Yeah, that's what I was going to say.
1176
00:58:17,520 --> 00:58:20,840
It's just about solving problems so they
1177
00:58:20,840 --> 00:58:23,800
get to bring you in to solve problems.
1178
00:58:23,800 --> 00:58:24,880
Fixer.
1179
00:58:24,880 --> 00:58:25,640
You're the fixer.
1180
00:58:25,640 --> 00:58:32,360
The closer, the fixer, both of them, I'll give you this one.
1181
00:58:32,360 --> 00:58:37,240
What's one piece of advice that you would give yourself
1182
00:58:37,240 --> 00:58:40,440
or you wish you received 20 years ago, 15 years ago?
1183
00:58:40,440 --> 00:58:41,880
All right, this is great.
1184
00:58:41,880 --> 00:58:44,080
Yeah, yeah, yeah, yeah.
1185
00:58:44,080 --> 00:58:44,960
Find a hobby.
1186
00:58:48,240 --> 00:58:48,800
OK.
1187
00:58:48,800 --> 00:58:51,880
I think this is something I had, like other friends had told me,
1188
00:58:51,880 --> 00:58:53,800
like, yeah, I should do this.
1189
00:58:53,800 --> 00:58:55,840
Especially as an entrepreneur, it's always just like,
1190
00:58:55,840 --> 00:58:57,240
oh, there's not a lot of time.
1191
00:58:57,240 --> 00:59:00,640
My hobby is this project.
1192
00:59:00,640 --> 00:59:04,240
And I've definitely found that there's
1193
00:59:04,240 --> 00:59:09,800
only so far that that goes before you're just kind of burnout
1194
00:59:09,800 --> 00:59:16,720
and now you're worse off than if you had just spent
1195
00:59:16,720 --> 00:59:21,400
your 10, 20 hours of free time last week doing something else
1196
00:59:21,400 --> 00:59:25,200
that you're interested in or excited about.
1197
00:59:25,200 --> 00:59:28,080
So any hobbies that you want to share?
1198
00:59:28,080 --> 00:59:29,720
Have something exciting, interesting on the side.
1199
00:59:32,880 --> 00:59:38,680
The sad part is I still don't have a great hobby.
1200
00:59:38,680 --> 00:59:41,240
So you wish that somebody gave you that advice, I guess.
1201
00:59:41,240 --> 00:59:42,440
Yeah, exactly.
1202
00:59:42,440 --> 00:59:45,400
That's like legit advice, yeah.
1203
00:59:45,400 --> 00:59:47,680
I mean, the regulars, I enjoy reading.
1204
00:59:47,680 --> 00:59:50,920
I enjoy long walks, traveling.
1205
00:59:50,920 --> 00:59:54,680
But I don't think any of those quite qualify as a hobby.
1206
00:59:54,680 --> 00:59:57,960
I'm thinking I should go to the clay studio
1207
00:59:57,960 --> 01:00:02,920
and throw some clay or go weld some metal together or something.
1208
01:00:02,920 --> 01:00:04,280
But yeah.
1209
01:00:04,280 --> 01:00:08,560
I was going to say something maybe in the art realm.
1210
01:00:08,560 --> 01:00:10,000
Yeah.
1211
01:00:10,000 --> 01:00:14,080
OK, the final and the juiciest of questions.
1212
01:00:14,080 --> 01:00:18,200
What has a career in machine learning and entrepreneurship
1213
01:00:18,200 --> 01:00:21,560
taught you about life?
1214
01:00:21,560 --> 01:00:22,240
Oh, man.
1215
01:00:26,200 --> 01:00:29,840
Well, I'd say the entrepreneurship part has taught me
1216
01:00:29,840 --> 01:00:38,360
that there's the business, there's this idea, the customer.
1217
01:00:38,360 --> 01:00:42,520
All of these things we think about when
1218
01:00:42,520 --> 01:00:43,920
we think of the kinds of problems you're
1219
01:00:43,920 --> 01:00:46,800
going to have to deal with within a company.
1220
01:00:46,800 --> 01:00:50,680
The thing that I never thought about that much,
1221
01:00:50,680 --> 01:00:54,640
but is actually what I found to be the most important,
1222
01:00:54,640 --> 01:01:01,280
is the people within the company that you're creating.
1223
01:01:01,280 --> 01:01:04,400
You're hiring a bunch of folks to work on a problem.
1224
01:01:07,920 --> 01:01:11,960
But each of those individuals is another person
1225
01:01:11,960 --> 01:01:14,920
with their own problems, own stuff going on.
1226
01:01:14,920 --> 01:01:17,800
And the only way the organization is going to be effective
1227
01:01:17,800 --> 01:01:23,240
is if the people within it feel respected and treated
1228
01:01:23,240 --> 01:01:24,600
as humans with dignity.
1229
01:01:24,600 --> 01:01:30,920
And there's not some magic formula.
1230
01:01:30,920 --> 01:01:33,320
But this is what you do, such that everyone in your company
1231
01:01:33,320 --> 01:01:36,440
will now be seen as their full and true self.
1232
01:01:36,440 --> 01:01:39,360
But I think it's something important, especially
1233
01:01:39,360 --> 01:01:41,920
as an entrepreneur, as a leader in the company to think about
1234
01:01:41,920 --> 01:01:47,040
and to try to engage with as many people in the organization
1235
01:01:47,040 --> 01:01:52,720
as human beings as possible.
1236
01:01:52,720 --> 01:01:54,320
That's definitely a lesson.
1237
01:01:54,320 --> 01:01:58,760
I think the other piece that I've
1238
01:01:58,760 --> 01:02:00,840
learned in doing this over the years
1239
01:02:00,840 --> 01:02:06,280
is that I could still find that joy, that happiness of imagining
1240
01:02:06,280 --> 01:02:11,280
how to solve a problem and going out and solving it.
1241
01:02:11,280 --> 01:02:16,120
That being a creator, that's one of the aspects
1242
01:02:16,120 --> 01:02:20,040
of entrepreneurship that I love the most.
1243
01:02:20,040 --> 01:02:23,040
And it's been just amazing, even over the last six months,
1244
01:02:23,040 --> 01:02:27,880
to go and experiment with these new language models
1245
01:02:27,880 --> 01:02:31,360
and see what kind of side projects and tools
1246
01:02:31,360 --> 01:02:32,520
that I can create.
1247
01:02:32,520 --> 01:02:36,120
And I still have that same joy I had as a teenager when I made
1248
01:02:36,120 --> 01:02:41,520
my first website, and that's been awesome.
1249
01:02:41,520 --> 01:02:44,960
And just continuing to learn and to build.
1250
01:02:44,960 --> 01:02:46,000
That's awesome.
1251
01:02:46,000 --> 01:02:46,680
I love it.
1252
01:02:46,680 --> 01:02:49,280
I love it.
1253
01:02:49,280 --> 01:02:52,320
For people that are interested in learning more about you
1254
01:02:52,320 --> 01:02:54,720
or some of the work that you're doing at Weights and Biases,
1255
01:02:54,720 --> 01:02:58,480
where would you direct them?
1256
01:02:58,480 --> 01:03:04,560
Well, WB.com has information about the product itself
1257
01:03:04,560 --> 01:03:06,120
in the company.
1258
01:03:06,120 --> 01:03:10,440
There's also really cool links to different, what we call,
1259
01:03:10,440 --> 01:03:12,880
reports in the Weights and Biases platform, which
1260
01:03:12,880 --> 01:03:15,000
can be bits of research or analysis
1261
01:03:15,000 --> 01:03:18,520
or leveraging some of the new large language model stuff
1262
01:03:18,520 --> 01:03:19,800
we were talking about today.
1263
01:03:19,800 --> 01:03:21,760
That's really good content.
1264
01:03:21,760 --> 01:03:26,120
We have a YouTube channel, and we're on Twitter, LinkedIn.
1265
01:03:26,120 --> 01:03:30,920
Those are our primary social media outlets.
1266
01:03:30,920 --> 01:03:34,840
I'm on Twitter at VanPelt.
1267
01:03:34,840 --> 01:03:35,400
Ping me.
1268
01:03:35,400 --> 01:03:37,040
Hit me up.
1269
01:03:37,040 --> 01:03:38,440
It's been a pleasure.
1270
01:03:38,440 --> 01:03:40,440
Yes, it's been absolutely fantastic.
1271
01:03:40,440 --> 01:03:42,920
I really appreciate you giving me the time.
1272
01:03:42,920 --> 01:03:44,800
Thank you so much for the incredible work
1273
01:03:44,800 --> 01:03:47,280
that you're doing at Weights and Biases.
1274
01:03:47,280 --> 01:03:50,000
Thanks for letting me pick your brain for a little bit.
1275
01:03:50,000 --> 01:03:50,360
You bet.
1276
01:03:50,360 --> 01:03:51,000
This is fun.
1277
01:03:51,000 --> 01:03:52,000
Thanks for having me.
1278
01:03:52,000 --> 01:03:57,200
Thank you for tuning in to Learning from Machine Learning.
1279
01:03:57,200 --> 01:03:59,920
On this episode, we delved into the experiences
1280
01:03:59,920 --> 01:04:04,160
of Chris Van Pelt, co-founder of Weights and Biases,
1281
01:04:04,160 --> 01:04:06,840
gaining valuable insights into the current landscape
1282
01:04:06,840 --> 01:04:08,080
of the industry.
1283
01:04:08,080 --> 01:04:11,320
Chris explained the pivotal role of Weights and Biases
1284
01:04:11,320 --> 01:04:14,720
as a powerful developer tool, enabling ML engineers
1285
01:04:14,720 --> 01:04:18,280
to navigate through the complexities of experimentation,
1286
01:04:18,280 --> 01:04:22,040
data visualization, and model improvement.
1287
01:04:22,040 --> 01:04:23,680
His candid reflections on the challenges
1288
01:04:23,680 --> 01:04:27,120
in evaluating ML models and addressing the gap between AI
1289
01:04:27,120 --> 01:04:30,240
hype and reality offered a profound understanding
1290
01:04:30,240 --> 01:04:32,120
of the field's intricacies.
1291
01:04:32,120 --> 01:04:34,680
Drawing from his entrepreneurial experiences,
1292
01:04:34,680 --> 01:04:37,120
co-founding two machine learning companies,
1293
01:04:37,120 --> 01:04:41,360
Chris leaves us with lessons in resilience, innovation,
1294
01:04:41,360 --> 01:04:44,240
and a deep appreciation for the human dimension
1295
01:04:44,240 --> 01:04:45,640
within the tech line.
1296
01:04:45,640 --> 01:04:48,360
Don't forget to subscribe and share this episode
1297
01:04:48,360 --> 01:04:50,320
with your friends and colleagues.
1298
01:04:50,320 --> 01:05:16,400
Until next time, keep on learning.