TRANSCRIÇÃO 🔗
1
00:00:00,000 --> 00:00:03,600
When I had entered into my graduate program,
2
00:00:03,600 --> 00:00:06,320
I was told that it was a waste of time
3
00:00:06,320 --> 00:00:10,000
to do like computer science and programming,
4
00:00:10,000 --> 00:00:12,560
that I would never be respected if all I did
5
00:00:12,560 --> 00:00:15,640
was like build tools or build libraries
6
00:00:15,640 --> 00:00:19,800
or write code to solve science problems.
7
00:00:24,120 --> 00:00:26,160
How did the best machine learning practitioners
8
00:00:26,160 --> 00:00:28,120
get involved in the field?
9
00:00:28,120 --> 00:00:30,600
What challenges have they faced?
10
00:00:30,600 --> 00:00:32,760
What has helped them flourish?
11
00:00:32,760 --> 00:00:34,560
Let's ask them.
12
00:00:34,560 --> 00:00:36,960
Welcome to Learning from Machine Learning.
13
00:00:36,960 --> 00:00:38,720
I'm your host, Seth Levine.
14
00:00:41,180 --> 00:00:43,520
Welcome to Learning from Machine Learning.
15
00:00:43,520 --> 00:00:46,680
On this episode, we have a very special guest,
16
00:00:46,680 --> 00:00:49,560
Paige Bailey, the lead product manager
17
00:00:49,560 --> 00:00:52,480
for generative models at Google DeepMind.
18
00:00:52,480 --> 00:00:56,720
And at such an exciting time, this week was Google I.O.
19
00:00:56,720 --> 00:01:01,200
where they got to announce new advancements for Google Bard,
20
00:01:01,200 --> 00:01:04,520
talk about Gemini, and how they're incorporating
21
00:01:04,520 --> 00:01:06,760
many aspects of generative models
22
00:01:06,760 --> 00:01:09,880
into all of their products, really.
23
00:01:09,880 --> 00:01:12,160
Paige, welcome to the show.
24
00:01:12,160 --> 00:01:12,680
Awesome.
25
00:01:12,680 --> 00:01:14,720
Thank you for having me.
26
00:01:14,720 --> 00:01:17,000
It's such a pleasure to have you.
27
00:01:17,000 --> 00:01:20,480
You have such an interesting background.
28
00:01:20,480 --> 00:01:22,600
You want to just give a little bit of background,
29
00:01:22,600 --> 00:01:24,920
introduce yourself, how you got interested in computers
30
00:01:24,920 --> 00:01:26,640
and machine learning?
31
00:01:26,640 --> 00:01:27,760
Sure, that sounds good.
32
00:01:27,760 --> 00:01:31,680
So my background, I did geophysics and applied math
33
00:01:31,680 --> 00:01:34,080
when I was in school.
34
00:01:34,080 --> 00:01:37,360
That was kind of the focus of my career.
35
00:01:37,360 --> 00:01:39,000
I think when I was younger, I wanted
36
00:01:39,000 --> 00:01:41,920
to be like some sort of lady, Carl Sagan,
37
00:01:41,920 --> 00:01:43,720
like focused on planetary science
38
00:01:43,720 --> 00:01:47,360
and sort of being able to explain
39
00:01:47,360 --> 00:01:51,280
these complicated technical topics to the world.
40
00:01:51,280 --> 00:01:54,240
I got into computers very early.
41
00:01:54,240 --> 00:01:57,360
I grew up in quite a small town.
42
00:01:57,360 --> 00:02:02,280
And my family kind of rescued an apple, too,
43
00:02:02,280 --> 00:02:03,560
from being thrown away.
44
00:02:03,560 --> 00:02:06,960
And that was how I first learned how to program.
45
00:02:06,960 --> 00:02:12,080
And then really, I think this time in particular
46
00:02:12,080 --> 00:02:15,680
is kind of what I've been waiting my entire career for.
47
00:02:15,680 --> 00:02:19,120
I've been doing machine learning for a bit over a decade.
48
00:02:19,120 --> 00:02:23,000
And previously, you would have to go through all of this pain
49
00:02:23,000 --> 00:02:25,200
to get the data, to build a model,
50
00:02:25,200 --> 00:02:27,760
to try to choose an algorithm, and then
51
00:02:27,760 --> 00:02:30,560
to do the really hard work of trying to get
52
00:02:30,560 --> 00:02:33,000
those models into production.
53
00:02:33,000 --> 00:02:35,560
And even then, those models were just kind of single task
54
00:02:35,560 --> 00:02:36,520
models.
55
00:02:36,520 --> 00:02:40,080
And today, we have these highly capable, very general purpose
56
00:02:40,080 --> 00:02:45,800
models that are doing an overwhelming number of things.
57
00:02:45,800 --> 00:02:49,360
And really, we keep discovering new ways
58
00:02:49,360 --> 00:02:53,800
that they're useful based on the way that people are sort
59
00:02:53,800 --> 00:02:58,160
of testing them out and stress testing their boundaries.
60
00:02:58,160 --> 00:03:03,000
So it's been really exhilarating, honestly.
61
00:03:03,000 --> 00:03:05,080
I would never have imagined that this
62
00:03:05,080 --> 00:03:08,240
would happen back when I started doing machine learning in 2009,
63
00:03:08,240 --> 00:03:09,720
2010.
64
00:03:09,720 --> 00:03:14,600
So back in 2009, 2010, what were the types of machine learning
65
00:03:14,600 --> 00:03:17,280
problems that you were working on?
66
00:03:17,280 --> 00:03:18,800
Oh, lord.
67
00:03:18,800 --> 00:03:23,240
So back then, I was fortunate enough to do,
68
00:03:23,240 --> 00:03:30,320
I got some NSF kind of grants to do planetary sciences research
69
00:03:30,320 --> 00:03:32,800
both at the Laboratory for Atmospheric and Space Physics
70
00:03:32,800 --> 00:03:36,760
in Boulder, as well as at Southwest Research Institute.
71
00:03:36,760 --> 00:03:38,720
And most of the machine learning there
72
00:03:38,720 --> 00:03:41,520
was just kind of fancy statistics, right?
73
00:03:41,520 --> 00:03:43,760
So you were doing linear regression,
74
00:03:43,760 --> 00:03:44,920
logistic regression.
75
00:03:44,920 --> 00:03:48,160
You might be doing decision trees
76
00:03:48,160 --> 00:03:51,040
or support vector machines.
77
00:03:51,040 --> 00:03:55,600
But really, it was just taking this very tabular data
78
00:03:55,600 --> 00:03:58,040
and attempting to make sense out of it.
79
00:03:58,040 --> 00:04:01,200
And it was useful, very, very useful,
80
00:04:01,200 --> 00:04:06,680
and capable of driving many interesting scientific advances,
81
00:04:06,680 --> 00:04:09,280
but nothing like what we have today.
82
00:04:09,280 --> 00:04:20,760
Load full transcript
And also using sometimes very, I would say, very niche tools.
83
00:04:20,760 --> 00:04:23,320
So there was, in the space sciences,
84
00:04:23,320 --> 00:04:25,560
there's something called IDL.
85
00:04:25,560 --> 00:04:30,080
There's also, I remember, many, many sleepless nights
86
00:04:30,080 --> 00:04:33,680
attempting to wrangle MATLAB into doing
87
00:04:33,680 --> 00:04:36,480
what I needed it to do, whereas Python always
88
00:04:36,480 --> 00:04:38,840
seemed to make much more sense.
89
00:04:38,840 --> 00:04:45,080
So just, again, like a sort of an explosion
90
00:04:45,080 --> 00:04:47,880
and a revolution in terms of the kinds of models
91
00:04:47,880 --> 00:04:50,520
that we can build now and then also the tools that are
92
00:04:50,520 --> 00:04:52,520
available to help build those models.
93
00:04:52,520 --> 00:04:53,400
Yeah.
94
00:04:53,400 --> 00:04:55,480
I love MATLAB, by the way.
95
00:04:55,480 --> 00:04:56,240
Oh, wow.
96
00:04:56,240 --> 00:04:57,320
Yeah.
97
00:04:57,320 --> 00:04:59,960
A lot of my undergrad work, I use MATLAB.
98
00:04:59,960 --> 00:05:02,600
And one of my first big machine learning projects,
99
00:05:02,600 --> 00:05:04,000
I inherited the project.
100
00:05:04,000 --> 00:05:05,480
And a lot of it was in MATLAB.
101
00:05:05,480 --> 00:05:08,440
So I started out doing a lot of work there.
102
00:05:08,440 --> 00:05:10,520
I think it's great.
103
00:05:10,520 --> 00:05:11,880
People don't talk about it enough.
104
00:05:11,880 --> 00:05:15,560
Python has kind of eaten it up like a lot of other things.
105
00:05:15,560 --> 00:05:17,480
But I love MATLAB.
106
00:05:17,480 --> 00:05:19,600
Yeah.
107
00:05:19,600 --> 00:05:21,920
One of my professors always used to say
108
00:05:21,920 --> 00:05:24,280
that it doesn't really matter what tool you're using.
109
00:05:24,280 --> 00:05:27,000
It doesn't matter if it's Python, a spreadsheet, MATLAB,
110
00:05:27,000 --> 00:05:32,840
SPSS data, or R, or whatever it happens to be.
111
00:05:32,840 --> 00:05:34,840
The important thing is that you're
112
00:05:34,840 --> 00:05:36,400
asking an interesting question.
113
00:05:36,400 --> 00:05:39,560
And you're being thoughtful about analyzing data.
114
00:05:39,560 --> 00:05:42,000
And the tool is just kind of something
115
00:05:42,000 --> 00:05:43,360
that gets you to the answer.
116
00:05:43,360 --> 00:05:45,400
It shouldn't be the answer itself.
117
00:05:45,400 --> 00:05:46,640
Yeah.
118
00:05:46,640 --> 00:05:47,840
Yeah, definitely.
119
00:05:47,840 --> 00:05:51,200
That's for sure.
120
00:05:51,200 --> 00:05:54,320
Yeah, I remember I did some very interesting work with SPSS
121
00:05:54,320 --> 00:05:55,840
as well.
122
00:05:55,840 --> 00:05:58,160
But yeah, it's not the tool, really.
123
00:05:58,160 --> 00:06:02,480
It's understanding the problem, understanding that,
124
00:06:02,480 --> 00:06:04,600
do you even have the right data to start
125
00:06:04,600 --> 00:06:06,520
approaching this problem?
126
00:06:06,520 --> 00:06:10,640
So yeah, so I guess at face value,
127
00:06:10,640 --> 00:06:11,880
people might not understand.
128
00:06:11,880 --> 00:06:13,320
So you're in geophysics.
129
00:06:13,320 --> 00:06:19,400
How did you become the lead product for generative models
130
00:06:19,400 --> 00:06:21,200
at Google DeepMind?
131
00:06:21,200 --> 00:06:22,560
But I mean, it makes sense.
132
00:06:22,560 --> 00:06:25,640
There's so much dealing with so much data.
133
00:06:25,640 --> 00:06:30,080
And I guess a lot of the lessons that you
134
00:06:30,080 --> 00:06:32,960
learn with how to handle all of that kind of data apply.
135
00:06:32,960 --> 00:06:36,000
But if you want to speak to that.
136
00:06:36,000 --> 00:06:36,680
Absolutely.
137
00:06:36,680 --> 00:06:40,280
So one of the cool things about geophysics
138
00:06:40,280 --> 00:06:45,720
is that geophysicists have been using GPUs for a long time.
139
00:06:45,720 --> 00:06:51,760
And they've also been kind of the flavor of Earth scientists
140
00:06:51,760 --> 00:06:57,040
that are more likely to be delighted by computers
141
00:06:57,040 --> 00:07:02,200
as opposed to running away from computers to the mountains.
142
00:07:02,200 --> 00:07:10,240
So seismic data is very massive, as well as well-logged data,
143
00:07:10,240 --> 00:07:14,080
kind of the interpretations of subsurface data.
144
00:07:14,080 --> 00:07:20,320
And all of this Earth sciences data
145
00:07:20,320 --> 00:07:23,680
really needs heavy kind of computational horsepower
146
00:07:23,680 --> 00:07:25,360
in order to analyze it.
147
00:07:25,360 --> 00:07:28,240
So even before deep learning was a thing,
148
00:07:28,240 --> 00:07:33,120
Earth scientists were already building models
149
00:07:33,120 --> 00:07:35,720
and sort of attempting to analyze these patterns
150
00:07:35,720 --> 00:07:39,480
in the subsurface and in seismic using GPUs.
151
00:07:39,480 --> 00:07:43,360
And that's something that I don't know if many people know.
152
00:07:43,360 --> 00:07:48,840
But I actually had started experimenting with CUDA,
153
00:07:48,840 --> 00:07:53,640
not just for deep learning, but because of Earth sciences
154
00:07:53,640 --> 00:07:56,640
problems and quantitative hydrogeology,
155
00:07:56,640 --> 00:08:02,440
and also understanding fluid dynamics problems.
156
00:08:02,440 --> 00:08:04,880
Yeah, that's fascinating.
157
00:08:04,880 --> 00:08:07,840
I guess having that experience with CUDA
158
00:08:07,840 --> 00:08:10,760
and dealing with that type of data,
159
00:08:10,760 --> 00:08:15,760
understanding how to process it set you up probably pretty
160
00:08:15,760 --> 00:08:19,000
well for your role with TensorFlow.
161
00:08:19,000 --> 00:08:22,400
I know that you played a pivotal role
162
00:08:22,400 --> 00:08:25,360
in the development of that.
163
00:08:25,360 --> 00:08:28,960
Can you speak to some of your work with TensorFlow?
164
00:08:28,960 --> 00:08:30,680
I don't know if it was a pivotal role
165
00:08:30,680 --> 00:08:32,280
in the development of TensorFlow,
166
00:08:32,280 --> 00:08:34,640
but I certainly was delighted.
167
00:08:34,640 --> 00:08:38,240
So TensorFlow was, I've been experimenting
168
00:08:38,240 --> 00:08:42,560
with kind of open source machine learning tools for a long time.
169
00:08:42,560 --> 00:08:44,800
I think I mentioned before, Python always
170
00:08:44,800 --> 00:08:49,080
seemed to make more sense than MATLAB.
171
00:08:49,080 --> 00:08:53,800
And there's kind of a really nice Python data science
172
00:08:53,800 --> 00:08:58,280
ecosystem, SciPy, NumPy, Matplotlib, Scikit-learn,
173
00:08:58,280 --> 00:09:02,080
like all of those great things.
174
00:09:02,080 --> 00:09:07,040
But then TensorFlow came out and was open sourced in late 2015.
175
00:09:07,040 --> 00:09:11,400
And it was kind of revolutionary in a few ways.
176
00:09:11,400 --> 00:09:12,920
It had some documentation.
177
00:09:12,920 --> 00:09:15,640
It had some examples that you could use.
178
00:09:15,640 --> 00:09:17,800
I think all of those deep dream images
179
00:09:17,800 --> 00:09:20,320
went viral over the internet.
180
00:09:20,320 --> 00:09:21,920
But it was also in Python.
181
00:09:21,920 --> 00:09:25,600
And even though it was a really kind of janky, weird sort
182
00:09:25,600 --> 00:09:30,280
of Python that you had to construct graphs to use it,
183
00:09:30,280 --> 00:09:34,720
it was still something that felt a little bit more approachable
184
00:09:34,720 --> 00:09:37,680
than maybe some of the other deep learning frameworks that
185
00:09:37,680 --> 00:09:44,440
were implemented in C++ or Lua or whatever they might have been.
186
00:09:44,440 --> 00:09:49,760
So I got very excited about it, started
187
00:09:49,760 --> 00:09:52,600
learning how to use it, trying it for my projects.
188
00:09:52,600 --> 00:09:56,400
My first deep learning project was
189
00:09:56,400 --> 00:09:59,120
applied to the Earth sciences, so understanding
190
00:09:59,120 --> 00:10:06,320
how to categorize different shapes of its holes and reefs
191
00:10:06,320 --> 00:10:11,360
and just being overwhelmed at how something would have taken
192
00:10:11,360 --> 00:10:14,200
a poor grad student six months to do.
193
00:10:14,200 --> 00:10:16,440
And then suddenly, this was able to blast through it
194
00:10:16,440 --> 00:10:19,240
in just five minutes with a really modest amount
195
00:10:19,240 --> 00:10:21,520
of training data.
196
00:10:21,520 --> 00:10:24,440
But I started contributing.
197
00:10:24,440 --> 00:10:29,040
Eventually, the TensorFlow team kind of took notice.
198
00:10:29,040 --> 00:10:35,440
And I got to join them and to work at Google Brain.
199
00:10:35,440 --> 00:10:40,200
And in addition to getting to work with the TensorFlow team,
200
00:10:40,200 --> 00:10:45,680
the JAX team, which is also an open source numerical library
201
00:10:45,680 --> 00:10:50,120
for doing deep learning in addition to many other things,
202
00:10:50,120 --> 00:10:53,480
especially in the sciences, I got to work with them quite closely.
203
00:10:53,480 --> 00:10:55,440
And then all of our machine learning frameworks
204
00:10:55,440 --> 00:10:56,920
teams at Alphabet.
205
00:10:56,920 --> 00:11:00,200
So like I said, it's very exhilarating.
206
00:11:00,200 --> 00:11:03,120
And it's been really interesting to see
207
00:11:03,120 --> 00:11:07,720
how the space has evolved over the course of the last many years.
208
00:11:07,720 --> 00:11:09,200
Yeah.
209
00:11:09,200 --> 00:11:13,040
So fast forwarding to today and some
210
00:11:13,040 --> 00:11:17,440
of your more recent projects with all the things that
211
00:11:17,440 --> 00:11:21,520
are happening with these large language models,
212
00:11:21,520 --> 00:11:24,120
can you speak to the work that you've
213
00:11:24,120 --> 00:11:26,400
been doing with applying large language models
214
00:11:26,400 --> 00:11:30,000
to different software development tasks?
215
00:11:30,000 --> 00:11:30,760
Absolutely.
216
00:11:30,760 --> 00:11:36,080
So I guess this is also a nice segue.
217
00:11:36,080 --> 00:11:39,040
So I'm boomerang back to Alphabet.
218
00:11:39,040 --> 00:11:41,760
During the pandemic, I spent a little bit over a year
219
00:11:41,760 --> 00:11:45,000
at Microsoft, specifically GitHub,
220
00:11:45,000 --> 00:11:49,880
helping with introducing machine learning features into VS code,
221
00:11:49,880 --> 00:11:54,080
GPUs in code spaces, and then also, of course, copilot.
222
00:11:54,080 --> 00:12:02,240
And so I think the potential for generative models
223
00:12:02,240 --> 00:12:05,880
in the software development space is huge.
224
00:12:05,880 --> 00:12:10,840
Historically, single task models for things
225
00:12:10,840 --> 00:12:12,960
like doc string generation or single task
226
00:12:12,960 --> 00:12:16,720
models for code generation or for code completions,
227
00:12:16,720 --> 00:12:19,480
single task models for build repair
228
00:12:19,480 --> 00:12:22,440
or for helping resolve or identify errors.
229
00:12:22,440 --> 00:12:24,800
And now we're seeing models do all of these things
230
00:12:24,800 --> 00:12:29,800
and even more with just a singular model.
231
00:12:29,800 --> 00:12:32,720
One of the coolest things about the announcements
232
00:12:32,720 --> 00:12:35,600
that we had at I.O. this past week
233
00:12:35,600 --> 00:12:39,600
is that they're all using kind of our latest large language
234
00:12:39,600 --> 00:12:41,680
model, the technical paper is out.
235
00:12:41,680 --> 00:12:43,840
I encourage everyone to read it.
236
00:12:43,840 --> 00:12:48,320
We also have a website at g.co. slash Palm2.
237
00:12:48,320 --> 00:12:53,040
But this model, based on kind of the way
238
00:12:53,040 --> 00:12:57,840
that it was trained in it and the input data,
239
00:12:57,840 --> 00:13:02,080
it's capable of doing a broad variety of software development
240
00:13:02,080 --> 00:13:02,800
tasks.
241
00:13:02,800 --> 00:13:06,560
And it's supporting code generation, code explanation,
242
00:13:06,560 --> 00:13:11,360
error explanation, error fixing across so many alphabet
243
00:13:11,360 --> 00:13:11,960
products.
244
00:13:11,960 --> 00:13:16,000
So both are tools within Google Cloud.
245
00:13:16,000 --> 00:13:19,840
The product is called Duet, as well as
246
00:13:19,840 --> 00:13:23,120
all of those features within Google Colab, which is,
247
00:13:23,120 --> 00:13:25,360
if folks aren't familiar, it's kind of a data science
248
00:13:25,360 --> 00:13:28,920
notebook environment that's ready to use
249
00:13:28,920 --> 00:13:32,360
and that you can kind of have handy in your Google Drive
250
00:13:32,360 --> 00:13:37,600
instance, as well as Android Studio, as well as code,
251
00:13:37,600 --> 00:13:39,480
features, and BARD.
252
00:13:39,480 --> 00:13:42,560
But it's all powered by the same model.
253
00:13:42,560 --> 00:13:46,200
And it's been really energizing to see
254
00:13:46,200 --> 00:13:49,280
how people have been testing it out and using it.
255
00:13:49,280 --> 00:13:52,480
And then also the features that we have coming down the pipes,
256
00:13:52,480 --> 00:13:57,680
so things like self-healing code, and then also
257
00:13:57,680 --> 00:13:58,960
things like tool use.
258
00:13:58,960 --> 00:14:05,640
So being able to access a Python interpreter with code.
259
00:14:05,640 --> 00:14:08,440
Yeah, so I'm familiar with a lot of those tools.
260
00:14:08,440 --> 00:14:12,680
I think I was an early adopter for Google Colab.
261
00:14:12,680 --> 00:14:15,040
Yeah, I've loved it for so long, the ability
262
00:14:15,040 --> 00:14:18,520
to have free access to GPUs.
263
00:14:18,520 --> 00:14:20,640
In the past, it was a little bit longer access.
264
00:14:20,640 --> 00:14:23,080
Now it's a little bit less, but that's OK.
265
00:14:23,080 --> 00:14:26,520
Just being able to, any practitioner getting access
266
00:14:26,520 --> 00:14:30,560
to a GPU is just like, yeah, it just
267
00:14:30,560 --> 00:14:32,800
changes your iteration speed.
268
00:14:32,800 --> 00:14:35,960
And you can kind of work so much faster.
269
00:14:35,960 --> 00:14:39,040
For the other things, I don't know why,
270
00:14:39,040 --> 00:14:42,520
but I was more of a late adopter for Copilot.
271
00:14:42,520 --> 00:14:47,240
I got it within the last month, it's embarrassing to say.
272
00:14:47,240 --> 00:14:50,800
But that's when I finally got around to trying it.
273
00:14:50,800 --> 00:14:52,280
I don't know, I was hesitant.
274
00:14:52,280 --> 00:14:54,360
I just thought, oh, maybe it's going
275
00:14:54,360 --> 00:14:57,400
to introduce bugs or something like that,
276
00:14:57,400 --> 00:15:00,280
or maybe it was my pride or something.
277
00:15:00,280 --> 00:15:05,040
I wanted to just continue to be coding on my own.
278
00:15:05,040 --> 00:15:08,480
But I started to use ChatGPT, I started
279
00:15:08,480 --> 00:15:12,600
to use Bard to help me
with certain things.
280
00:15:12,600 --> 00:15:14,400
I mean, it's not like I wasn't finding myself
281
00:15:14,400 --> 00:15:17,640
on Stack Overflow like every other programmer looking
282
00:15:17,640 --> 00:15:19,480
at things.
283
00:15:19,480 --> 00:15:21,560
But yeah, it's incredible.
284
00:15:21,560 --> 00:15:26,560
I think that this paradigm shift that's taking place,
285
00:15:26,560 --> 00:15:28,160
I mean, with machine learning in general,
286
00:15:28,160 --> 00:15:32,280
but for generating code, it's like what it used to be
287
00:15:32,280 --> 00:15:34,640
is you used to write a function, and then you
288
00:15:34,640 --> 00:15:37,240
would struggle to write your documentation
289
00:15:37,240 --> 00:15:38,960
or your comments and things.
290
00:15:38,960 --> 00:15:43,960
Now with this new technology to do code generation,
291
00:15:43,960 --> 00:15:46,720
you're writing what you want the code to do,
292
00:15:46,720 --> 00:15:49,200
and then it's making these suggestions
293
00:15:49,200 --> 00:15:50,560
for what the code should be.
294
00:15:50,560 --> 00:15:53,680
And multiple lines also, which is really cool.
295
00:15:53,680 --> 00:15:57,000
And it's not just trying to autocomplete,
296
00:15:57,000 --> 00:16:00,480
it's very smart.
297
00:16:00,480 --> 00:16:03,720
At least it appears to be very smart.
298
00:16:03,720 --> 00:16:07,600
One of the questions that I have is, so yeah,
299
00:16:07,600 --> 00:16:11,440
there's a lot of code out there, just like there's
300
00:16:11,440 --> 00:16:13,880
a lot of information out there.
301
00:16:13,880 --> 00:16:19,560
GitHub is filled with so many libraries,
302
00:16:19,560 --> 00:16:23,960
but not all of it is battle tested.
303
00:16:23,960 --> 00:16:26,920
Not all of it is peer reviewed.
304
00:16:26,920 --> 00:16:29,040
Code goes through code review, so it might not all
305
00:16:29,040 --> 00:16:31,400
be the highest quality.
306
00:16:31,400 --> 00:16:35,360
Or it's from, I mean, the way that things are moving
307
00:16:35,360 --> 00:16:38,520
these days, it's just from a year ago or two years ago,
308
00:16:38,520 --> 00:16:43,080
and it's using a different version of a library that
309
00:16:43,080 --> 00:16:46,960
has a different dependency or whatever the actual specific is.
310
00:16:46,960 --> 00:16:50,400
But I guess, how do you mitigate those sorts of problems?
311
00:16:50,400 --> 00:16:51,960
I know this is like a loaded question,
312
00:16:51,960 --> 00:16:54,120
but how do you mitigate those sorts of problems
313
00:16:54,120 --> 00:16:56,440
where you might be training on data that's
314
00:16:56,440 --> 00:17:00,640
either not the highest quality or out of date?
315
00:17:00,640 --> 00:17:02,800
Yeah, that's a great question.
316
00:17:02,800 --> 00:17:08,840
So we have a collection of fine tuned versions
317
00:17:08,840 --> 00:17:15,800
of our large models for internal use only.
318
00:17:15,800 --> 00:17:18,040
So kind of supporting the software engineers
319
00:17:18,040 --> 00:17:20,600
within Google who are building the software that
320
00:17:20,600 --> 00:17:23,280
powers all of the Alphabet products.
321
00:17:23,280 --> 00:17:27,040
And these fine tuned models are fine tuned on Google 3 internal
322
00:17:27,040 --> 00:17:34,400
source code, which is many, many tokens of super high quality
323
00:17:34,400 --> 00:17:40,360
peer reviewed data that is the equivalent of an L5
324
00:17:40,360 --> 00:17:42,360
suite or more.
325
00:17:42,360 --> 00:17:44,320
And that moves the needle quite a bit
326
00:17:44,320 --> 00:17:46,920
in terms of making sure that the recommendations that we're
327
00:17:46,920 --> 00:17:52,720
giving, the explanations that we're giving are pretty solid.
328
00:17:52,720 --> 00:17:58,760
For the GitHub code, I can certainly attest.
329
00:17:58,760 --> 00:18:01,360
There's a lot of very low quality code on GitHub.
330
00:18:01,360 --> 00:18:02,920
A lot of it doesn't run.
331
00:18:02,920 --> 00:18:06,600
A lot of it is using dated APIs or maybe
332
00:18:06,600 --> 00:18:08,920
have this very low process of somebody just
333
00:18:08,920 --> 00:18:12,640
push committed it to the repo without going
334
00:18:12,640 --> 00:18:16,840
through any evaluation from a peer.
335
00:18:16,840 --> 00:18:21,560
And so I think that if you are building these kinds of models
336
00:18:21,560 --> 00:18:24,200
externally, there's a lot of work
337
00:18:24,200 --> 00:18:28,120
that needs to go into kind of carefully curating and cleaning
338
00:18:28,120 --> 00:18:31,680
the GitHub data sets in order to make it solid for use.
339
00:18:31,680 --> 00:18:34,760
And if people are curious about this topic,
340
00:18:34,760 --> 00:18:37,160
I strongly, strongly recommend taking a look.
341
00:18:37,160 --> 00:18:41,160
There's a recent paper from the Hugging Face and Service Now
342
00:18:41,160 --> 00:18:43,600
team called Star Coder.
343
00:18:43,600 --> 00:18:47,160
And they go through kind of with their big code data set
344
00:18:47,160 --> 00:18:50,120
like all of the things that they needed
345
00:18:50,120 --> 00:18:54,000
to do in order to get a higher quality data set to train.
346
00:18:54,000 --> 00:18:57,600
And it includes things like deduplication.
347
00:18:57,600 --> 00:19:00,280
I think they also were considering preferentially
348
00:19:00,280 --> 00:19:03,360
waiting code that is newer or code that
349
00:19:03,360 --> 00:19:06,480
comes from repos that follow software engineering best
350
00:19:06,480 --> 00:19:10,440
practices as opposed to code that might be from like a Python
351
00:19:10,440 --> 00:19:12,560
101 student.
352
00:19:12,560 --> 00:19:17,560
And all of those kind of careful bits of attention
353
00:19:17,560 --> 00:19:22,680
to the pre-training data set move the needle significantly
354
00:19:22,680 --> 00:19:25,600
in terms of model performance down the line.
355
00:19:25,600 --> 00:19:28,040
But those things, there are also other tricks
356
00:19:28,040 --> 00:19:31,640
that you can do like retrieval techniques
357
00:19:31,640 --> 00:19:37,440
or being able to do these kind of self-healing operations
358
00:19:37,440 --> 00:19:40,040
where you recursively apply the model to the output code
359
00:19:40,040 --> 00:19:42,960
to see if it would actually run and then like fixing anything
360
00:19:42,960 --> 00:19:46,520
that might be wrong or spotting any security vulnerabilities
361
00:19:46,520 --> 00:19:49,480
if there are any in the output code.
362
00:19:49,480 --> 00:19:52,840
But it's certainly like an ongoing field
363
00:19:52,840 --> 00:19:54,760
of research in order to understand
364
00:19:54,760 --> 00:19:58,920
what the best options might be.
365
00:19:58,920 --> 00:19:59,960
Right.
366
00:19:59,960 --> 00:20:05,520
So yeah, so I guess designing, I don't know what to call it.
367
00:20:05,520 --> 00:20:09,520
I guess systems that help software engineers
368
00:20:09,520 --> 00:20:13,360
or machine learning practitioners sort of do their job,
369
00:20:13,360 --> 00:20:17,720
generate code, test code, help you write documentation,
370
00:20:17,720 --> 00:20:20,360
do all these things.
371
00:20:20,360 --> 00:20:25,320
I guess other than dealing with maybe outdated code
372
00:20:25,320 --> 00:20:27,320
are the things that we were just talking about.
373
00:20:27,320 --> 00:20:29,280
Are there any other challenges that you
374
00:20:29,280 --> 00:20:31,800
face when trying to design those systems?
375
00:20:31,800 --> 00:20:34,440
Oh, absolutely.
376
00:20:34,440 --> 00:20:38,160
And I'm sure I'm going to list a few, but there are many more.
377
00:20:38,160 --> 00:20:45,120
And I think people are discovering even more every day.
378
00:20:45,120 --> 00:20:51,880
So one is likelihood of reciting code.
379
00:20:51,880 --> 00:20:54,880
There's something that we've implemented for BARD
380
00:20:54,880 --> 00:20:57,800
where if you generate code and a portion of it
381
00:20:57,800 --> 00:21:00,440
is verbatim identical to something that
382
00:21:00,440 --> 00:21:03,160
might be within a GitHub repo, we point you
383
00:21:03,160 --> 00:21:05,240
towards the GitHub repo and then also tell you
384
00:21:05,240 --> 00:21:06,600
what license it was under.
385
00:21:06,600 --> 00:21:08,760
So whether it was Apache 2 or MIT,
386
00:21:08,760 --> 00:21:11,520
which are very permissive, versus GPL,
387
00:21:11,520 --> 00:21:13,640
which is not permissive at all and is something
388
00:21:13,640 --> 00:21:16,600
that you would probably, you would not certainly
389
00:21:16,600 --> 00:21:19,520
want to use for your business.
390
00:21:19,520 --> 00:21:24,880
I think there are also questions about security vulnerabilities
391
00:21:24,880 --> 00:21:26,920
and performance.
392
00:21:26,920 --> 00:21:28,680
So if you generate code, ideally you
393
00:21:28,680 --> 00:21:30,720
would want it to be efficient.
394
00:21:30,720 --> 00:21:32,840
You wouldn't want it to be something
395
00:21:32,840 --> 00:21:36,360
that would take 10 or 100 times longer, more compute
396
00:21:36,360 --> 00:21:38,880
in order to execute.
397
00:21:38,880 --> 00:21:40,680
And you would also hopefully want
398
00:21:40,680 --> 00:21:44,320
the code that you generate to be consistent in terms
399
00:21:44,320 --> 00:21:46,680
of syntax and conventions with the code
400
00:21:46,680 --> 00:21:49,360
in your existing code bases.
401
00:21:49,360 --> 00:21:53,720
So all of these things are considerations
402
00:21:53,720 --> 00:21:58,880
that folks have to think about when implementing tools
403
00:21:58,880 --> 00:22:02,160
for their own users.
404
00:22:02,160 --> 00:22:05,000
And it's something that we certainly
405
00:22:05,000 --> 00:22:10,480
think very deeply about when orchestrating ML applied
406
00:22:10,480 --> 00:22:14,240
to software systems internally at Alphabet.
407
00:22:14,240 --> 00:22:15,960
Right.
408
00:22:15,960 --> 00:22:19,400
So for Bard, this week you guys announced
409
00:22:19,400 --> 00:22:21,840
that the underlying model, I guess,
410
00:22:21,840 --> 00:22:28,600
was upgraded from POM1 to POM2.
411
00:22:28,600 --> 00:22:31,080
What is it you think about POM2 that makes it
412
00:22:31,080 --> 00:22:33,000
like a better base model?
413
00:22:33,000 --> 00:22:34,560
Is it more data?
414
00:22:34,560 --> 00:22:40,440
Is it whatever you can speak to about it?
415
00:22:40,440 --> 00:22:41,520
That's a great question.
416
00:22:41,520 --> 00:22:43,560
So the model that we upgraded from,
417
00:22:43,560 --> 00:22:45,880
it wasn't actually POMv1.
418
00:22:45,880 --> 00:22:51,560
And POMv1 was a very, very large model
419
00:22:51,560 --> 00:22:56,680
that there's a paper about it so folks can go
420
00:22:56,680 --> 00:22:58,640
read if they're curious.
421
00:22:58,640 --> 00:23:04,680
But it was not ever used in production for Alphabet,
422
00:23:04,680 --> 00:23:06,320
I don't believe.
423
00:23:06,320 --> 00:23:13,920
But the first model for Bard was a version of our Lambda model.
424
00:23:13,920 --> 00:23:16,400
The model that we have upgraded to
425
00:23:16,400 --> 00:23:22,160
is a version of POMv2, which was announced on Wednesday.
426
00:23:22,160 --> 00:23:24,320
And some of the capabilities of it
427
00:23:24,320 --> 00:23:27,960
are that it's much, much better at code, at math, at reasoning.
428
00:23:27,960 --> 00:23:31,280
It's also much better at multilingual tasks.
429
00:23:31,280 --> 00:23:36,400
So POMv2 was trained on over 100 spoken word languages,
430
00:23:36,400 --> 00:23:39,800
dozens and dozens of computer programming languages.
431
00:23:39,800 --> 00:23:44,480
And as a result of that really robust and very diverse
432
00:23:44,480 --> 00:23:46,480
pre-training data mixture, it can
433
00:23:46,480 --> 00:23:49,240
do things like translate from one language to another.
434
00:23:49,240 --> 00:23:54,000
It can explain idioms and riddles.
435
00:23:54,000 --> 00:23:55,840
Even in different languages, it can
436
00:23:55,840 --> 00:23:58,560
translate from one programming language to another.
437
00:23:58,560 --> 00:24:02,720
It can tell you if code might be vulnerable
438
00:24:02,720 --> 00:24:04,720
or if it needs performance fixes.
439
00:24:04,720 --> 00:24:05,960
It can generate code.
440
00:24:05,960 --> 00:24:06,960
It can explain code.
441
00:24:12,360 --> 00:24:15,400
It can write mathematical proofs.
442
00:24:15,400 --> 00:24:17,440
It's just lots and lots of different things.
443
00:24:17,440 --> 00:24:20,480
We're discovering new uses for it every day.
444
00:24:20,480 --> 00:24:24,840
And one of the other most compelling features
445
00:24:24,840 --> 00:24:27,360
about this POMv2 family of models
446
00:24:27,360 --> 00:24:30,840
is that it comes in a broad variety of sizes.
447
00:24:30,840 --> 00:24:34,800
So everything from our smallest version, which
448
00:24:34,800 --> 00:24:37,640
can fit directly on a mobile device,
449
00:24:37,640 --> 00:24:43,120
to more modest sized versions that are still,
450
00:24:43,120 --> 00:24:45,320
despite being an order of magnitude
451
00:24:45,320 --> 00:24:48,760
or more tinier than the largest version,
452
00:24:48,760 --> 00:24:51,880
still preserving all of the capabilities
453
00:24:51,880 --> 00:25:00,320
and doing it just faster, more efficient, and cheaper.
454
00:25:00,320 --> 00:25:02,840
So it's definitely, from a business perspective,
455
00:25:02,840 --> 00:25:06,160
the POMv2 family makes a lot of sense.
456
00:25:06,160 --> 00:25:08,040
Yeah, that makes sense.
457
00:25:08,040 --> 00:25:10,440
Just to dig into the idea, because this
458
00:25:10,440 --> 00:25:12,960
is something that you see across a lot of large language
459
00:25:12,960 --> 00:25:20,120
models, you see Llama that has four maybe different sizes
460
00:25:20,120 --> 00:25:24,960
and obviously POM as well.
461
00:25:24,960 --> 00:25:26,840
So the reason for the different sizes,
462
00:25:26,840 --> 00:25:28,240
the different number of parameters,
463
00:25:28,240 --> 00:25:30,680
is it just to deal with the different trade-offs
464
00:25:30,680 --> 00:25:33,960
of where you're running the model, what your trade-off is
465
00:25:33,960 --> 00:25:35,920
for latency and performance?
466
00:25:35,920 --> 00:25:37,560
Is there anything else that goes into it?
467
00:25:41,320 --> 00:25:43,000
That's a great synopsis.
468
00:25:43,000 --> 00:25:46,640
So the smaller versions of the models
469
00:25:46,640 --> 00:25:49,760
make it easier to serve in a broad variety of locations.
470
00:25:49,760 --> 00:25:54,640
It also makes them much, much quicker at inference.
471
00:25:54,640 --> 00:25:57,400
And then the kinds of capabilities
472
00:25:57,400 --> 00:25:59,760
that we've been seeing from these smaller models that
473
00:25:59,760 --> 00:26:03,880
were open sourced is you can have very, very modest sized
474
00:26:03,880 --> 00:26:04,920
models.
475
00:26:04,920 --> 00:26:07,640
And as long as you
fine tune or instruction
476
00:26:07,640 --> 00:26:10,760
tune on very high quality data sets,
477
00:26:10,760 --> 00:26:13,440
you're capable of getting the exact same performance
478
00:26:13,440 --> 00:26:15,740
that you would from a much larger model,
479
00:26:15,740 --> 00:26:20,160
despite it being faster, cheaper, easier to deploy.
480
00:26:20,160 --> 00:26:24,680
And I personally, I think that's the most interesting field
481
00:26:24,680 --> 00:26:27,120
of research right now is trying to take
482
00:26:27,120 --> 00:26:33,360
these highly capable models and make them more accessible
483
00:26:33,360 --> 00:26:37,200
for people all over the world, even if the only device
484
00:26:37,200 --> 00:26:40,200
that they have to work with is a mobile device.
485
00:26:40,200 --> 00:26:44,600
Yeah, that's something that I'm also really interested in.
486
00:26:44,600 --> 00:26:46,880
Training like teacher-student models
487
00:26:46,880 --> 00:26:50,880
and distilling information from these large language models
488
00:26:50,880 --> 00:26:55,080
to get them to be stored into smaller systems.
489
00:26:55,080 --> 00:26:58,520
Because yeah, it really depends on your use case.
490
00:26:58,520 --> 00:27:00,320
Sometimes there's just something really nice
491
00:27:00,320 --> 00:27:02,640
about being able to run it on your laptop.
492
00:27:02,640 --> 00:27:05,720
And a lot, I mean, almost all of these models,
493
00:27:05,720 --> 00:27:08,160
it's just like it's not even possible.
494
00:27:08,160 --> 00:27:10,360
You have to access it through an API
495
00:27:10,360 --> 00:27:14,360
or you have to access it through the interface
496
00:27:14,360 --> 00:27:15,520
that the company offers it.
497
00:27:15,520 --> 00:27:18,000
But yeah, I think that there's something really nice.
498
00:27:18,000 --> 00:27:21,080
Like even, I mean, I'm doing some of my work.
499
00:27:21,080 --> 00:27:23,720
Like I'm trying to get things that are like 400 megs down
500
00:27:23,720 --> 00:27:26,720
to 40 megs, just because it gives you the ability
501
00:27:26,720 --> 00:27:31,280
to like run maybe five or 10 times the amount of things
502
00:27:31,280 --> 00:27:35,360
in a much faster iteration time.
503
00:27:38,000 --> 00:27:41,640
So there's been some like dropping of some ideas
504
00:27:41,640 --> 00:27:45,000
about the new release, the new model
505
00:27:45,000 --> 00:27:48,200
that I believe it's DeepMind with Gemini.
506
00:27:49,440 --> 00:27:51,880
What can you tell us about Gemini?
507
00:27:52,880 --> 00:27:57,680
So many of the people from the Palm B2 team
508
00:27:57,680 --> 00:28:00,000
are on the Gemini project, including myself.
509
00:28:00,980 --> 00:28:04,040
And we're still actively training the model.
510
00:28:04,040 --> 00:28:09,040
It's intended to be Alphabet's most compelling model,
511
00:28:09,040 --> 00:28:14,040
which kind of tracks with, every time we build a model,
512
00:28:15,040 --> 00:28:19,000
we hope that it is a super set
of the capabilities
513
00:28:19,000 --> 00:28:20,400
of the models that preceded it.
514
00:28:21,360 --> 00:28:24,560
But Gemini is very special in that it was built
515
00:28:24,560 --> 00:28:27,760
from kind of the ground up to be multimodal.
516
00:28:27,760 --> 00:28:30,840
So we're already seeing multimodal features
517
00:28:30,840 --> 00:28:35,200
in the first versions of the model that we've trained.
518
00:28:35,200 --> 00:28:40,200
And we're anticipating that it will be in production
519
00:28:40,200 --> 00:28:42,720
very quickly, or at least the smaller versions of it
520
00:28:42,720 --> 00:28:45,240
will be in production quite quickly.
521
00:28:45,240 --> 00:28:50,240
But I guess the only thing that I can say specifically
522
00:28:51,120 --> 00:28:53,400
is stay tuned.
523
00:28:53,400 --> 00:28:54,440
We're very excited.
524
00:28:54,440 --> 00:28:56,440
It's the first model that Google DeepMind
525
00:28:56,440 --> 00:28:58,600
has trained kind of jointly together.
526
00:28:59,600 --> 00:29:03,320
And it should be particularly compelling
527
00:29:03,320 --> 00:29:06,480
for not just the text and the code use cases,
528
00:29:06,480 --> 00:29:09,040
but also the multimodal use cases.
529
00:29:09,040 --> 00:29:11,560
Yeah, that's very exciting.
530
00:29:11,560 --> 00:29:13,120
I'm looking forward to it.
531
00:29:14,000 --> 00:29:16,160
In regards to multimodal,
532
00:29:16,160 --> 00:29:20,000
what multimodal use cases are you most excited for?
533
00:29:21,120 --> 00:29:26,120
Yeah, so multimodal, I love this idea
534
00:29:26,800 --> 00:29:31,800
that you can have audio, video, images, text, code,
535
00:29:31,800 --> 00:29:36,800
as inputs, including many of them
536
00:29:36,800 --> 00:29:39,320
kind of being interspersed together.
537
00:29:39,320 --> 00:29:43,600
And then kind of define what your output should be,
538
00:29:43,600 --> 00:29:47,600
either by, you might get some text as output
539
00:29:47,600 --> 00:29:48,760
from your original model,
540
00:29:48,760 --> 00:29:50,720
and then you stack a diffusion model on top
541
00:29:50,720 --> 00:29:53,120
such that you can generate an image back out,
542
00:29:53,120 --> 00:29:56,880
or you can generate audio or video.
543
00:29:56,880 --> 00:30:00,240
But some of the use cases that I'm most excited about
544
00:30:00,240 --> 00:30:04,240
for multimodal models is that you can imagine,
545
00:30:04,920 --> 00:30:07,760
say you're taking a physics course,
546
00:30:07,760 --> 00:30:11,760
and you just can't nail a concept.
547
00:30:11,760 --> 00:30:14,480
Like you just, for whatever reason,
548
00:30:14,480 --> 00:30:17,400
like angular velocity just isn't clicking.
549
00:30:17,400 --> 00:30:22,400
And you could easily just ask, like, hey,
550
00:30:22,640 --> 00:30:25,640
I really, what I would love to have is,
551
00:30:25,640 --> 00:30:30,640
like a video that explains angular velocity to me.
552
00:30:30,920 --> 00:30:34,920
And I really want it to just have
like cats.
553
00:30:34,920 --> 00:30:38,480
And then I also want you to have like a quiz
554
00:30:38,480 --> 00:30:42,480
to check my comprehension every like one minute of the video.
555
00:30:42,480 --> 00:30:45,920
I don't want this video to be longer than four minutes.
556
00:30:45,920 --> 00:30:49,920
And then I also want to have like an outline at the end
557
00:30:49,920 --> 00:30:53,920
that explains what the concepts were in the video.
558
00:30:53,920 --> 00:30:55,720
Also with images.
559
00:30:55,720 --> 00:30:57,760
And that's something that a multimodal model
560
00:30:57,760 --> 00:30:59,320
would be able to support.
561
00:31:00,160 --> 00:31:04,680
So just being able to do something like that is huge.
562
00:31:05,560 --> 00:31:10,560
And it also brings about this world of,
563
00:31:10,840 --> 00:31:15,840
super, super tailored custom materials for folks.
564
00:31:16,640 --> 00:31:19,480
Like you could imagine each kid gets
565
00:31:19,480 --> 00:31:22,040
a new bedtime story every night.
566
00:31:22,040 --> 00:31:25,360
That's complete with, you know, a new story,
567
00:31:25,360 --> 00:31:27,640
a new adventure and new images.
568
00:31:27,640 --> 00:31:30,400
It's like the, I'm not sure if you've ever read
569
00:31:30,400 --> 00:31:32,640
The Diamond Age by Neil Stevenson,
570
00:31:32,640 --> 00:31:35,560
but it's like having the primer just kind of like
571
00:31:35,560 --> 00:31:38,440
available for every person.
572
00:31:38,440 --> 00:31:41,320
And that's, it's really energizing.
573
00:31:41,320 --> 00:31:42,320
That's awesome.
574
00:31:42,320 --> 00:31:45,120
Yeah, I think that that's such a cool use case
575
00:31:45,120 --> 00:31:48,360
because, yeah, everyone's unique.
576
00:31:48,360 --> 00:31:50,560
Everyone learns in their own, in their own ways.
577
00:31:50,560 --> 00:31:55,560
People respond to different mediums, you know, differently.
578
00:31:55,560 --> 00:31:59,560
So being able to tailor the learning approach,
579
00:31:59,560 --> 00:32:01,560
that, yeah, that could change,
580
00:32:01,560 --> 00:32:04,080
that can really change the way that we learn,
581
00:32:04,080 --> 00:32:05,400
the way that we interact,
582
00:32:05,400 --> 00:32:08,600
the way that we interact with the different, you know,
583
00:32:08,600 --> 00:32:09,960
material that we're trying to learn.
584
00:32:09,960 --> 00:32:11,560
That's really cool.
585
00:32:11,560 --> 00:32:15,000
Yeah, I saw a tweet the other day
586
00:32:15,000 --> 00:32:18,360
that someone with the, with the,
587
00:32:18,360 --> 00:32:21,760
that someone with the, with the learning disability,
588
00:32:21,760 --> 00:32:25,720
they didn't share what their learning disability was,
589
00:32:25,720 --> 00:32:29,200
but they mentioned that interacting
590
00:32:29,200 --> 00:32:34,200
with these generative models has sort of, you know,
591
00:32:35,120 --> 00:32:37,400
made it that their learning disability
592
00:32:37,400 --> 00:32:40,480
isn't hindering their life as much anymore.
593
00:32:40,480 --> 00:32:43,480
You know, they're able to, you know,
594
00:32:43,480 --> 00:32:48,000
instead of having to attempt to digest all of this content,
595
00:32:48,000 --> 00:32:52,160
even though it's not architected in a way that is,
596
00:32:52,160 --> 00:32:55,600
that's optimal for their learning style,
597
00:32:55,600 --> 00:32:57,960
they're capable of working with generative models
598
00:32:57,960 --> 00:33:00,600
to consume the information in a different way.
599
00:33:00,600 --> 00:33:03,760
And it clicks, and it's the first time in their life
600
00:33:03,760 --> 00:33:06,280
that they had said that it felt like that.
601
00:33:06,280 --> 00:33:08,080
And that's huge, right?
602
00:33:08,080 --> 00:33:12,600
Like being able to, being able to, you know,
603
00:33:12,600 --> 00:33:14,560
unlock the joy of learning for people
604
00:33:14,560 --> 00:33:16,720
who had previously been frustrating.
605
00:33:16,720 --> 00:33:19,880
That's opening up the world to so many more creators
606
00:33:19,880 --> 00:33:22,240
and so many more potential engineers
607
00:33:22,240 --> 00:33:25,800
and folks that, you know, can contribute.
608
00:33:26,760 --> 00:33:27,960
Yeah, absolutely.
609
00:33:27,960 --> 00:33:30,200
That's so rewarding.
610
00:33:30,200 --> 00:33:32,160
It's so nice to hear, you know,
611
00:33:32,160 --> 00:33:35,760
especially now with all of the people that are sort of
612
00:33:35,760 --> 00:33:37,800
bringing up all of these negative
613
00:33:37,800 --> 00:33:40,320
potential future use cases for AI,
614
00:33:40,320 --> 00:33:42,960
but to know that there are all of these use cases
615
00:33:42,960 --> 00:33:46,640
where it can help level the playing field
616
00:33:46,640 --> 00:33:51,160
you know, in some ways and open up opportunities for people.
617
00:33:51,160 --> 00:33:55,200
You know, things are changing, things are changing rapidly.
618
00:33:56,320 --> 00:33:58,880
And machine learning does have this tendency
619
00:33:58,880 --> 00:34:02,000
to perpetuate a lot of the things from the past,
620
00:34:02,960 --> 00:34:05,800
but it's nice to know that there are times
621
00:34:05,800 --> 00:34:08,880
when it can actually increase accessibility.
622
00:34:08,880 --> 00:34:12,560
So that's a nice use case.
623
00:34:12,560 --> 00:34:14,280
One of my favorite stories,
624
00:34:14,280 --> 00:34:16,840
one of my favorite stories from the copilot days
625
00:34:16,840 --> 00:34:19,800
were somebody, usually on GitHub,
626
00:34:19,800 --> 00:34:23,040
when somebody files an issue, it's like not a fun thing.
627
00:34:23,040 --> 00:34:25,080
You know, it's like, man, this is broken,
628
00:34:25,080 --> 00:34:28,760
or like, I'm confused or whatever.
629
00:34:28,760 --> 00:34:29,840
But there was one person
630
00:34:29,840 --> 00:34:32,640
whenever we had first released copilot
631
00:34:32,640 --> 00:34:37,640
who wrote and said, you know, like, I have tremors,
632
00:34:37,640 --> 00:34:40,680
like, and they've gotten so bad recently
633
00:34:40,680 --> 00:34:45,680
that, you know, I was forced to, you know, stop doing my job
634
00:34:45,800 --> 00:34:48,240
and this person had been a software engineer.
635
00:34:49,440 --> 00:34:53,560
And they said, you know, you've introduced this kind of like
636
00:34:53,560 --> 00:34:57,200
speech to code, basically,
637
00:34:58,200 --> 00:35:01,200
generation features within the IDE.
638
00:35:01,200 --> 00:35:02,720
And, you know, it's made it
639
00:35:02,720 --> 00:35:04,720
so that I can actually build software again.
640
00:35:04,720 --> 00:35:07,480
And I never thought that I would be able to do that.
641
00:35:07,480 --> 00:35:10,960
And that is also, you know, not late.
642
00:35:10,960 --> 00:35:14,320
That's the kind of thing that makes you delighted
643
00:35:14,320 --> 00:35:16,320
to come to work every day, I think,
644
00:35:16,320 --> 00:35:21,320
is, you know, the potential of making it
645
00:35:21,560 --> 00:35:24,520
so that people can do the things that they love,
646
00:35:24,520 --> 00:35:27,440
even though they might be physically limited in some way.
647
00:35:27,440 --> 00:35:28,280
Right.
648
00:35:28,280 --> 00:35:31,320
Yeah, knowing that you're having a positive impact
649
00:35:31,320 --> 00:35:32,640
on that person, I mean, who knows?
650
00:35:32,640 --> 00:35:34,960
There's probably many other people that are also, you know,
651
00:35:34,960 --> 00:35:38,120
getting that positive impact.
652
00:35:38,120 --> 00:35:41,200
So yeah, that it's definitely delightful.
653
00:35:41,200 --> 00:35:42,960
It's rewarding to hear, you know,
654
00:35:42,960 --> 00:35:44,840
working in this field.
655
00:35:47,080 --> 00:35:48,680
Thanks for geeking out with me,
656
00:35:48,680 --> 00:35:51,920
talking about large language models and all of that.
657
00:35:51,920 --> 00:35:54,520
But I wanna switch and zoom out
658
00:35:54,520 --> 00:35:58,440
into just the machine learning field in general.
659
00:36:00,480 --> 00:36:04,120
So what do you believe is an important question
660
00:36:04,120 --> 00:36:07,160
that remains unanswered in machine learning?
661
00:36:08,520 --> 00:36:10,040
There's so many, right?
662
00:36:10,040 --> 00:36:15,040
And I think one of the questions that I'm most interested in
663
00:36:16,280 --> 00:36:21,280
is how do kind of the pre-training data mixtures
664
00:36:22,840 --> 00:36:25,240
for large language models,
665
00:36:25,240 --> 00:36:29,560
how do they impact performance on downstream tasks?
666
00:36:29,560 --> 00:36:31,440
And this is an unsolved problem.
667
00:36:31,440 --> 00:36:32,720
There's this notion of like,
668
00:36:32,720 --> 00:36:35,000
oh, well, I wanna do code stuff.
669
00:36:35,000 --> 00:36:37,600
So like, perhaps I should have more code
670
00:36:39,000 --> 00:36:41,120
or I wanna do multilingual stuff.
671
00:36:41,120 --> 00:36:44,760
So perhaps I should have more multilingual data,
672
00:36:44,760 --> 00:36:47,120
but nobody knows how much,
673
00:36:47,120 --> 00:36:49,720
nobody knows how much quality impacts
674
00:36:49,720 --> 00:36:52,680
that performance for data.
675
00:36:52,680 --> 00:36:55,960
Nobody knows like how the data should be structured
676
00:36:55,960 --> 00:36:59,080
or formatted or if it should be included
677
00:36:59,080 --> 00:37:00,920
in a broad variety of ways.
678
00:37:00,920 --> 00:37:03,520
Like if you want to predict edits,
679
00:37:03,520 --> 00:37:06,520
perhaps you should have like code dips, right?
680
00:37:06,520 --> 00:37:10,840
Like instead of just the source code itself.
681
00:37:10,840 --> 00:37:14,360
So all of this experimentation
682
00:37:14,360 --> 00:37:17,000
ends up being pretty expensive, right?
683
00:37:17,000 --> 00:37:22,000
Like, and people end up taking like really expensive risks.
684
00:37:22,240 --> 00:37:25,080
So for Palm V2 as an example,
685
00:37:26,000 --> 00:37:28,720
the input pre-training data mixture
686
00:37:28,720 --> 00:37:31,520
included an awful lot of multilingual tokens,
687
00:37:31,520 --> 00:37:34,440
which meant that the number of English tokens
688
00:37:34,440 --> 00:37:35,400
was much lower.
689
00:37:37,240 --> 00:37:40,360
And there was a risk from the team that like,
690
00:37:40,360 --> 00:37:42,240
perhaps we'll train this model,
691
00:37:42,240 --> 00:37:43,840
this like super expensive,
692
00:37:43,840 --> 00:37:45,760
like millions of dollars a model,
693
00:37:46,600 --> 00:37:48,960
and then we'll get something
694
00:37:48,960 --> 00:37:51,360
and it won't be as good at English tasks
695
00:37:51,360 --> 00:37:53,600
as it is on all of these other things.
696
00:37:53,600 --> 00:37:54,680
Right.
697
00:37:54,680 --> 00:37:57,800
And that was a risk that the team took.
698
00:37:57,800 --> 00:38:01,720
And it ended up paying off because the performance
699
00:38:01,720 --> 00:38:04,400
ended up being better actually
700
00:38:04,400 --> 00:38:07,960
across all of the task ranges.
701
00:38:07,960 --> 00:38:11,360
But that could have been something
702
00:38:11,360 --> 00:38:14,600
that turned out very differently.
703
00:38:14,600 --> 00:38:19,400
And I think for people who are interested
704
00:38:19,400 --> 00:38:21,040
in the large language model space
705
00:38:21,040 --> 00:38:22,720
and the deep learning space,
706
00:38:22,720 --> 00:38:26,480
understanding how the data choices that you make,
707
00:38:26,480 --> 00:38:28,920
and then also the quality of the data that you make
708
00:38:28,920 --> 00:38:33,160
impacts how your model performs
709
00:38:33,160 --> 00:38:36,440
is really, really compelling and important.
710
00:38:36,440 --> 00:38:39,160
And then also how does that,
711
00:38:39,160 --> 00:38:42,720
like the pre-training phase
712
00:38:42,720 --> 00:38:46,000
and the attention that you pay in pre-training,
713
00:38:46,000 --> 00:38:51,000
how is that compared to like downstream fine tuning
714
00:38:52,640 --> 00:38:53,840
and instruction tuning?
715
00:38:53,840 --> 00:38:55,520
Because we're increasingly seeing
716
00:38:55,520 --> 00:38:59,520
that the fine tuning instruction tuning RLHF portions
717
00:38:59,520 --> 00:39:01,960
are much more impactful and move the needle
718
00:39:01,960 --> 00:39:05,120
much, much more than everything else.
719
00:39:05,120 --> 00:39:05,960
Right.
720
00:39:05,960 --> 00:39:08,280
What's your intuition on that?
721
00:39:08,280 --> 00:39:10,880
Why do you think it has such a big effect?
722
00:39:12,200 --> 00:39:16,720
So I think it's because it helps the model focus,
723
00:39:16,720 --> 00:39:20,480
which is not a technical way to describe it at all,
724
00:39:20,480 --> 00:39:23,360
but I think it's intuitively maybe a little bit easier
725
00:39:23,360 --> 00:39:24,520
to understand, right?
726
00:39:24,520 --> 00:39:29,240
Like the model, you initially train it on a lot of data,
727
00:39:29,240 --> 00:39:31,160
like the entire internet, right?
728
00:39:31,160 --> 00:39:33,640
Or a lot of data.
729
00:39:34,840 --> 00:39:38,320
And so it's capable of doing a broad variety of things.
730
00:39:38,320 --> 00:39:40,520
Whenever you instruction tune it or fine tune it
731
00:39:40,520 --> 00:39:44,000
or RLHF it, you help it kind of focus
732
00:39:44,000 --> 00:39:47,000
on the kinds of questions that you're really interested
733
00:39:47,000 --> 00:39:48,840
in answering or the kind of format
734
00:39:48,840 --> 00:39:50,840
that you would really, really like to see.
735
00:39:50,840 --> 00:39:55,840
Like say, your model is giving relatively short outputs
736
00:39:57,560 --> 00:40:01,080
and you want them to be a little bit more long form.
737
00:40:01,080 --> 00:40:04,480
You can kind of tune that with RLHF
738
00:40:04,480 --> 00:40:09,480
such that your model is kind of rewarded
739
00:40:10,320 --> 00:40:14,040
whenever it outputs longer context things
740
00:40:14,040 --> 00:40:15,800
as opposed to shorter things.
741
00:40:17,240 --> 00:40:19,640
Right, so you can kind of tune it
742
00:40:19,640 --> 00:40:23,040
to how you want those outputs to be.
743
00:40:23,040 --> 00:40:24,480
Yeah.
744
00:40:24,480 --> 00:40:26,720
One of the things that you mentioned before,
745
00:40:26,720 --> 00:40:29,760
it made me think about like, well, two things.
746
00:40:29,760 --> 00:40:33,160
I guess how there are these different waves
747
00:40:33,160 --> 00:40:36,240
that have happened, I view in machine learning.
748
00:40:36,240 --> 00:40:38,800
Like I think there was a time a couple of years ago
749
00:40:38,800 --> 00:40:42,160
where it was like, you need to have a fine tune model
750
00:40:42,160 --> 00:40:43,440
on this task, right?
751
00:40:43,440 --> 00:40:48,120
And now with all of the advances in generative models,
752
00:40:48,120 --> 00:40:50,920
it's like, oh, maybe there's one model
753
00:40:50,920 --> 00:40:54,400
that can kind of answer all of these questions.
754
00:40:54,400
--> 00:40:56,880
But there still is this question about,
755
00:40:56,880 --> 00:41:01,040
can we use that generative model to then point you
756
00:41:01,040 --> 00:41:03,640
in the right direction of the best tool
757
00:41:03,640 --> 00:41:06,120
to use to solve your problem?
758
00:41:07,600 --> 00:41:12,040
I guess, what's your take on that
759
00:41:12,040 --> 00:41:15,240
and how have you viewed that sort of transition
760
00:41:15,240 --> 00:41:17,200
between having a generative model
761
00:41:17,200 --> 00:41:19,600
that can kind of solve many problems
762
00:41:19,600 --> 00:41:24,600
versus fine tune models that are for specific tasks?
763
00:41:24,760 --> 00:41:25,600
Yeah.
764
00:41:25,600 --> 00:41:28,960
So we found just anecdotally,
765
00:41:30,960 --> 00:41:35,080
so initially for some of our large models
766
00:41:36,680 --> 00:41:40,640
that perhaps weren't pre-trained on as much source code,
767
00:41:41,640 --> 00:41:45,320
we did have to have fine tuned versions
768
00:41:45,320 --> 00:41:48,080
of those models using source code.
769
00:41:48,080 --> 00:41:52,320
And then for future iterations of the larger models,
770
00:41:52,320 --> 00:41:55,200
we included all of the tokens that we had used
771
00:41:55,200 --> 00:41:56,840
during the fine tuning process
772
00:41:56,840 --> 00:41:59,280
within the pre-training data mixture.
773
00:41:59,280 --> 00:42:03,600
And the resulting model exceeded the performance
774
00:42:03,600 --> 00:42:08,600
of the fine tuned model based on those kinds of inclusions
775
00:42:09,680 --> 00:42:12,680
and then also using a tokenizer
776
00:42:12,680 --> 00:42:15,240
that was a little bit friendlier for code.
777
00:42:15,240 --> 00:42:20,240
So practically we've seen that the generative models
778
00:42:22,280 --> 00:42:26,920
provided you keep adding in high quality pre-training data,
779
00:42:26,920 --> 00:42:30,440
they can sort of absorb the tasks
780
00:42:30,440 --> 00:42:32,200
from the fine tuned models.
781
00:42:33,320 --> 00:42:37,080
But I also will say that for businesses,
782
00:42:38,760 --> 00:42:42,280
whatever solves your problem,
783
00:42:42,280 --> 00:42:45,240
like that's the thing that you should use.
784
00:42:45,240 --> 00:42:50,240
And Lord knows the world is like running on Excel spreadsheets
785
00:42:50,400 --> 00:42:53,120
for much of the finance sector.
786
00:42:53,120 --> 00:42:55,920
And the government is running on COBOL.
787
00:42:55,920 --> 00:43:00,200
And like the cost of migrating off of either of those things
788
00:43:00,200 --> 00:43:01,600
is just overwhelming.
789
00:43:01,600 --> 00:43:03,280
And I don't wanna be the one causing
790
00:43:03,280 --> 00:43:05,800
like the financial downturn of like
791
00:43:05,800 --> 00:43:07,440
telling the finance community
792
00:43:07,440 --> 00:43:09,760
that they shouldn't be using Excel spreadsheets
793
00:43:09,760 --> 00:43:12,680
or basic or whatever it is.
794
00:43:12,680 --> 00:43:17,680
So I would say that whatever modeling meets your needs
795
00:43:21,960 --> 00:43:23,720
is what you should use.
796
00:43:23,720 --> 00:43:27,080
And certainly experiment with generative models,
797
00:43:27,080 --> 00:43:29,240
see if it makes sense.
798
00:43:29,240 --> 00:43:31,680
And then also see if it makes financial sense.
799
00:43:31,680 --> 00:43:34,520
Cause if you can get by with a smaller model
800
00:43:34,520 --> 00:43:38,320
that perhaps is open source and easier to maintain,
801
00:43:38,320 --> 00:43:42,280
then why should you be paying for API calls?
802
00:43:44,720 --> 00:43:47,360
But that's personal opinion.
803
00:43:47,360 --> 00:43:51,120
Like new technology is always going to be very cool
804
00:43:51,120 --> 00:43:52,200
and push boundaries.
805
00:43:52,200 --> 00:43:56,520
But at the end of the day, what matters is your business,
806
00:43:56,520 --> 00:43:58,520
your users and their problems.
807
00:43:59,800 --> 00:44:01,080
Yeah, absolutely.
808
00:44:01,080 --> 00:44:03,560
I agree 100%.
809
00:44:03,560 --> 00:44:06,280
Identifying the problem, the underlying need,
810
00:44:06,280 --> 00:44:10,240
knowing that there's many approaches.
811
00:44:10,240 --> 00:44:15,240
I view that generative models are one tool in your toolset.
812
00:44:18,600 --> 00:44:22,360
You don't necessarily need to use it for everything.
813
00:44:22,360 --> 00:44:25,320
It has a lot of amazing use cases, but yeah.
814
00:44:27,040 --> 00:44:31,320
Certain business practices like you were mentioning are,
815
00:44:32,360 --> 00:44:33,720
I'll say stuck in there,
816
00:44:33,720 --> 00:44:35,400
stuck in there certain ways.
817
00:44:35,400 --> 00:44:39,440
And we're moving towards this new future,
818
00:44:39,440 --> 00:44:43,640
but everyone's not ready for the leap over the canyon.
819
00:44:46,880 --> 00:44:47,700
I don't know.
820
00:44:47,700 --> 00:44:49,840
There's a place, there's a space
821
00:44:49,840 --> 00:44:52,080
and we have to get people over.
822
00:44:52,080 --> 00:44:57,080
And some people will take smaller steps than others.
823
00:44:57,400 --> 00:45:01,840
Yeah, and there's also like some appetite for risk.
824
00:45:01,840 --> 00:45:02,960
I think you have to have
825
00:45:02,960 --> 00:45:04,960
if you're preferring generative models,
826
00:45:04,960 --> 00:45:09,960
just given that we're still exploring factuality
827
00:45:10,560 --> 00:45:12,320
and being able to verify
828
00:45:12,320 --> 00:45:14,860
that models are returning correct responses.
829
00:45:14,860 --> 00:45:16,960
Cause a lot of the time they don't.
830
00:45:16,960 --> 00:45:21,960
Like there are always examples towards like counterfactuals.
831
00:45:24,920 --> 00:45:29,920
But the one thing that I saw on Twitter earlier this week,
832
00:45:30,480 --> 00:45:32,240
cause all of these generative models things
833
00:45:32,240 --> 00:45:34,880
seem to be percolating to Twitter.
834
00:45:34,880 --> 00:45:38,880
But someone instead of like
consulting
835
00:45:38,880 --> 00:45:41,960
with a financial analyst, they were like, I have some money
836
00:45:41,960 --> 00:45:43,840
and I wanna invest it in some stocks,
837
00:45:43,840 --> 00:45:46,440
like recommend some stocks that I should buy.
838
00:45:46,440 --> 00:45:49,980
And they got Chad GBT to do that.
839
00:45:49,980 --> 00:45:52,200
And then they purchased the stock.
840
00:45:52,200 --> 00:45:54,800
They kind of mapped it out over time
841
00:45:54,800 --> 00:45:56,120
as to what would have happened
842
00:45:56,120 --> 00:45:59,800
if they followed the portfolio recommendation
843
00:45:59,800 --> 00:46:04,120
versus the buy some stock recommendation.
844
00:46:04,120 --> 00:46:09,120
And the Chad GBT recommended stock purchases
845
00:46:09,440 --> 00:46:13,380
or the generative model recommended stock purchases
846
00:46:13,380 --> 00:46:15,440
actually ended up doing quite, quite well
847
00:46:15,440 --> 00:46:20,440
compared to the kind of investment portfolio recommendations.
848
00:46:21,120 --> 00:46:22,720
But all I could think was just like,
849
00:46:22,720 --> 00:46:25,760
I don't want my 401k to be at the mercy
850
00:46:25,760 --> 00:46:28,420
of a generative model, at least not now.
851
00:46:28,420 --> 00:46:32,000
Like it's, and it might be that perhaps
852
00:46:32,000 --> 00:46:36,600
these investment strategists are recommending certain things
853
00:46:36,600 --> 00:46:39,500
because they know that it's more stable
854
00:46:39,500 --> 00:46:44,120
than perhaps these stocks that might be in the near term,
855
00:46:45,240 --> 00:46:47,040
high performers, but long-term,
856
00:46:47,040 --> 00:46:52,040
like much more variable in terms of return.
857
00:46:52,880 --> 00:46:56,040
So it's like choose your own adventure
858
00:46:56,040 --> 00:47:01,040
and be very careful for betting all of your money
859
00:47:01,040 --> 00:47:02,980
on generative models.
860
00:47:02,980 --> 00:47:07,320
Yeah, I mean, yeah, in terms of financial decisions,
861
00:47:07,320 --> 00:47:08,980
I think, yeah, everyone's sort of like,
862
00:47:08,980 --> 00:47:11,300
how much risk are you willing to take?
863
00:47:11,300 --> 00:47:12,780
And then there should be like another step,
864
00:47:12,780 --> 00:47:14,260
like, are you willing to take the risk
865
00:47:14,260 --> 00:47:17,140
of using a generative model for your financial portfolio?
866
00:47:17,140 --> 00:47:19,140
Maybe some people will say yes, who knows?
867
00:47:22,120 --> 00:47:24,340
Okay, so switching into like
868
00:47:25,180 --> 00:47:28,020
learning from machine learning standpoint
869
00:47:28,020 --> 00:47:33,020
and just sort of general advice for people in the industry.
870
00:47:35,140 --> 00:47:36,680
Well, I'll start with this one.
871
00:47:37,660 --> 00:47:40,140
Who are some people in the machine learning field
872
00:47:40,140 --> 00:47:41,480
that influence you?
873
00:47:43,740 --> 00:47:47,260
So there have been so many,
874
00:47:48,500 --> 00:47:51,220
I think, and I'm fortunate to work
875
00:47:51,220 --> 00:47:53,620
with many of them every day.
876
00:47:53,620 --> 00:47:58,620
So I've been coming from kind of the open source tools,
877
00:48:00,420 --> 00:48:02,020
developer tool space.
878
00:48:02,020 --> 00:48:06,040
I've really loved working with the JAX team.
879
00:48:06,040 --> 00:48:08,620
So JAX is for folks who might not know,
880
00:48:08,620 --> 00:48:13,140
it's an open source library for doing,
881
00:48:14,060 --> 00:48:15,940
kind of building deep learning models,
882
00:48:15,940 --> 00:48:20,940
but also doing kind of scientific experimentation
883
00:48:20,940 --> 00:48:23,900
and all of the models that we build at AlphaBet
884
00:48:23,900 --> 00:48:25,400
are built using JAX.
885
00:48:26,820 --> 00:48:30,940
Matt Johnson, Peter Hawkins, James Bradbury,
886
00:48:30,940 --> 00:48:33,620
like they are all, Sky Wonderman-Milm,
887
00:48:33,620 --> 00:48:35,540
they're all very, very close collaborators,
888
00:48:35,540 --> 00:48:39,060
Roy Frosting as well, Yash Kataria,
889
00:48:39,060 --> 00:48:42,840
and they've been delightful to work with and to learn from.
890
00:48:42,840 --> 00:48:47,840
I've also really, really loved working with Jeff Dean,
891
00:48:54,120 --> 00:48:58,840
as well as the collaborators at DeepMind.
892
00:48:58,840 --> 00:49:00,920
So as part of this Gemini effort,
893
00:49:00,920 --> 00:49:05,920
I've gotten to work more closely with Oriol Benyals,
894
00:49:06,040 --> 00:49:10,160
who's kind of driving much of the efforts.
895
00:49:10,160 --> 00:49:13,520
And then also, of course, the learning for code folks,
896
00:49:14,520 --> 00:49:18,080
who care very deeply about machine learning
897
00:49:18,080 --> 00:49:19,760
as applied to software systems
898
00:49:19,760 --> 00:49:23,100
and think very carefully and thoughtfully about it.
899
00:49:23,100 --> 00:49:26,560
And if you're on Twitter,
900
00:49:26,560 --> 00:49:29,600
I would also highly recommend following
901
00:49:29,600 --> 00:49:31,920
the Hugging Face team,
902
00:49:31,920 --> 00:49:34,160
because they have a lot of passion
903
00:49:34,160 --> 00:49:36,600
around open source machine learning and deep learning
904
00:49:36,600 --> 00:49:38,120
and sharing their knowledge,
905
00:49:38,120 --> 00:49:40,800
as well as Andre Karpathy,
906
00:49:40,800 --> 00:49:45,280
who is a dear, sweet human just generally,
907
00:49:45,280 --> 00:49:49,000
but also cares a lot about education and advocacy
908
00:49:49,000 --> 00:49:51,000
and making sure that these concepts
909
00:49:51,000 --> 00:49:53,560
are understandable to everyday humans.
910
00:49:54,520 --> 00:49:59,520
So there are a lot of people
911
00:50:01,320 --> 00:50:02,680
to admire in this community.
912
00:50:02,680 --> 00:50:04,440
I feel very fortunate.
913
00:50:04,440 --> 00:50:09,080
Yeah, absolutely. Yeah, there's so many great people
914
00:50:09,080 --> 00:50:12,720
in the field and
people are pretty generous
915
00:50:12,720 --> 00:50:13,760
with their time also.
916
00:50:13,760 --> 00:50:18,760
And there's this sense of wanting to help other people out.
917
00:50:19,940 --> 00:50:22,440
And it's a really nice community
918
00:50:22,440 --> 00:50:24,440
to be in the machine learning community.
919
00:50:26,080 --> 00:50:30,560
On your website, which is nicely named
920
00:50:30,560 --> 00:50:32,800
Dynamic Webpage, right?
921
00:50:32,800 --> 00:50:33,640
Is that?
922
00:50:33,640 --> 00:50:37,800
Oh, so my website,
923
00:50:37,800 --> 00:50:42,800
I have a tendency to purchase URLs that are puns.
924
00:50:43,520 --> 00:50:47,880
And I think probably most of us in this industry
925
00:50:47,880 --> 00:50:50,480
are like URL hoarders,
926
00:50:50,480 --> 00:50:54,400
but the one that might have,
927
00:50:55,600 --> 00:50:57,920
I think the URL might have even expired,
928
00:50:57,920 --> 00:51:01,960
but page views is the one.
929
00:51:01,960 --> 00:51:04,400
Okay, so under a section that I saw
930
00:51:04,400 --> 00:51:06,680
on one of your websites, it was under page views.
931
00:51:06,680 --> 00:51:10,840
You had some really interesting tenants.
932
00:51:10,840 --> 00:51:12,800
I'm gonna read them if you don't mind.
933
00:51:12,800 --> 00:51:14,200
Yep. Yep.
934
00:51:14,200 --> 00:51:17,960
So bring data to opinion fights,
935
00:51:17,960 --> 00:51:20,760
relentlessly ask questions,
936
00:51:20,760 --> 00:51:24,800
communication is everything, especially in open source,
937
00:51:24,800 --> 00:51:27,160
choose growth opportunities,
938
00:51:27,160 --> 00:51:29,820
nurture and build communities.
939
00:51:29,820 --> 00:51:31,200
If you don't have documentation,
940
00:51:31,200 --> 00:51:33,240
you don't have an MVP,
941
00:51:33,240 --> 00:51:35,360
give without expecting a return
942
00:51:35,360 --> 00:51:38,280
and believe in people, not acronyms.
943
00:51:38,280 --> 00:51:39,400
I love these.
944
00:51:39,400 --> 00:51:41,440
I love them so much.
945
00:51:41,440 --> 00:51:42,280
Thank you.
946
00:51:42,280 --> 00:51:43,400
Are there any that you,
947
00:51:43,400 --> 00:51:45,560
is there anything that you would add to any of them
948
00:51:45,560 --> 00:51:48,600
or anything that you wanna talk about with any of those?
949
00:51:48,600 --> 00:51:53,600
So those, I think I put down on paper
950
00:51:54,800 --> 00:51:56,440
about half a decade ago
951
00:51:56,440 --> 00:52:01,440
and they still seem like pretty good life roles.
952
00:52:01,520 --> 00:52:06,520
I think now, especially the believe in people,
953
00:52:06,520 --> 00:52:11,520
not acronyms, one is going to become even more compelling
954
00:52:13,440 --> 00:52:14,960
and important.
955
00:52:14,960 --> 00:52:18,120
So COVID kind of turned everything on its head, right?
956
00:52:18,120 --> 00:52:22,940
Like I personally, like it helped prioritize things for me.
957
00:52:22,940 --> 00:52:24,880
It was part of the reason why I left Alphabet.
958
00:52:24,880 --> 00:52:26,680
Like I wanted to move back to Texas
959
00:52:26,680 --> 00:52:28,000
to take care of my parents.
960
00:52:28,000 --> 00:52:33,000
They're like 70 plus, 80 plus.
961
00:52:34,880 --> 00:52:39,880
And I think a lot of other people also,
962
00:52:40,480 --> 00:52:42,320
it helped them prioritize
963
00:52:42,320 --> 00:52:45,520
what they should be spending their time on.
964
00:52:45,520 --> 00:52:49,400
So we saw a lot of kids choose not to go to college
965
00:52:49,400 --> 00:52:51,400
and to perhaps enter into the workforce
966
00:52:51,400 --> 00:52:54,520
or to perhaps like start building software
967
00:52:54,520 --> 00:52:57,160
or to build their own businesses, right?
968
00:52:57,160 --> 00:52:59,040
And these kids are incredible.
969
00:52:59,040 --> 00:53:01,480
Like they're doing such amazing things.
970
00:53:01,480 --> 00:53:05,240
Like the most killer use cases for generative models
971
00:53:05,240 --> 00:53:08,080
are all these kids that are like entrepreneurs
972
00:53:08,080 --> 00:53:11,320
and somehow like 16 years old, right?
973
00:53:11,320 --> 00:53:16,320
And so I feel like, and then also
974
00:53:16,520 --> 00:53:18,360
in the deep learning community,
975
00:53:18,360 --> 00:53:21,140
we're increasingly seeing that folks
976
00:53:21,140 --> 00:53:23,800
aren't really publishing papers anymore, right?
977
00:53:23,800 --> 00:53:27,840
Like whereas previously there was this academic ivory tower
978
00:53:27,840 --> 00:53:30,680
of like, oh, you do some research, you produce a paper,
979
00:53:30,680 --> 00:53:32,000
you present to the conference,
980
00:53:32,000 --> 00:53:33,160
and then you add it to like
981
00:53:33,160 --> 00:53:36,120
this extensive scholarly pedigree.
982
00:53:36,120 --> 00:53:39,560
Whereas now it's just kinda like, well, I did some work.
983
00:53:39,560 --> 00:53:41,600
I produced something that people can test out.
984
00:53:41,600 --> 00:53:44,360
I will perhaps write a blog post, but for the most part,
985
00:53:44,360 --> 00:53:47,080
it's just like something cool that's out in the world
986
00:53:47,080 --> 00:53:50,160
that people can like see and touch and experience.
987
00:53:50,160 --> 00:53:55,160
So I think that if you don't have a college degree,
988
00:53:57,720 --> 00:54:01,640
if you are debating whether or not college is right for you,
989
00:54:01,640 --> 00:54:04,880
if you're concerned that like perhaps not having a degree
990
00:54:04,880 --> 00:54:06,620
will limit you in some way,
991
00:54:06,620 --> 00:54:09,160
whether it's an undergrad degree or a graduate degree,
992
00:54:09,160 --> 00:54:11,680
or if you have like a journalism degree and you're like,
993
00:54:11,680 --> 00:54:13,820
but can I do these work?
994
00:54:13,820 --> 00:54:17,760
You can, there are no rules in this space.
995
00:54:17,760 --> 00:54:20,960
Like, you can do anything that you would like to.
996
00:54:20,960 --> 00:54:24,200
And the important
part is just kind of focus
997
00:54:24,200 --> 00:54:28,860
on building something great that delights you
998
00:54:28,860 --> 00:54:31,120
and that delights other people.
999
00:54:31,120 --> 00:54:32,960
And that's all that really matters.
1000
00:54:33,960 --> 00:54:36,880
That's really good advice for people
1001
00:54:36,880 --> 00:54:38,440
starting out in the field.
1002
00:54:40,760 --> 00:54:43,380
A little other, another variation of that,
1003
00:54:43,380 --> 00:54:48,380
when you were just kind of getting your feet under you,
1004
00:54:49,220 --> 00:54:51,180
or I mean, I don't know if that is even the case,
1005
00:54:51,180 --> 00:54:53,500
but what advice would you give yourself
1006
00:54:53,500 --> 00:54:57,200
when you were starting your career in data science?
1007
00:54:58,860 --> 00:55:03,860
So I would say, I would just kind of reinforce,
1008
00:55:04,660 --> 00:55:08,200
do what you think is right and what interests you,
1009
00:55:08,200 --> 00:55:13,200
and then don't feel, give yourself permission
1010
00:55:13,200 --> 00:55:18,200
to do that and don't feel guilty or like you aren't,
1011
00:55:18,840 --> 00:55:23,840
like you aren't meeting up to the expectations
1012
00:55:24,320 --> 00:55:28,520
of your peers or your academic advisors.
1013
00:55:28,520 --> 00:55:33,520
I was told when I had entered into my graduate program,
1014
00:55:33,620 --> 00:55:36,360
I was told that it was a waste of time
1015
00:55:36,360 --> 00:55:40,020
to do like computer science and programming,
1016
00:55:40,020 --> 00:55:41,760
that I would never be respected
1017
00:55:41,760 --> 00:55:45,680
if all I did was like build tools or build libraries
1018
00:55:45,680 --> 00:55:49,840
or write code to solve science problems.
1019
00:55:50,760 --> 00:55:52,880
And I was also told that machine learning
1020
00:55:52,880 --> 00:55:55,320
would never have a place in the earth sciences,
1021
00:55:56,760 --> 00:56:01,760
which was in hindsight, kind of silly.
1022
00:56:04,680 --> 00:56:06,400
But at the time I was just like,
1023
00:56:06,400 --> 00:56:12,040
am I making the worst career choice of my entire life?
1024
00:56:12,040 --> 00:56:16,400
And I had a lot of internal emotional angst about it
1025
00:56:17,920 --> 00:56:22,920
back then, but now it's definitely the right choice.
1026
00:56:23,400 --> 00:56:26,840
If you feel like something is important,
1027
00:56:28,560 --> 00:56:32,280
sometimes you can see the future,
1028
00:56:32,280 --> 00:56:36,920
even if other people are standing,
1029
00:56:36,920 --> 00:56:38,580
facing a different direction.
1030
00:56:40,720 --> 00:56:42,440
That's great.
1031
00:56:42,440 --> 00:56:44,200
Yeah, it's so interesting, right?
1032
00:56:44,200 --> 00:56:49,040
When you're in the moment, things can seem so chaotic
1033
00:56:49,040 --> 00:56:51,440
and there could be so much uncertainty.
1034
00:56:51,440 --> 00:56:54,560
And then, I stop myself,
1035
00:56:54,560 --> 00:56:56,000
like the things that you're saying,
1036
00:56:56,000 --> 00:56:59,120
they're like looking at things now,
1037
00:56:59,120 --> 00:57:00,960
it's like some of that's like laughable.
1038
00:57:00,960 --> 00:57:04,800
Like, you look now, I mean, hindsight's 2020, of course,
1039
00:57:04,800 --> 00:57:05,640
but it's just, I mean,
1040
00:57:05,640 --> 00:57:08,200
machine learning has become so pervasive
1041
00:57:08,200 --> 00:57:11,640
and using all of these sorts of techniques and technologies.
1042
00:57:13,040 --> 00:57:17,000
But it's amazing how, yeah, in the moment,
1043
00:57:17,000 --> 00:57:20,000
it can be so uncertain, but then when you look back,
1044
00:57:20,000 --> 00:57:23,160
it can become so clear.
1045
00:57:24,320 --> 00:57:29,320
And that was just like eight or nine years ago, by the way,
1046
00:57:29,320 --> 00:57:33,120
that people were thinking that machine learning
1047
00:57:33,120 --> 00:57:34,980
wouldn't have a place in the earth sciences,
1048
00:57:34,980 --> 00:57:37,220
wouldn't have a place in the physical sciences.
1049
00:57:37,220 --> 00:57:39,060
All of this is super new.
1050
00:57:40,080 --> 00:57:44,220
And so I think that given the trajectory
1051
00:57:44,220 --> 00:57:46,960
and kind of the exponential increase
1052
00:57:46,960 --> 00:57:49,680
and how these things are progressing,
1053
00:57:49,680 --> 00:57:52,400
it's really tricky to predict the future.
1054
00:57:52,400 --> 00:57:54,720
And there's, I forget who said it,
1055
00:57:54,720 --> 00:57:58,040
but like, and it's kind of cliche, I guess now,
1056
00:57:58,040 --> 00:58:00,160
but the best way to predict the future
1057
00:58:00,160 --> 00:58:01,680
is to be the one building it.
1058
00:58:01,680 --> 00:58:04,080
Like, it is very clear that the future
1059
00:58:04,080 --> 00:58:08,440
is going to be built using the generative technologies
1060
00:58:08,440 --> 00:58:10,440
and assistive technologies
1061
00:58:10,440 --> 00:58:13,040
and productivity enhancing technologies.
1062
00:58:13,040 --> 00:58:15,000
So if you're doing that,
1063
00:58:15,000 --> 00:58:16,960
you're probably gonna be heading in
1064
00:58:16,960 --> 00:58:18,360
towards the right direction.
1065
00:58:19,360 --> 00:58:20,800
Absolutely.
1066
00:58:20,800 --> 00:58:24,360
So speaking of how fast things are changing
1067
00:58:24,360 --> 00:58:28,920
in the last decade, in the last five years,
1068
00:58:28,920 --> 00:58:31,360
even in the last year or week,
1069
00:58:32,920 --> 00:58:37,920
how do you stay up on all of the newest techniques?
1070
00:58:38,720 --> 00:58:41,280
I mean, I know in a way you're part of it,
1071
00:58:41,280 --> 00:58:43,280
you're building the future,
1072
00:58:43,280 --> 00:58:47,280
but there's so many people that are releasing stuff.
1073
00:58:48,520 --> 00:58:50,200
Do you have any techniques
1074
00:58:50,200 --> 00:58:54,400
to sort of stay up on top of everything?
1075
00:58:54,400 --> 00:58:57,200
It's a really tricky problem.
1076
00:58:57,200 --> 00:59:00,640
I would, and again, this is gonna sound silly.
1077
00:59:00,640 --> 00:59:04,160
I would be on Twitter
1078
00:59:04,160 --> 00:59:06,560
just because the machine learning community
1079
00:59:06,560 --> 00:59:09,440
seems to hang out on Twitter
1080
00:59:09,440 --> 00:59:12,240
and they talk about things that they find interesting.
1081
00:59:12,240 --> 00:59:15,280
It's like getting to sneakily listen in
1082
00:59:15,280 --> 00:59:17,280
to all of the hallway conversations
1083
00:59:17,280 --> 00:59:20,120
in all of these AI research labs.
1084
00:59:20,120 --> 00:59:24,200
And increasingly, we had mentioned it earlier,
1085
00:59:24,200 --> 00:59:27,840
but people aren't writing papers anymore.
1086
00:59:29,040 --> 00:59:32,680
And so the best insights that you can give
1087
00:59:32,680 --> 00:59:37,680
into kind of pedagogy and kind of how these models
1088
00:59:37,920 --> 00:59:41,920
were built is by hearing what people
1089
00:59:41,920 --> 00:59:43,280
are currently focused on.
1090
00:59:43,280 --> 00:59:45,120
Like, is it long context?
1091
00:59:45,120 --> 00:59:49,680
Is it this new tokenizer that seems interesting?
1092
00:59:49,680 --> 00:59:51,960
Is it some sort of algorithmic advance
1093
00:59:51,960 --> 00:59:55,000
that people are really paying attention to?
1094
00:59:55,000 --> 00:59:56,960
Is it the data mixtures?
1095
00:59:56,960 --> 01:00:00,680
Is it some other model architecture?
1096
01:00:00,680 --> 01:00:05,080
But the kind of honing in on where the conversation
1097
01:00:05,080 --> 01:00:09,080
seems to be happening is really, really important.
1098
01:00:09,080 --> 01:00:11,040
And right now, if you're just looking
1099
01:00:11,040 --> 01:00:13,200
or attempting to look at every deep learning paper
1100
01:00:13,200 --> 01:00:16,200
that's posted on archive, your life's gonna be insane.
1101
01:00:16,200 --> 01:00:18,880
Like there was a website a while back,
1102
01:00:18,880 --> 01:00:21,040
I think it was created by Andre Carpathy,
1103
01:00:21,040 --> 01:00:25,000
the archive sanity website, and it would post the paper
1104
01:00:25,000 --> 01:00:30,000
and then have like a really nice kind of images with PDFs.
1105
01:00:31,080 --> 01:00:34,760
But in the last three or four years,
1106
01:00:34,760 --> 01:00:37,920
it's gotten overwhelming to keep up even with that.
1107
01:00:37,920 --> 01:00:41,040
So I would strongly encourage you,
1108
01:00:41,040 --> 01:00:44,000
pick people that work at the AI research labs
1109
01:00:44,000 --> 01:00:49,000
that you care about, Anthropic, Google DeepMind, OpenAI,
1110
01:00:50,320 --> 01:00:53,280
and perhaps some of the others,
1111
01:00:53,280 --> 01:00:55,400
like Hugging Face that I had mentioned,
1112
01:00:56,360 --> 01:00:59,440
and follow them, see what they're talking about,
1113
01:00:59,440 --> 01:01:01,300
see who else they're following.
1114
01:01:02,200 --> 01:01:04,720
And then also don't be afraid to ask questions
1115
01:01:04,720 --> 01:01:07,160
if something isn't clear, because a lot of time,
1116
01:01:07,160 --> 01:01:09,800
if you ask a question to these people on Twitter,
1117
01:01:09,800 --> 01:01:12,480
they will respond, and that's huge, right?
1118
01:01:12,480 --> 01:01:17,480
Like being able to have access to some of the folks
1119
01:01:17,960 --> 01:01:20,620
that are building the future is massive.
1120
01:01:21,760 --> 01:01:24,520
Yeah, absolutely.
1121
01:01:24,520 --> 01:01:27,120
One thing that I didn't get to touch on,
1122
01:01:27,120 --> 01:01:30,800
so yeah, obviously machine learning, AI, generative models
1123
01:01:30,800 --> 01:01:33,060
has really made it into the mainstream,
1124
01:01:33,060 --> 01:01:37,200
and it's created this like frenzied hype.
1125
01:01:37,200 --> 01:01:42,200
I'm wondering from your perspective,
1126
01:01:42,640 --> 01:01:45,120
how do you view the gap between the hype
1127
01:01:45,120 --> 01:01:48,080
and the reality of AI?
1128
01:01:49,560 --> 01:01:52,960
Yeah, and this industry in particular,
1129
01:01:52,960 --> 01:01:57,320
there are all these hype cycles can be overwhelming.
1130
01:01:57,320 --> 01:02:01,200
I personally cannot wait for the AI hype cycle to be overt
1131
01:02:01,200 --> 01:02:03,960
so that we can all have a little bit more peace
1132
01:02:03,960 --> 01:02:05,700
to do the work.
1133
01:02:05,700 --> 01:02:08,360
I miss when AI was not cool,
1134
01:02:08,360 --> 01:02:12,480
because like, and NeurIPS was a lot more fun than two.
1135
01:02:12,480 --> 01:02:15,600
Like all of the academic conferences were much more chill.
1136
01:02:15,600 --> 01:02:18,680
Now it feels like there's more VCs than researchers.
1137
01:02:19,680 --> 01:02:24,680
But I do think whenever you're trying to distinguish
1138
01:02:25,880 --> 01:02:30,160
between who is like an AI influencer versus
1139
01:02:30,160 --> 01:02:33,440
who is someone who does this for their day job,
1140
01:02:33,440 --> 01:02:36,560
like definitely look and see where they work,
1141
01:02:36,560 --> 01:02:38,280
look and see what they've built.
1142
01:02:39,320 --> 01:02:43,920
And if it's somebody that decided to get excited about AI,
1143
01:02:43,920 --> 01:02:46,880
just about the time that web three went downhill,
1144
01:02:48,080 --> 01:02:51,720
because like somehow a lot of these web three influencers
1145
01:02:51,720 --> 01:02:54,280
have turned into like generative AI influencers,
1146
01:02:54,280 --> 01:02:55,960
and I don't know how that happened.
1147
01:02:55,960 --> 01:02:59,280
But just kind of like look at the backlog,
1148
01:02:59,280 --> 01:03:02,160
see what the person has accomplished
1149
01:03:02,160 --> 01:03:04,400
and what they've been interested in.
1150
01:03:04,400 --> 01:03:08,600
And if they haven't been doing AI for longer
1151
01:03:08,600 --> 01:03:10,640
than the last year or two,
1152
01:03:10,640 --> 01:03:14,560
then I would take what they say with a grain of
salt.
1153
01:03:15,920 --> 01:03:17,320
Yeah, absolutely.
1154
01:03:17,320 --> 01:03:20,480
I think it's really good advice.
1155
01:03:20,480 --> 01:03:23,120
That's why it's really nice talking to somebody like you,
1156
01:03:24,080 --> 01:03:29,080
who has experience at Microsoft and GitHub and Google
1157
01:03:29,080 --> 01:03:32,640
and Alphabet and DeepMind and everything like that.
1158
01:03:33,960 --> 01:03:36,080
The last juicy meaty question,
1159
01:03:37,240 --> 01:03:41,440
what has a career in machine learning taught you about life?
1160
01:03:44,240 --> 01:03:45,800
That is a wonderful question.
1161
01:03:45,800 --> 01:03:48,400
I think what it has taught me the most
1162
01:03:48,400 --> 01:03:52,060
is what to appreciate about being human.
1163
01:03:52,060 --> 01:03:56,920
Like the more you see what these models can do,
1164
01:03:56,920 --> 01:03:59,920
the more you can understand what they're doing.
1165
01:03:59,920 --> 01:04:04,920
And it's very exciting and it's very cool.
1166
01:04:05,000 --> 01:04:10,000
And it's like discovering new capabilities every single day,
1167
01:04:11,160 --> 01:04:13,920
new ways in which I can automate myself
1168
01:04:14,880 --> 01:04:17,960
and sort of remove the tedious parts of the day
1169
01:04:17,960 --> 01:04:22,440
that I like creating meeting transcripts
1170
01:04:22,440 --> 01:04:27,440
or emails or drafting a paragraph for a doc or something.
1171
01:04:30,080 --> 01:04:34,320
But also seeing the ways in which it falls short.
1172
01:04:34,320 --> 01:04:39,320
Like it's not going to be able to give you a hug.
1173
01:04:42,560 --> 01:04:47,560
It's not going to sort of understand, at least not yet.
1174
01:04:48,760 --> 01:04:51,280
It's not going to understand that you might have a bad day
1175
01:04:51,280 --> 01:04:55,460
and that it should ask you, how are you doing?
1176
01:04:56,440 --> 01:05:01,440
It's not going to do those things
1177
01:05:03,240 --> 01:05:06,640
that are uniquely nice experiences
1178
01:05:06,640 --> 01:05:09,540
of interacting with humans in real life every day.
1179
01:05:10,640 --> 01:05:13,600
And I think that's important.
1180
01:05:13,600 --> 01:05:18,600
And I think one of the nicest possible outcomes
1181
01:05:18,760 --> 01:05:21,160
for generative AI becoming ingrained
1182
01:05:21,160 --> 01:05:23,480
in every person's life is that,
1183
01:05:25,120 --> 01:05:29,640
it helps us appreciate more what it is to be a human
1184
01:05:29,640 --> 01:05:31,240
and to interact with humans.
1185
01:05:32,640 --> 01:05:34,600
That's really nice.
1186
01:05:34,600 --> 01:05:35,660
Yeah, that's beautiful.
1187
01:05:35,660 --> 01:05:40,560
So by exploring the capabilities of machine learning and AI,
1188
01:05:40,560 --> 01:05:44,040
you can really appreciate the human connection.
1189
01:05:44,040 --> 01:05:45,360
Yeah.
1190
01:05:45,360 --> 01:05:48,200
So for folks that want to learn more about you
1191
01:05:48,200 --> 01:05:49,400
or the work that you're doing,
1192
01:05:49,400 --> 01:05:52,200
what would be a good resource for them?
1193
01:05:53,400 --> 01:05:54,240
Cool.
1194
01:05:54,240 --> 01:05:59,240
So I strongly recommend taking a look at both the Google AI
1195
01:05:59,480 --> 01:06:01,040
and the DeepMind websites,
1196
01:06:01,040 --> 01:06:04,400
though they will probably be merging at some point
1197
01:06:04,400 --> 01:06:06,120
in the near future.
1198
01:06:06,120 --> 01:06:09,240
I'm also chronically available on Twitter.
1199
01:06:09,240 --> 01:06:11,560
So twitter.com slash Dynamic Web Page,
1200
01:06:11,560 --> 01:06:12,720
and I'm Dynamic Web Page
1201
01:06:12,720 --> 01:06:15,520
pretty much everywhere else on the internet.
1202
01:06:15,520 --> 01:06:19,320
And then I also strongly, strongly encourage
1203
01:06:19,320 --> 01:06:24,320
everyone to just kind of get involved.
1204
01:06:26,760 --> 01:06:30,720
There are certainly ways to begin learning more
1205
01:06:30,720 --> 01:06:33,400
about the AI community without necessarily
1206
01:06:33,400 --> 01:06:35,160
having to have a career in it.
1207
01:06:36,160 --> 01:06:39,320
And then if you do want to have a career in it,
1208
01:06:39,320 --> 01:06:42,600
I think it's going to become increasingly possible
1209
01:06:42,600 --> 01:06:46,400
as businesses seem to be adopting AI
1210
01:06:46,400 --> 01:06:50,600
at a much quicker rate these days.
1211
01:06:50,600 --> 01:06:53,600
So don't be afraid to pitch to your boss
1212
01:06:53,600 --> 01:06:56,200
if you have an idea and you want to create a prototype.
1213
01:06:57,680 --> 01:06:58,720
Absolutely.
1214
01:06:59,680 --> 01:07:02,280
Paige, it has been such a pleasure.
1215
01:07:02,280 --> 01:07:04,000
I really appreciate your time.
1216
01:07:04,000 --> 01:07:07,200
Thank you for letting me pick your brain for this time.
1217
01:07:07,200 --> 01:07:08,800
Thank you so much.
1218
01:07:08,800 --> 01:07:09,620
No, thank you.
1219
01:07:09,620 --> 01:07:10,560
This has been delightful.
1220
01:07:10,560 --> 01:07:14,080
I am so glad to have gotten the chance to chat
1221
01:07:14,080 --> 01:07:17,000
and thank you for the awesome questions.
1222
01:07:17,000 --> 01:07:17,920
Really appreciate it.
1223
01:07:17,920 --> 01:07:19,520
Thanks so much.
1224
01:07:19,520 --> 01:07:20,360
Cool, cool.
1225
01:07:25,240 --> 01:07:27,360
Thank you for listening to this episode
1226
01:07:27,360 --> 01:07:28,920
of Learning from Machine Learning
1227
01:07:28,920 --> 01:07:31,000
with the remarkable Paige Bailey,
1228
01:07:31,000 --> 01:07:33,360
the lead product manager for generative models
1229
01:07:33,360 --> 01:07:35,000
at Google DeepMind.
1230
01:07:35,000 --> 01:07:37,360
Her work is pushing the boundaries of innovation
1231
01:07:37,360 --> 01:07:40,560
with Bard and the soon to be released Gemini.
1232
01:07:40,560 --> 01:07:43,640
Don't miss out on the valuable resources in the show notes.
1233
01:07:43,640 --> 01:07:46,240
Please leave a review, share it with your friends
1234
01:07:46,240 --> 01:07:49,400
and let's create a community of continuous learning.
1235
01:07:49,400 --> 01:08:14,000
Until next time, keep on learning.