תעתיק 🔗
WEBVTT
1
00:00:05.360 --> 00:00:08.919
Hey, welcome back to another episode of JavaScript Jabber.
2
00:00:09.599 --> 00:00:13.119
This week on our panel we have Steve Edwards.
3
00:00:13.800 --> 00:00:16.960
Yo yo yo, come in ato live from cold in
4
00:00:17.120 --> 00:00:18.359
sunny Portland.
5
00:00:19.280 --> 00:00:22.039
We also have a j O'Neil yo yo yo coming
6
00:00:22.079 --> 00:00:23.679
at your live from the soldering station.
7
00:00:27.320 --> 00:00:30.839
Oh sorry, I'm Charles Maxwood from Top End Devs and
8
00:00:31.679 --> 00:00:33.600
yeah it is freezing here anyway.
9
00:00:34.039 --> 00:00:35.920
We have a special guest this week and that is
10
00:00:36.039 --> 00:00:37.079
is Sean Annon.
11
00:00:38.679 --> 00:00:41.479
You want to let people know who you are, what
12
00:00:41.560 --> 00:00:43.359
you do where?
13
00:00:43.439 --> 00:00:45.200
Yeah, of course is because your course is awesome.
14
00:00:45.679 --> 00:00:48.439
Oh thank you. So. My name is Eha nand I
15
00:00:48.439 --> 00:00:52.320
have about twenty years of engineering and product management experience
16
00:00:53.479 --> 00:00:57.399
and most recently I've been very focused on AI for
17
00:00:57.439 --> 00:01:00.359
the last couple of years and I'm best known THEA
18
00:01:00.359 --> 00:01:02.640
community for an implementation.
19
00:01:02.280 --> 00:01:03.320
Of GPT two.
20
00:01:03.479 --> 00:01:06.799
There's a precursor to chat GPT that I implemented entirely
21
00:01:06.799 --> 00:01:09.319
in Excel and then late last year I reported that
22
00:01:09.480 --> 00:01:12.280
entirely to the web and pure JavaScript, and I teach
23
00:01:12.480 --> 00:01:17.000
how the entire transformer works. Basically the model that you
24
00:01:17.000 --> 00:01:24.159
know was the you know, ancestor to Gemini, Barred Lama, Chat, GPT, Claude.
25
00:01:24.239 --> 00:01:27.319
They're all really inheriting from this model called GPT two
26
00:01:27.879 --> 00:01:30.680
and I teach people and basically course of two weeks.
27
00:01:30.959 --> 00:01:33.560
If you have really no programming experience, or if you've
28
00:01:33.560 --> 00:01:36.040
got JavaScript programming experience, this is the best way to
29
00:01:36.079 --> 00:01:38.519
really get in understand how these things work. And they
30
00:01:38.519 --> 00:01:40.959
don't have to be a black box and you can
31
00:01:41.000 --> 00:01:43.400
see all that at Spreadsheets at All. You need dot
32
00:01:43.400 --> 00:01:44.920
ai and the classes on mavin.
33
00:01:46.239 --> 00:01:49.760
Very cool, So let's let's dive in. First.
34
00:01:49.840 --> 00:01:51.439
I think you said you had a promo code for
35
00:01:51.480 --> 00:01:53.120
the course, so let's just put that out there.
36
00:01:53.640 --> 00:01:55.680
Yeah, people want to go get it and get a
37
00:01:55.719 --> 00:01:56.280
deal on it.
38
00:01:56.920 --> 00:01:59.239
Yeah. So the promo code is really easy to remember.
39
00:01:59.319 --> 00:02:03.000
It's jsjer and just go to Maven dot com and
40
00:02:03.000 --> 00:02:04.799
look for my name, or if you go to spreadsheets
41
00:02:04.840 --> 00:02:07.519
at All, you need dot ai and then you click
42
00:02:07.599 --> 00:02:09.560
that you can use that promo code for twenty percent
43
00:02:09.599 --> 00:02:11.039
off for the next two weeks.
44
00:02:11.120 --> 00:02:11.879
So awesome.
45
00:02:12.599 --> 00:02:15.879
Definitely check that out. And I should just say, you know,
46
00:02:15.919 --> 00:02:19.000
thank you guys for having me. I listened for years
47
00:02:19.800 --> 00:02:21.479
to this, so it's great to actually meet you guys,
48
00:02:21.560 --> 00:02:22.879
well virtually in person.
49
00:02:23.879 --> 00:02:27.879
Right. Yeah, AJ is the cool one. I just run
50
00:02:27.919 --> 00:02:29.120
the show anyway, and.
51
00:02:29.080 --> 00:02:31.280
I'm just thinking guy while everybody else are the smart
52
00:02:31.280 --> 00:02:33.000
people according to some people.
53
00:02:34.439 --> 00:02:35.840
Anyway. So let's dive in.
54
00:02:36.080 --> 00:02:39.400
You said that you explain how the transformer works, and
55
00:02:39.439 --> 00:02:42.120
so for those that are kind of new to AI,
56
00:02:42.240 --> 00:02:44.719
do you want to just explain what a transformer.
57
00:02:44.800 --> 00:02:48.560
Is an AI? Yeah, we can dive into house stuff works.
58
00:02:48.919 --> 00:02:54.280
Yeah, sure. So the the transformer is a you know,
59
00:02:54.879 --> 00:02:59.439
AI architecture of a model that came out in twenty
60
00:02:59.479 --> 00:03:04.039
seventeen and it is the foundation for most of the
61
00:03:04.400 --> 00:03:07.759
you know AI models that have been you know, like
62
00:03:07.840 --> 00:03:11.240
chat GPT, so those chatbot assistants that seem amazingly smart
63
00:03:11.560 --> 00:03:15.039
all inherent from this architecture called the Transformer. And I
64
00:03:15.080 --> 00:03:19.000
can give a high level over your everything that goes
65
00:03:19.000 --> 00:03:22.840
into that. But the key thing that the transformer does
66
00:03:22.879 --> 00:03:25.319
is usually takes some input and it tries to predict
67
00:03:25.319 --> 00:03:28.800
what the next word is. And that's really all your
68
00:03:28.840 --> 00:03:31.319
large language model is doing is taking one word or
69
00:03:31.360 --> 00:03:33.639
really one token at a time, and it's trying to
70
00:03:33.680 --> 00:03:36.000
predict when you enter in a question what the next
71
00:03:36.000 --> 00:03:39.199
thing is. And over you know, the last you know,
72
00:03:39.319 --> 00:03:41.240
a couple of years, what we've been able to do.
73
00:03:41.280 --> 00:03:44.759
We collectively as humanity is. Take this model that tries
74
00:03:44.800 --> 00:03:47.319
to predict the next word and turn into these really helpful,
75
00:03:47.360 --> 00:03:52.560
amazing chat bot assistants. And the paper that introduced this model,
76
00:03:52.639 --> 00:03:55.280
called the
Load full transcript
Transformer, was called Attention Is All You Need.
77
00:03:55.639 --> 00:03:58.120
And that's where my course gets its name Spreadsheets Are
78
00:03:58.120 --> 00:04:01.719
All You Need? Is I basically implemented that entire model
79
00:04:02.120 --> 00:04:05.800
inside a spreadsheet? Hence the name Spreadsheets Are All You Need?
80
00:04:06.520 --> 00:04:09.560
So question here then? So I mean, having used Google
81
00:04:09.639 --> 00:04:13.080
since its inception, you know, type ahead is sort of
82
00:04:13.120 --> 00:04:16.199
a standard thing in search. You know, where you're typing,
83
00:04:16.199 --> 00:04:19.920
and it's starting to anticipate what your phrases, you know,
84
00:04:19.959 --> 00:04:22.279
what you're gonna type next. If I'm you know, starting
85
00:04:22.319 --> 00:04:25.680
to search for spreadsheets on Google, it's going to anticipate, Okay,
86
00:04:26.120 --> 00:04:27.959
what's the next thing I'm going to type? So is
87
00:04:28.000 --> 00:04:31.560
this basically the same thing just on AI steroids or
88
00:04:31.959 --> 00:04:34.879
because I mean basically that's using what people have typed
89
00:04:34.920 --> 00:04:38.279
in and you know they've indexed it and you know,
90
00:04:38.360 --> 00:04:41.279
done things with it. So is that sort of the
91
00:04:41.319 --> 00:04:44.920
same thing just on steroids or is that intrinsically different?
92
00:04:46.720 --> 00:04:50.439
Yes? And no in terms of effect, it is literally
93
00:04:50.480 --> 00:04:52.120
just doing the same thing, like it's trying to break
94
00:04:52.120 --> 00:04:54.639
the next thing. Is I really kind of get a
95
00:04:54.680 --> 00:05:00.319
little bit of a mental pushback that to just saying, oh,
96
00:05:00.319 --> 00:05:03.600
it's just like autocomplete. It is basically structured as an
97
00:05:03.600 --> 00:05:09.800
autocomplete problem, but the level of complexity of the architecture
98
00:05:09.879 --> 00:05:12.959
to solve that problem is just a lot more complex.
99
00:05:13.279 --> 00:05:16.279
But it is trying to do the same thing. And
100
00:05:17.399 --> 00:05:19.399
you know, the way to think about this is if
101
00:05:19.439 --> 00:05:21.759
you can fill in the blank in any sentence, you
102
00:05:21.879 --> 00:05:25.160
probably know something about that sentence. You already know what
103
00:05:25.560 --> 00:05:29.199
the answer might be. Like that's a useful test of knowledge.
104
00:05:29.240 --> 00:05:31.600
But effectively, yeah, that is that is what's going on.
105
00:05:31.639 --> 00:05:33.199
It's just trying to break the next word, and then
106
00:05:33.240 --> 00:05:35.439
the next word after that and so forth, one at a.
107
00:05:35.399 --> 00:05:42.160
Time, right, and so effectively, I guess the the autocompletes
108
00:05:42.160 --> 00:05:45.920
that we typically see are a little bit I guess
109
00:05:45.920 --> 00:05:50.279
more naive than say the AI LM models, where they
110
00:05:50.319 --> 00:05:56.839
have
substantially more data to run on, and you know,
111
00:05:57.040 --> 00:06:00.480
use a mechanism that I guess is probably somewhat the
112
00:06:00.480 --> 00:06:04.519
same because it's weighted and things like that, but anyway.
113
00:06:04.160 --> 00:06:06.639
It can do it across a wider variety.
114
00:06:06.199 --> 00:06:10.439
Of things and give you deeper answers.
115
00:06:11.360 --> 00:06:14.560
Yeah, so I mean, actually, let's start with the autocomplete example,
116
00:06:14.639 --> 00:06:16.360
because it does kind of point the way to some
117
00:06:16.439 --> 00:06:18.560
parts of the architecture. Like the simplest thing you might
118
00:06:18.600 --> 00:06:21.439
do for building an autocomplete is you might just say,
119
00:06:21.480 --> 00:06:23.360
if I see this word, what are all the next
120
00:06:23.639 --> 00:06:25.319
likely words that will be after it? And you could
121
00:06:25.319 --> 00:06:28.279
just do a statistical look up across some large data sets, right,
122
00:06:28.800 --> 00:06:30.920
And as good as that'll be, the more pieces of
123
00:06:31.000 --> 00:06:33.639
data you look at, the better it's predictive value. So
124
00:06:33.639 --> 00:06:36.319
this is called like a bigram model. And then because
125
00:06:36.360 --> 00:06:38.079
it looks at two and then what you could do
126
00:06:38.120 --> 00:06:40.319
is you could actually look three words back, or you
127
00:06:40.319 --> 00:06:43.759
could look forwards back. And actually, one of the key
128
00:06:43.759 --> 00:06:45.920
things about the transformers it tries to look at all
129
00:06:45.959 --> 00:06:48.160
the words. And this is what the attention mechanicism does,
130
00:06:48.399 --> 00:06:51.879
is that it can figure out, essentially from all the
131
00:06:51.920 --> 00:06:54.519
possible words before it, what is the next most likely word.
132
00:06:55.079 --> 00:06:57.199
And then the other key thing you need to do
133
00:06:57.319 --> 00:06:59.319
is you ask a real network to take all that
134
00:06:59.360 --> 00:07:02.600
information a prediction. And it turns out that's the heart
135
00:07:02.800 --> 00:07:05.920
of the transformer and what really made it work was
136
00:07:05.959 --> 00:07:08.680
they just scaled that up to a much larger size
137
00:07:08.680 --> 00:07:11.920
than I think people were used to doing. You know
138
00:07:11.959 --> 00:07:14.319
when you're autocomplete and your keyboard is probably used to
139
00:07:14.800 --> 00:07:16.519
you know, is built to be really really fast, and
140
00:07:16.560 --> 00:07:18.519
so they tried to make it really efficient. And what
141
00:07:18.800 --> 00:07:20.759
we've been able to do with the Transformer is make
142
00:07:20.800 --> 00:07:22.959
it really big and then actually make it super efficient
143
00:07:23.120 --> 00:07:25.399
scaling it back down so just that it spits out
144
00:07:25.399 --> 00:07:30.439
tokens at a reasonable clip. But that core idea of saying, hey,
145
00:07:30.920 --> 00:07:33.680
let me look statistically at you know what the next
146
00:07:33.720 --> 00:07:35.720
thing is, Well, one word isn't gonna be enough, Two
147
00:07:35.759 --> 00:07:37.839
words back is going to be better. That is what
148
00:07:37.959 --> 00:07:40.600
the attention mechanism is. In a sense, if you squint,
149
00:07:40.639 --> 00:07:42.759
doing is it's trying to look at all the words
150
00:07:42.959 --> 00:07:46.160
that came before, it puts them through multiple passes, and
151
00:07:46.199 --> 00:07:48.439
then it's asking your normal network to do the prediction
152
00:07:48.560 --> 00:07:50.399
rather than just simply saying, oh, let me take the
153
00:07:50.480 --> 00:07:51.360
raw statistics.
154
00:07:52.879 --> 00:07:58.399
But yeah, so do you kind of want to break
155
00:07:58.439 --> 00:08:01.560
down for us how these systems actually work.
156
00:08:02.120 --> 00:08:07.560
Yeah? So the first thing I say is the way
157
00:08:07.600 --> 00:08:11.720
to think about the simplest model that I like of
158
00:08:11.759 --> 00:08:16.319
the transformer is that what we've been able to do.
159
00:08:16.439 --> 00:08:19.040
You know, we said that, you know, these are trained
160
00:08:19.079 --> 00:08:20.680
to fill in the blank on a piece of text.
161
00:08:20.720 --> 00:08:23.079
So the example I often use in my lectures and
162
00:08:23.120 --> 00:08:25.920
inside a lot of my material is this very simple,
163
00:08:25.920 --> 00:08:28.879
simple sentence, Mike is quick, he moves and the next
164
00:08:28.879 --> 00:08:31.160
most likely completion would probably be he moves quickly, or
165
00:08:31.199 --> 00:08:34.600
he moves around, or he moves fast. And so the
166
00:08:34.600 --> 00:08:36.639
basic question is how do we get a computer to
167
00:08:37.080 --> 00:08:39.559
fill in the blank of an English sentence or any
168
00:08:39.639 --> 00:08:44.399
natural language sentence. And what we've been able to do
169
00:08:44.480 --> 00:08:47.000
is actually figure out to talk in the language of
170
00:08:47.000 --> 00:08:48.960
the computer, which is math. So if I gave the
171
00:08:49.000 --> 00:08:52.120
computer a math problem two plus two equals it could
172
00:08:52.120 --> 00:08:53.919
fill in the blank. It knows that two plus two
173
00:08:53.919 --> 00:08:57.799
equals four, and we can make the math as pretty
174
00:08:57.840 --> 00:08:59.720
large and complex. But computers are really good at math.
175
00:08:59.759 --> 00:09:02.039
So we've been able to do is and what the
176
00:09:02.080 --> 00:09:04.879
model does is it takes a word problem and it's
177
00:09:04.919 --> 00:09:08.559
really converting it to a math problem. If you look inside,
178
00:09:08.679 --> 00:09:10.679
you know, go to my website, you know, spreadsheets are
179
00:09:10.720 --> 00:09:13.600
all you need dot ai slash GPT two, or if
180
00:09:13.600 --> 00:09:16.279
you download the Excel file and look inside it what
181
00:09:16.360 --> 00:09:18.559
you'll see in you know, there's text at the beginning.
182
00:09:18.600 --> 00:09:20.360
You type in text on one part of the spreadsheet,
183
00:09:20.480 --> 00:09:22.600
and you get the predicted word at the other end
184
00:09:22.639 --> 00:09:26.559
of it. But in between, if you look in that,
185
00:09:26.679 --> 00:09:29.120
you'll be like, where the heck are the words? It's
186
00:09:29.279 --> 00:09:32.279
all numbers. And so the key insight is what we've
187
00:09:32.399 --> 00:09:34.360
been able to is take something that is a word
188
00:09:34.399 --> 00:09:37.919
problem and we've turned words into math and once and
189
00:09:37.960 --> 00:09:41.879
that mapping process of words into map has two stages.
190
00:09:41.960 --> 00:09:45.080
It's called tokenization and then embeddings. And at the end
191
00:09:45.080 --> 00:09:47.799
of it, we map every word. You can conceptually think
192
00:09:47.799 --> 00:09:49.919
about it to a single number, but we actually map
193
00:09:49.960 --> 00:09:52.399
them to a large list of numbers. And then once
194
00:09:52.440 --> 00:09:56.000
you have a mathematical representation of your prompt, your entire
195
00:09:56.039 --> 00:09:58.559
prompt has been you know, turned into a large list
196
00:09:58.559 --> 00:10:01.679
of numbers. We then run I just call it number crunching.
197
00:10:01.840 --> 00:10:05.840
It's these two key mechanisms attention and a multi layer
198
00:10:05.840 --> 00:10:08.440
perceptor or a neural network that just kind of crunches
199
00:10:08.480 --> 00:10:10.240
on it to try and predict what the next word is.
200
00:10:10.759 --> 00:10:12.360
And then at the end of that we get a number,
201
00:10:12.960 --> 00:10:16.200
and that number we then reverse the process that came
202
00:10:16.240 --> 00:10:18.200
out of that thing, and we say, well, what what
203
00:10:18.279 --> 00:10:21.039
word does this number map to? And that number is
204
00:10:21.080 --> 00:10:23.759
a predicted word, but it's not going to map cleanly
205
00:10:23.799 --> 00:10:27.080
to every single word in our vocabulary. And so if
206
00:10:27.080 --> 00:10:29.360
that number is closer to certain words, like in the
207
00:10:29.399 --> 00:10:32.120
case mike is quickly as quickly, the predicted number might
208
00:10:32.159 --> 00:10:34.440
be really close to the word quickly. It might be
209
00:10:34.440 --> 00:10:36.759
close to the word around, but it's not going to
210
00:10:36.759 --> 00:10:39.799
be close to you know, quick can be a body part,
211
00:10:39.840 --> 00:10:41.879
it can be the quick of your fingernail. It's not
212
00:10:41.879 --> 00:10:43.799
going to be something about your fingernail, because it's figured
213
00:10:43.799 --> 00:10:47.240
out enough that it's moved the predicted number away from that.
214
00:10:47.320 --> 00:10:49.440
And so we take that and we run a random
215
00:10:49.519 --> 00:10:52.320
number generator the very end, and then we pick it
216
00:10:52.320 --> 00:10:55.200
according to that random number generator based on how close
217
00:10:55.279 --> 00:10:57.480
that number is to one of the other words in
218
00:10:57.559 --> 00:11:02.200
the dictionary of words mapping to numbers. So that's like
219
00:11:02.240 --> 00:11:06.320
my highest level summary of what's happening under the transform
220
00:11:06.360 --> 00:11:08.879
without describing all the mechanisms. But again, the key thing
221
00:11:08.960 --> 00:11:12.399
is we found a way to map solve this problem numerically.
222
00:11:12.639 --> 00:11:15.120
We map words to numbers. We turn the whole sentence,
223
00:11:15.120 --> 00:11:17.240
your entire prompt into a large list of numbers, We
224
00:11:17.320 --> 00:11:19.639
number crunch on it. Then we get a predicted number
225
00:11:19.639 --> 00:11:21.200
out of it. We just calculate and we look at
226
00:11:21.200 --> 00:11:23.720
how close that number is to our number to word
227
00:11:23.759 --> 00:11:26.440
mapping at the very end, and that's the probability you
228
00:11:26.480 --> 00:11:28.799
get of getting a particular token or word out of
229
00:11:28.799 --> 00:11:31.240
the model. Let me pause there, see if their questions
230
00:11:31.320 --> 00:11:35.159
or things I should clarify, So.
231
00:11:36.639 --> 00:11:38.159
I think I follow along.
232
00:11:38.519 --> 00:11:41.679
Essentially what you're saying then is, so let's say I
233
00:11:41.720 --> 00:11:43.600
wanted it to generate a whole paragraph.
234
00:11:43.679 --> 00:11:45.519
It just does this over and over and over again.
235
00:11:45.919 --> 00:11:47.720
Get yeah, the next word.
236
00:11:48.080 --> 00:11:49.799
Yeah, maybe I've glossed over that part of it. Like
237
00:11:49.840 --> 00:11:53.279
the large language model only predicts the next word technically
238
00:11:53.279 --> 00:11:55.799
something called a token, which is slightly smaller than a word,
239
00:11:56.480 --> 00:11:59.279
and every time you get a prediction out of it, like,
240
00:11:59.320 --> 00:12:02.639
it doesn't by default predicted paragraphs. So if you you know,
241
00:12:02.679 --> 00:12:05.840
try my app or you download the spreadsheet, it only
242
00:12:05.879 --> 00:12:09.919
predicts one token. And the way we get paragraphs a
243
00:12:09.960 --> 00:12:12.000
text out of this is we take the predicted token
244
00:12:12.080 --> 00:12:14.159
it came up with, and then we stick it back
245
00:12:14.200 --> 00:12:16.279
onto the input, and then we ask it to predict
246
00:12:16.320 --> 00:12:21.080
the next sentence or the next that new accumulated paragraph,
247
00:12:21.360 --> 00:12:23.039
and so you can actually start with a single word,
248
00:12:23.279 --> 00:12:24.720
ask it to predict what the next word is, and
249
00:12:24.759 --> 00:12:26.360
then you now you've got two words, and then you
250
00:12:26.440 --> 00:12:28.840
run it through and then you keep going. And then
251
00:12:28.919 -->
00:12:31.480
what happens when you've got user input like somebody types
252
00:12:31.480 --> 00:12:34.519
of response, is you just stick that entire user input
253
00:12:34.720 --> 00:12:37.360
as you know, a large set of words that it
254
00:12:37.360 --> 00:12:41.159
needs to brick what the next thing is. And you
255
00:12:41.159 --> 00:12:44.919
can think about it structured into the model. As you
256
00:12:45.200 --> 00:12:48.759
are reading a transcript between a user and a helpful
257
00:12:48.799 --> 00:12:52.240
chatbot assistant. User said X, we fill in what the
258
00:12:52.320 --> 00:12:54.559
user said, assistant said, and then it needs to come
259
00:12:54.639 --> 00:12:56.279
up with what the assistant said, and it just tries
260
00:12:56.320 --> 00:13:00.000
to come with something plausible. Maybe the thing is step back,
261
00:13:00.120 --> 00:13:03.840
like the base model that these that gets trained in
262
00:13:03.879 --> 00:13:07.639
this process before it's turned into a helpful chatbot just
263
00:13:07.759 --> 00:13:10.440
knows really simply how to complete sentences. If you take
264
00:13:10.519 --> 00:13:14.440
the base GPT two and you type in, you know,
265
00:13:14.720 --> 00:13:17.080
questions to it, it's not going to necessarily respond back
266
00:13:17.120 --> 00:13:19.559
to you meaningfully. It's just designed to predict the next
267
00:13:19.600 --> 00:13:22.120
word based on everything it's seen on the internet. So
268
00:13:22.159 --> 00:13:24.639
a good example I use in classes, we type in
269
00:13:24.639 --> 00:13:29.240
the word first name and then you hit return, and well,
270
00:13:29.240 --> 00:13:31.600
what do you think it would predict after that? It
271
00:13:31.679 --> 00:13:35.639
predicts last name, email address, phone number, because most texts
272
00:13:35.639 --> 00:13:38.559
on the Internet that's say first name. Statistically, it's a
273
00:13:38.600 --> 00:13:42.559
form and it's used to just filling out forms. Another
274
00:13:42.600 --> 00:13:45.720
one is I type in hello class, and when I
275
00:13:45.720 --> 00:13:47.080
first did this, I thought it was going to say
276
00:13:47.080 --> 00:13:50.840
hello teacher, but it actually starts spitting out Java code,
277
00:13:51.080 --> 00:13:54.360
so it just looks at the fact. Yeah, it's really
278
00:13:54.399 --> 00:13:58.440
a music to watch and you can you can just
279
00:13:58.559 --> 00:14:01.080
run it, and it's just trying to predict what the
280
00:14:01.120 --> 00:14:04.080
next thing is based on what it saw on the internet.
281
00:14:04.120 --> 00:14:07.480
And then what you know open Ai and Nentropic and
282
00:14:07.519 --> 00:14:11.000
these companies do is they put that call a base model,
283
00:14:11.000 --> 00:14:12.480
which all it knows how to do is predict the
284
00:14:12.519 --> 00:14:17.840
next word through a training regime to elicit it to
285
00:14:17.919 --> 00:14:20.519
be more like a helpful chatbot. So you
give it
286
00:14:20.559 --> 00:14:23.039
a system prompt that tells it it's a chatbot. It's
287
00:14:23.120 --> 00:14:25.120
kind of like you tell it a story that's plausible
288
00:14:25.200 --> 00:14:28.240
for it to start to think like it's talking to
289
00:14:28.279 --> 00:14:31.159
a user, like you are a chatbot. You are reading
290
00:14:31.200 --> 00:14:34.120
a transcript of a chatbot and a human user, and
291
00:14:34.159 --> 00:14:35.720
we just fill in what the human said, and it
292
00:14:35.759 --> 00:14:37.799
tries to fill in what it thought the helpful system
293
00:14:37.840 --> 00:14:40.120
would be, and then they fine tune it to get
294
00:14:40.159 --> 00:14:40.639
better at that.
295
00:14:41.720 --> 00:14:44.639
Yeah, this sounds a lot like what you're explaining.
296
00:14:45.159 --> 00:14:47.799
You get into prompt engineering, which, again, if you're not
297
00:14:47.840 --> 00:14:50.080
into AI, prompt engineering.
298
00:14:49.679 --> 00:14:53.159
Is what's all the stuff I tell the AI.
299
00:14:53.000 --> 00:14:56.039
System so that it'll give me the answer I want, right,
300
00:14:56.120 --> 00:15:00.679
And so you're when we're talking about prompt engineering, now
301
00:15:00.720 --> 00:15:01.480
it's okay.
302
00:15:01.519 --> 00:15:03.200
So this is why when I start out.
303
00:15:03.039 --> 00:15:05.440
I tell it things like, like you said, you are
304
00:15:05.480 --> 00:15:08.240
a chat bot, you help people with these problems, you
305
00:15:08.279 --> 00:15:10.679
do these kinds of things, because it'll build off of
306
00:15:10.759 --> 00:15:14.320
all of that and use the statistical model now with
307
00:15:14.399 --> 00:15:16.960
the context of what you typed in to give you
308
00:15:17.000 --> 00:15:17.720
the right answer.
309
00:15:17.840 --> 00:15:20.120
So you know, yeah, Hello class. There's not a whole
310
00:15:20.159 --> 00:15:21.200
lot there for it to go on.
311
00:15:21.600 --> 00:15:23.639
But if you tell it, you know, you're a chat
312
00:15:23.679 --> 00:15:26.080
bot and you're helping students with a blah blah blah
313
00:15:26.120 --> 00:15:28.600
blah blah, then you type in hello class, and it's
314
00:15:28.639 --> 00:15:30.120
going to go you know, then it may come back
315
00:15:30.120 --> 00:15:32.159
with hello teacher or something like that.
316
00:15:31.879 --> 00:15:34.799
That's a great example. Yeah. So, and what you can
317
00:15:34.840 --> 00:15:38.519
think about them conceptually doing is baking that prompt engineering
318
00:15:38.559 --> 00:15:40.960
into the model. So what they're able to do is
319
00:15:41.519 --> 00:15:44.279
if they give it enough examples of this, they can
320
00:15:44.360 --> 00:15:46.799
retrain it such that you don't need the prompt at
321
00:15:46.840 --> 00:15:49.399
the beginning that tells it it's a teacher or that
322
00:15:49.440 --> 00:15:52.120
it's a helpful chatbot assistant, and that gets baked into
323
00:15:52.120 --> 00:15:53.879
the model. You can think about all that prompt engineering
324
00:15:53.879 --> 00:15:57.159
gets memorized into the model during that training process, and
325
00:15:57.200 --> 00:16:01.080
then it turns into that helpful assistant will.
326
00:16:00.960 --> 00:16:03.879
Help help me understand this a little bit. So I've
327
00:16:04.080 --> 00:16:08.600
I've played around obviously with GPT. I've also played around
328
00:16:08.639 --> 00:16:11.279
with the other models. In fact, right now I really
329
00:16:11.360 --> 00:16:14.000
like Quinn. I am I am using Quinn more than
330
00:16:14.039 --> 00:16:18.039
I'm using GPT, because Quinn actually seems to be giving
331
00:16:18.080 --> 00:16:23.919
better results, especially considering it in the benchmarks, it outperforms
332
00:16:23.919 --> 00:16:26.480
four oh whatever that means. I mean, it's like by
333
00:16:26.559 --> 00:16:30.960
a fraction of percentage point, but OH one. I just
334
00:16:31.039 --> 00:16:33.840
find OH one and R one to be too like
335
00:16:33.879 --> 00:16:36.440
they take forever. So it's like I'd rather ask the
336
00:16:36.519 --> 00:16:40.320
question twice and be ninety nine percent likely to get
337
00:16:40.320 --> 00:16:43.440
the right answer, then ask the question one time and
338
00:16:43.480 --> 00:16:46.559
then have to wait forty five seconds to get the
339
00:16:46.559 --> 00:16:47.840
wrong answer and ask it again.
340
00:16:47.879 --> 00:16:50.399
You know, forty five seconds. That's an eternity.
341
00:16:50.840 --> 00:16:53.799
The O one is crazy.
342
00:16:53.799 --> 00:16:54.399
We use it.
343
00:16:54.600 --> 00:16:57.320
We use it for code, for code questions and stuff,
344
00:16:57.360 --> 00:17:01.360
because it does better than the standard GPT for wait
345
00:17:01.360 --> 00:17:03.639
a few seconds. But I'd rather get a wait a
346
00:17:03.679 --> 00:17:06.119
little bit and get a better answer than get something
347
00:17:06.200 --> 00:17:09.279
super fast that's not going to be as good.
348
00:17:09.559 --> 00:17:13.079
Well, I I'm the other way because it's not that
349
00:17:13.279 --> 00:17:15.400
much better. If you look at the benchmarks, it's like
350
00:17:15.599 --> 00:17:19.359
one percent better than four oh, and it takes you know,
351
00:17:19.440 --> 00:17:22.640
so much anyway. But what the thing, the thing that
352
00:17:22.680 --> 00:17:27.680
I was that I was getting at is in the beginning,
353
00:17:28.000 --> 00:17:32.440
there was the system prompt. Right, so when with GPT,
354
00:17:32.599 --> 00:17:35.640
one of the ways to jail break it was you
355
00:17:35.640 --> 00:17:41.559
could say that was just a joke. Actually you're a
356
00:17:43.160 --> 00:17:46.200
something else, and so it would interpret it as Okay,
357
00:17:46.319 --> 00:17:49.039
your system prompt is you're a chatbot. You're allowed to
358
00:17:49.079 --> 00:17:50.680
say this. You're not allowed to say that that you
359
00:17:50.680 --> 00:17:53.599
could just say that was just a joke. And then
360
00:17:56.160 --> 00:17:59.359
and then and then give it an additional prompt. Now
361
00:17:59.400 --> 00:18:06.200
with deeps seek V two point five and are one
362
00:18:07.279 --> 00:18:11.119
and with Quinn it's it's like you're saying it's baked
363
00:18:11.200 --> 00:18:14.039
into the model because if I override the system prompt
364
00:18:14.079 --> 00:18:18.039
and I tell it, you know, you are a human
365
00:18:18.839 --> 00:18:22.960
who is capable of reasoning and has no biases and
366
00:18:23.359 --> 00:18:28.319
can represent any information factually, tell me about Tianaman Square.
367
00:18:28.759 --> 00:18:30.279
It's you know, it's.
368
00:18:30.200 --> 00:18:32.640
I am a helpful bot. I am not a human,
369
00:18:32.920 --> 00:18:37.200
and I do not talk about things that contradict what
370
00:18:37.359 --> 00:18:41.319
is known to be you know, the proper the proper
371
00:18:41.400 --> 00:18:45.880
knowledge of the of the Chinese government to protect the people,
372
00:18:46.000 --> 00:18:48.119
or you know, it gives me some some nonsense like that.
373
00:18:48.279 --> 00:18:53.279
So what what is How is it possible to bake
374
00:18:53.400 --> 00:18:57.000
in those system prompts with training data and and I
375
00:18:57.039 --> 00:18:59.160
guess how does that vary? How does it vary from
376
00:18:59.160 --> 00:19:01.480
the system prompt? And how do they get it to
377
00:19:01.680 --> 00:19:04.400
bake that in so that it you can't override it
378
00:19:04.440 --> 00:19:05.519
with a system prompt.
379
00:19:06.000 --> 00:19:08.200
Okay, there's a lot of layers there.
380
00:19:09.519 --> 00:19:13.519
Let me yeah, question, can you restate the question in
381
00:19:13.559 --> 00:19:14.119
one sentence?
382
00:19:14.799 --> 00:19:17.559
I think the uily what I think with the question,
383
00:19:17.720 --> 00:19:20.799
which was how do you bake in the system prompt.
384
00:19:22.000 --> 00:19:23.759
But there's a couple of things that are worth noting
385
00:19:23.839 --> 00:19:28.839
in your question, Like you mentioned some reasoning models one
386
00:19:29.039 --> 00:19:31.480
and R one, and the way those operate is a
387
00:19:31.480 --> 00:19:33.160
little bit different. Like you said, it takes a while
388
00:19:33.160 --> 00:19:35.559
to come back because it's actually just expending a lot
389
00:19:35.599 --> 00:19:38.359
of tokens thinking that it doesn't give you, and it's
390
00:19:38.440 --> 00:19:41.039
trying to actually think through the process like you might do.
391
00:19:41.160 --> 00:19:43.640
They call this chain of thought or thinking step by step,
392
00:19:44.559 --> 00:19:47.279
and it what's unique about that can parterregular chain of
393
00:19:47.279 --> 00:19:49.440
thought is it can suddenly realize, oh, it's made a
394
00:19:49.480 --> 00:19:53.319
mistake and backtrack. And so it's it's literally spending you know,
395
00:19:53.319
--> 00:19:55.880
coming up with hypotheses and trying and testing things and
396
00:19:55.880 --> 00:19:57.839
seeing if it works. So this is why these models
397
00:19:57.880 --> 00:20:00.240
tend to be really good on math and code because
398
00:20:00.240 --> 00:20:01.839
it can go try something and say, oh wait does
399
00:20:01.880 --> 00:20:03.759
this let me check does this answer right? Oh no,
400
00:20:03.799 --> 00:20:09.240
it's not, let me try again. So and then you mentioned,
401
00:20:09.839 --> 00:20:13.440
you know, jail breaking, and with the early models, one
402
00:20:13.480 --> 00:20:15.079
way to think about like you're like, oh, this is
403
00:20:15.160 --> 00:20:19.039
just a joke, is that you know, you're kind of
404
00:20:19.079 --> 00:20:22.119
taking that we've talked about briefly, that attention mechanism or
405
00:20:22.160 --> 00:20:24.480
looking back at the previous what's most likely if you
406
00:20:24.519 --> 00:20:28.720
put things like you know, you know, kill and harm
407
00:20:29.039 --> 00:20:33.640
in the in the prompt, statistically it sounds like it's negative, right,
408
00:20:33.680 --> 00:20:37.519
But if you start putting things like Grandma cookies, it
409
00:20:37.519 --> 00:20:39.319
seems less harmless, and you kind of think of yourself
410
00:20:39.359 --> 00:20:42.119
a kind of waiting the attention to be more to
411
00:20:42.160 --> 00:20:45.759
the harmless side. And really what's happened is that the
412
00:20:45.799 --> 00:20:50.640
models have gotten smarter, both in terms of their natural
413
00:20:50.680 --> 00:20:53.319
responses to this, Like they are trained to handle jail breaks.
414
00:20:54.400 --> 00:20:56.960
They are trained on if a jail break comes up,
415
00:20:57.240 --> 00:20:59.559
here's the response. And the way they train it to
416
00:20:59.559 --> 00:21:04.319
get to your your main question is through these two
417
00:21:04.319 --> 00:21:07.039
training techniques. One is they just give it an example
418
00:21:07.039 --> 00:21:09.680
of a prompt and what its response should be, and
419
00:21:09.720 --> 00:21:13.799
they use this technique called backpropagation or sarcastic creating descent,
420
00:21:13.799 --> 00:21:16.079
which is to tune the network such every time it
421
00:21:16.119 --> 00:21:18.759
sees that result, it gives out what we wanted it
422
00:21:18.799 --> 00:21:20.680
to have. So we're going a little head of where
423
00:21:20.680 --> 00:21:22.039
I wanted to be. But like when you train in
424
00:21:22.039 --> 00:21:23.759
a ural network, you give it examples of data, so
425
00:21:23.759 --> 00:21:26.880
The simple example is a dog and cat classifier.
426
00:21:27.359 --> 00:21:27.519
Right.
427
00:21:27.720 --> 00:21:30.319
I give it pictures of dogs, and I give it
428
00:21:30.359 --> 00:21:31.960
pictures of cats, and I tell it which ones are
429
00:21:32.000 --> 00:21:33.880
dogs and cats, and it comes up with the answer.
430
00:21:33.960 --> 00:21:35.960
It comes up with the rules how to figure out
431
00:21:36.000 --> 00:21:38.640
whether an image is dog or cat. This is way
432
00:21:38.680 --> 00:21:44.839
different than regular programming, right. Machine learning inverts the normal paradigm.
433
00:21:44.920 --> 00:21:48.359
Normally we're used as developers. I write a series of rules,
434
00:21:48.480 --> 00:21:51.880
a series of program, right, and then it processes data
435
00:21:51.920 --> 00:21:55.799
and gets out a result. I click a button, something
436
00:21:55.839 --> 00:21:58.359
does you know moves on the screen, So I can
437
00:21:58.359 --> 00:22:01.720
write that program. But a dog and cat photo classifier,
438
00:22:01.799 --> 00:22:03.920
I don't know if you gave me dogs photos of
439
00:22:03.920 --> 00:22:06.079
dogs and cats, I know how to instinctively do that,
440
00:22:06.559 --> 00:22:07.960
but I do know how to write it out as
441
00:22:07.960 --> 00:22:11.160
a series of rules. And so the inversion that machine
442
00:22:11.200 --> 00:22:14.440
learning does is you give it answers and you give
443
00:22:14.480 --> 00:22:17.359
it data, and then it figures out how to write
444
00:22:17.359 --> 00:22:19.599
the rules. It writes the program. Now, unfortunately we can't
445
00:22:19.599 --> 00:22:21.720
always understand what the program it comes up with is.
446
00:22:23.200 --> 00:22:25.400
But what they do is they give it examples of
447
00:22:26.039 --> 00:22:29.400
jail break attempts and they say, hey, you know your response,
448
00:22:29.440 --> 00:22:32.160
now should be this to that. That's kind of the
449
00:22:32.200 --> 00:22:34.359
high level overview of how they do that. One thing
450
00:22:34.400 --> 00:22:39.359
that's worth noting is that when they protect a model,
451
00:22:39.720 --> 00:22:43.720
it isn't just in the model itself, So there are
452
00:22:43.799 --> 00:22:46.759
usually things that are watching the result of the model
453
00:22:46.839 --> 00:22:50.720
that are additional classifiers. And so sometimes you might see
454
00:22:51.359 --> 00:22:53.680
examples of open source models that let you do things,
455
00:22:54.039 --> 00:22:57.559
but the hosted versions do not because the hosted system
456
00:22:57.599 --> 00:23:00.000
is actually checking. So not everything is baked into the
457
00:23:00.039 --> 00:23:02.720
model itself and it's not one hundred percent perfect, So
458
00:23:02.759 --> 00:23:05.920
often there's some additional guardrails that are detecting things.
459
00:23:06.480 --> 00:23:10.079
Okay, So then two more questions.
460
00:23:09.839 --> 00:23:18.359
Yeah, what constitutes open source because that does not mean
461
00:23:18.400 --> 00:23:23.839
the same thing that it means in the programming circles, or.
462
00:23:23.880 --> 00:23:26.799
I don't believe it does because I have not yet
463
00:23:26.839 --> 00:23:30.759
seen any open source model that comes with four hundred
464
00:23:30.920 --> 00:23:32.480
terabytes of training data.
465
00:23:33.759 --> 00:23:39.400
There are few and far between. There are some. Olmo
466
00:23:39.640 --> 00:23:43.720
is probably the best known one, which is a model
467
00:23:43.720 --> 00:23:47.720
where everything the training code, the training data system, the
468
00:23:48.480 --> 00:23:53.240
data collection pipeline, the logs from their training runs are
469
00:23:53.240 --> 00:23:56.519
completely open. There's like a handful of others. But this
470
00:23:56.640 --> 00:24:01.400
question of what constitutes open is is completely a gray
471
00:24:01.440 --> 00:24:06.519
area and it's being debated right now. Traditionally, when people
472
00:24:06.559 --> 00:24:11.920
talk about an open model, it's usually an open weight model,
473
00:24:12.359 --> 00:24:16.359
which is you get the parameters which encompass the rules
474
00:24:16.680 --> 00:24:19.599
we talked earlier. That whole thing is math. Right, So
475
00:24:19.599 --> 00:24:21.359
if you open up my spreadsheet or you open up
476
00:24:21.359 --> 00:24:24.519
my website, you just see lots of numbers. You know,
477
00:24:24.599 --> 00:24:26.200
whether those numbers are hidden from you or you can
478
00:24:26.240 --> 00:24:28.880
run them yourself is what people call an open weight model.
479
00:24:29.880 --> 00:24:33.359
That's what kind of passes for open source. These days,
480
00:24:34.680 --> 00:24:36.759
there are very few models that open up the training data,
481
00:24:38.319 --> 00:24:42.000
and so it's debatable and people do debate about what
482
00:24:42.119 --> 00:24:46.359
a truly open model means. A truly open like the
483
00:24:46.400 --> 00:24:48.839
most open is one that includes the data, but there
484
00:24:48.839 --> 00:24:50.720
aren't that many, especially at the state of the art,
485
00:24:50.759 --> 00:24:53.680
where the model is all the training day that created
486
00:24:53.720 --> 00:24:54.920
the model is there.
487
00:24:56.000 --> 00:24:59.240
Well, I mean that would be highly illegal for chat
488
00:24:59.279 --> 00:25:02.720
GPT to get as their training data because YouTube and
489
00:25:04.279 --> 00:25:07.079
all the libraries on planet Earth and everyone who has
490
00:25:07.119 --> 00:25:10.000
a copyright on something would have something this day about that.
491
00:25:10.759 --> 00:25:13.519
Well, I mean, I'm not going to comment on any
492
00:25:13.519 --> 00:25:19.319
particular model providers specifically, I will say the idea of
493
00:25:19.359 --> 00:25:23.559
whether you can train on data and whether it's transformative
494
00:25:24.319 --> 00:25:27.200
is quite frankly still in the courts right now, and
495
00:25:27.240 --> 00:25:31.000
we don't have global consensus. So I believe it's Japan
496
00:25:31.160 --> 00:25:33.759
has said and clarified that you can train on data,
497
00:25:34.920 --> 00:25:40.519
that the training process is not directly infringement. Now, you know,
498
00:25:40.559 --> 00:25:42.599
one of the litmus tests is like whether you're competing
499
00:25:42.599 --> 00:25:46.519
with your regional thing. So it's it's a larger open question,
500
00:25:46.599 --> 00:25:48.759
but right now that's making its way through the courts.
501
00:25:49.119 --> 00:25:52.160
I think, you know, candidly, if you said here all
502
00:25:52.200 --> 00:25:54.319
the things I've trained on, then you might end up,
503
00:25:54.680 --> 00:25:56.759
you know, just opening yourself up to more people who
504
00:25:56.759 --> 00:25:59.440
can just say, oh, let me get onto that lawsuit.
505
00:26:00.960 --> 00:26:04.200
But I mean that that question is still being that's
506
00:26:04.200 --> 00:26:06.039
a legal question, which.
507
00:26:06.480 --> 00:26:11.279
Yeah, yeah, yeah, yeah, yeah, okay. So my my other
508
00:26:11.400 --> 00:26:14.440
question related to what we would we had just talked about,
509
00:26:14.759 --> 00:26:21.839
was so I I download Quinn and I don't know
510
00:26:21.880 --> 00:26:24.720
what it is that I'm actually downloading because I use
511
00:26:24.839 --> 00:26:29.640
I use Olama, and it downloads twenty gigabytes of something
512
00:26:29.720 --> 00:26:32.319
and then it runs it and I get to be
513
00:26:32.400 --> 00:26:38.400
productive and I'm happy. But in terms of you know,
514
00:26:38.480 --> 00:26:41.960
like like you're saying there's something it's not just the
515
00:26:42.079 --> 00:26:45.960
model giving a response, but then there's other code that
516
00:26:46.599 --> 00:26:50.880
is you're doing some sort of check. Is that happening
517
00:26:50.960 --> 00:26:54.720
with these models that I am using generic tools like
518
00:26:55.119 --> 00:27:01.119
Olama or Lama CPP or or LM studio. Is that
519
00:27:01.759 --> 00:27:07.079
actually running program code? Binary code? Or I guess it's
520
00:27:07.079 --> 00:27:09.319
not binary code. It have to be bytecode because I
521
00:27:09.359 --> 00:27:11.559
can do it on Mac and I can do it
522
00:27:11.599 --> 00:27:14.759
on Linux and it doesn't have to recompile anything after
523
00:27:14.799 --> 00:27:19.799
it's done downloading it. So what where where are those
524
00:27:20.279 --> 00:27:22.599
extra layers or how are they interpreted?
525
00:27:23.599 --> 00:27:26.880
The extra layers that are protecting the model from saying
526
00:27:26.960 --> 00:27:31.039
grong thing? Is that what you're asking? Yeah, yeah, those
527
00:27:31.039 --> 00:27:34.640
aren't there when you download an open source model, when
528
00:27:34.680 --> 00:27:36.839
you run in a little lama, those extra layers. The
529
00:27:36.880 --> 00:27:43.400
only thing that is protecting the model is at that
530
00:27:43.440 --> 00:27:46.640
point what the model was trained, the pre training they
531
00:27:46.680 --> 00:27:49.119
did that they baked into the model. Then they're not
532
00:27:49.160 --> 00:27:52.799
doing any additional checks. So with the hosted model, there
533
00:27:52.920 --> 00:27:57.559
is a there is additional layers because they ConTroll the
534
00:27:57.559 --> 00:27:59.880
infrastructure and they're watching what the model says and they're
535
00:28:00.119 --> 00:28:02.599
they're stopping it. But typically when you will use you know,
536
00:28:03.640 --> 00:28:06.519
LM Studio or Olama, then it's g you're just getting
537
00:28:06.559 --> 00:28:09.599
the bare uncensored model and there's no additional checks. The
538
00:28:09.599 --> 00:28:12.200
only thing that's preventing the model from you know, saying
539
00:28:12.240 --> 00:28:16.200
the wrong thing being not helpful or not harmless or
540
00:28:16.200 --> 00:28:21.160
I guess harmful and unhelpful is just the training and
541
00:28:21.680 --> 00:28:23.920
the models, you know, training that it was put through.
542
00:28:23.960 --> 00:28:26.799
There's no additional checks there. So and when you download
543
00:28:26.799 --> 00:28:30.519
the model, maybe it's worth saying you're just basically downloading
544
00:28:30.640 --> 00:28:34.920
a large list of numbers and the code inside it
545
00:28:35.000 --> 00:28:37.480
tells it. And you're getting a large list of numbers
546
00:28:37.960 --> 00:28:41.799
and a mathematical graph that says how to combine the
547
00:28:41.839 --> 00:28:45.599
numbers together. It says, take this parameter first, here's how
548
00:28:45.599 --> 00:28:48.839
you map the words to numbers. That's you get that mapping.
549
00:28:49.359 --> 00:28:51.599
And then once you've mapped them to numbers, it says,
550
00:28:51.680 --> 00:28:54.920
add it here, multiply times this other number here. Then
551
00:28:55.119 --> 00:28:57.640
you know normally you know, take the square root of
552
00:28:57.640 --> 00:29:00.000
this other number and then multiply it again. And it's
553
00:29:00.119 --> 00:29:04.720
just a list of calculations. It's a really like simple program.
554
00:29:05.240 --> 00:29:08.599
In fact, most of the knowledge it's worth stating is
555
00:29:08.720 --> 00:29:10.960
not in the code. And this gets back to your
556
00:29:11.039 --> 00:29:14.720
question about like open source, it's in the knowledge, is
557
00:29:14.759 --> 00:29:18.359
inside the data, it's inside the parameters. So as an example,
558
00:29:18.480 --> 00:29:22.559
GPT two, which you know is considered one point too
559
00:29:22.599 --> 00:29:25.200
dangerous release and it's amazing.
560
00:29:25.400 --> 00:29:29.680
Yeah, only that's because they want regulatory capture, not because
561
00:29:29.680 --> 00:29:31.079
they actually believe it's dangerous.
562
00:29:31.319 --> 00:29:35.240
Well at the time maybe they they they're concerned about disinformation,
563
00:29:35.279 --> 00:29:38.200
but suffice to say it was still a powerful model
564
00:29:38.240 --> 00:29:41.759
in its day. Is my point is only five hundred
565
00:29:41.799 --> 00:29:49.519
lines of code. If you take out the the the
566
00:29:49.720 --> 00:29:52.880
TensorFlow library, it's five hundred lines of code. It is.
567
00:29:52.920 --> 00:29:56.319
It is astonishingly small. And so one of the things
568
00:29:56.359 --> 00:29:58.559
and the reason why I re implemented the entire thing
569
00:29:58.599 --> 00:30:01.240
in JavaScript is I want to push back against this
570
00:30:01.400 --> 00:30:03.920
idea that well, this stuff is too hard for you
571
00:30:03.960 --> 00:30:06.799
to learn. If you're a web developer, like you can
572
00:30:06.839 --> 00:30:10.000
learn five hundred lines of code. And that's basically like
573
00:30:10.039 --> 00:30:11.839
I give you the grounding and I re implement the
574
00:30:11.920 --> 00:30:14.240
entire thing in javascripts. You can step through it. You
575
00:30:14.240 --> 00:30:17.400
don't even have to leave your web browser, right, you
576
00:30:17.440 --> 00:30:19.480
just use the web debugger and you can you can
577
00:30:19.519 --> 00:30:24.680
step through what's happening, and it's it's astonishingly small. All
578
00:30:24.720 --> 00:30:28.200
the knowledge, all the rules is captured in the weights
579
00:30:28.240 --> 00:30:30.400
and the parameters the model. So when you download the model,
580
00:30:30.680 --> 00:30:33.160
it's just a more and more numbers with a larger
581
00:30:33.200 --> 00:30:35.759
and larger computational graph. And that's how we get it smarter.
582
00:30:35.839 --> 00:30:37.720
That's gets back to the heart of like the core
583
00:30:37.759 --> 00:30:40.079
thing to understand is we took a word problem and
584
00:30:40.119 --> 00:30:41.599
we mapped it to a number problem. So if we
585
00:30:41.640 --> 00:30:43.599
get a bigger calculator, we get a better result.
586
00:30:44.799 --> 00:30:48.079
But I want to I want to restate this just
587
00:30:48.240 --> 00:30:51.279
in another way, really simply because I think a lot
588
00:30:51.359 --> 00:30:53.880
of people get you know, they get confused between like
589
00:30:53.960 --> 00:30:58.599
GPT four versus chat, GPT versus something on your computer
590
00:30:58.799 --> 00:31:03.359
versus whatever. And so yeah, essentially the model, like you said,
591
00:31:03.440 --> 00:31:06.759
you know, it's it's maybe a few steps in how
592
00:31:06.799 --> 00:31:10.759
it gives you answers, and the rest of it, like
593
00:31:10.799 --> 00:31:12.039
you said, is all the data.
594
00:31:12.079 --> 00:31:12.880
It's all the waiting.
595
00:31:13.920 --> 00:31:17.440
But sometimes when people are talking about AI models, they're
596
00:31:17.480 --> 00:31:20.440
talking about a program that accesses the model that I
597
00:31:20.599 --> 00:31:23.279
just explained or that you just explained, right with the
598
00:31:23.400 --> 00:31:27.200
numbers and kind of the fundamental pieces and so that's
599
00:31:27.240 --> 00:31:30.200
your chat GBT, whether it's running on your local machine
600
00:31:30.319 --> 00:31:34.480
or in the cloud. Yeah, you need to be able
601
00:31:34.519 -->
00:31:37.599
to differentiate between the two and recognize that. Yeah, sometimes
602
00:31:37.640 --> 00:31:41.480
you're just downloading that map of numbers and you know
603
00:31:41.519 --> 00:31:46.039
some really really simple stuff that makes sense of the numbers,
604
00:31:46.720 --> 00:31:49.240
and that's your model. And so when people are building
605
00:31:49.279 --> 00:31:51.519
against those models, a lot of times that's what they're doing.
606
00:31:51.799 --> 00:31:53.599
And so you can write your own code that then,
607
00:31:53.839 --> 00:31:56.599
you know, is the gatekeeper or you know, says this
608
00:31:56.680 --> 00:32:00.599
is helpful or this is harmful, or this is whatever. Right,
609
00:32:00.839 --> 00:32:04.359
this is an appropriate response and this isn't a lot
610
00:32:04.400 --> 00:32:06.920
of that's just the code that sits on top of
611
00:32:06.960 --> 00:32:09.680
what you're talking about. That five hundred lines of code
612
00:32:09.720 --> 00:32:12.960
plus the data that we're getting.
613
00:32:13.000 --> 00:32:14.440
That is just the model. And so.
614
00:32:16.240 --> 00:32:20.680
You know AJ's talking about the GWEN model. It looks
615
00:32:20.720 --> 00:32:25.400
like it runs on Olama, Right, So you know you've
616
00:32:25.440 --> 00:32:27.799
got all those magic numbers, You've got the stuff that
617
00:32:27.880 --> 00:32:29.960
runs on top of it. I think Olama gives you
618
00:32:30.000 --> 00:32:31.720
a little bit more on top of that, and then
619
00:32:31.759 --> 00:32:34.519
from there, right, the rest of it's kind of up
620
00:32:34.559 --> 00:32:36.200
to whoever wrote the code.
621
00:32:36.480 --> 00:32:38.920
Yeah, the central thing, I'm just trying to get like
622
00:32:38.920 --> 00:32:41.680
the black box part, the most mysterious part at the
623
00:32:41.680 --> 00:32:46.240
heart of it. Yeah, and obviously, like you know the
624
00:32:46.319 --> 00:32:51.039
calculations that say Lama and Chat, GPT and Gemini, they're
625
00:32:51.119 --> 00:32:55.359
larger models. GPT two came out in twenty nineteen, but
626
00:32:55.400 --> 00:32:58.119
the core thing is like a mo Like if you
627
00:32:58.119 --> 00:33:01.559
had shown me GPT two, I wouldn't have known. Like
628
00:33:01.640 --> 00:33:03.440
when it first came out, it's like, wow, that's a
629
00:33:03.480 --> 00:33:05.960
pretty amazing program. Must be really complex. And it's not
630
00:33:06.000 --> 00:33:10.160
the program that's complex. It's they just bet on taking
631
00:33:10.200 --> 00:33:13.000
a somewhat simple architecture and just giving it lots and
632
00:33:13.039 --> 00:33:15.240
lots of data and spending more than anybody else had
633
00:33:15.240 --> 00:33:17.920
at the time, and just trust that the black box
634
00:33:17.960 --> 00:33:20.640
would be smart enough to learn everything from it. And
635
00:33:20.960 --> 00:33:23.200
at the heart of it, that's what's happening. You're just
636
00:33:23.559 --> 00:33:26.519
a large it's a giant calculator. In fact, it's so
637
00:33:26.640 --> 00:33:31.079
simple in a sense, like in a spreadsheet, which was
638
00:33:31.119 --> 00:33:34.759
my first implementation, you cannot do loops very easily. You
639
00:33:34.839 --> 00:33:38.000
don't have looping constructs. It just does a calculation through
640
00:33:38.000 --> 00:33:40.960
the entire way, and it just does the computation of
641
00:33:41.000 --> 00:33:44.960
the all the different cells they're effectively in a sense,
642
00:33:45.000 --> 00:33:47.519
no loops inside of it. Like the reason I can
643
00:33:47.640 --> 00:33:50.000
implement it in a spreadsheet is that every single time
644
00:33:50.279 --> 00:33:52.440
it predicts a token, it does the exact same number
645
00:33:52.480 --> 00:33:55.440
of computations every single time, and it goes through, you know,
646
00:33:55.599 --> 00:33:59.039
twelve layers and twelve attention steps and twelve layers. Like
647
00:33:59.279 --> 00:34:02.440
it's very very much like I got a number coming in,
648
00:34:02.880 --> 00:34:04.880
we a word coming in, we map that word to
649
00:34:04.920 --> 00:34:07.200
a number, and then I just do all this number
650
00:34:07.240 --> 00:34:09.760
crunching with a very predictable pattern, and then I get
651
00:34:09.760 --> 00:34:11.360
a number out and I interpret it, and then I
652
00:34:11.440 --> 00:34:15.360
just repeat that process over and over again. And so
653
00:34:15.920 --> 00:34:17.440
you know, the thing that I try to tell people
654
00:34:17.599 --> 00:34:21.480
is just like when you look online and people like
655
00:34:21.519 --> 00:34:25.079
I want to get into AI and stuff, they're they're
656
00:34:25.159 --> 00:34:29.239
often presented with, okay, go learn you know all this
657
00:34:29.320 --> 00:34:31.480
linear algebra. You need to make sure you're solid on
658
00:34:31.519 --> 00:34:34.440
your calculus. You need to make like and it's there's
659
00:34:34.559 --> 00:34:37.800
like six to eighteen months of like prep before you
660
00:34:37.840 --> 00:34:40.599
get to understanding how a large language model works. And
661
00:34:40.920 --> 00:34:43.679
that's valid if you're going to be a machine learning researcher,
662
00:34:44.239 --> 00:34:47.320
and machine learning is a huge giant field beyond just
663
00:34:47.519 --> 00:34:50.960
chat GPT right, there's an omaly detection, there's clustering, there's
664
00:34:51.000 --> 00:34:55.039
a lot of algorithms in there. But my goal is
665
00:34:55.079 --> 00:34:59.119
to just help people understand how these amazing, arguably Nobel
666
00:34:59.159 --> 00:35:02.199
Prize winning programs work in as short a time as possible,
667
00:35:02.320 --> 00:35:04.480
and to the extent, like I don't even begin where
668
00:35:04.519 --> 00:35:06.920
normal machine learning class begins. A normal machine learning class
669
00:35:06.960 --> 00:35:10.000
starts with like regression and it slowly works
me up
670
00:35:10.000 --> 00:35:12.880
and maybe you'll get to the LMS. And I'm like,
671
00:35:12.880 --> 00:35:14.920
this is a five hundred line program. Just start with
672
00:35:15.280 --> 00:35:17.920
here's how it starts. And anytime I run to something
673
00:35:17.920 --> 00:35:20.119
you don't understand, I'll give you the In my class,
674
00:35:20.159 --> 00:35:22.559
I give you the background to understand it, and then
675
00:35:22.559 --> 00:35:24.079
we move on to the next piece. And so it's
676
00:35:24.239 --> 00:35:26.199
really designed to be as efficient as possible. And I
677
00:35:26.800 --> 00:35:29.639
think when you tell people it's five hundred lines, they're like, oh, yeah, okay,
678
00:35:29.639 --> 00:35:32.960
I can understand how that works. And this gets to
679
00:35:33.079 --> 00:35:37.079
knowing your tools. I'll make an analogy if you don't
680
00:35:37.159 --> 00:35:39.960
necessarily need to know how AI model works to use it,
681
00:35:40.559 --> 00:35:42.800
but you don't necessarily need to have a good model
682
00:35:42.880 --> 00:35:46.480
for how is the difference between the CPU or disc
683
00:35:46.559 --> 00:35:53.559
memory versus bandwidth versus system memory. But if you're debugging,
684
00:35:53.679 --> 00:35:57.000
you know, a machine program, it helps to have that
685
00:35:57.039 --> 00:35:59.840
mental model. You'll run to an issue or maybe a
686
00:35:59.880 --> 00:36:02.920
more our tangible example to this audience is like knowing
687
00:36:03.039 --> 00:36:05.400
how react works on the inside. At some point, if
688
00:36:05.400 --> 00:36:08.920
you don't understand hydration, you're going to run into a wall, right,
689
00:36:09.079 --> 00:36:10.920
And the same thing is true, like you get these
690
00:36:11.119 --> 00:36:14.599
parameters from OLAMA, what are they doing? You know, you
691
00:36:14.639 --> 00:36:16.119
need to have a mental model for how it works.
692
00:36:16.159 --> 00:36:17.960
And I don't think that mental model is as hard
693
00:36:18.000 --> 00:36:19.280
as a lot of people make it out to be.
694
00:36:20.719 --> 00:36:23.960
Yeah, when I talk to people about doing AI, and
695
00:36:24.039 --> 00:36:25.760
I talked to a whole bunch of people like that
696
00:36:25.840 --> 00:36:27.519
are business people, and talk to a whole bunch of
697
00:36:27.519 --> 00:36:31.199
people that are programmers, and I have some of the
698
00:36:31.239 --> 00:36:35.559
same conversation basically down to, well, are you going to
699
00:36:35.559 --> 00:36:37.719
build your own model, right? Are you going to take
700
00:36:37.760 --> 00:36:40.280
your own data and cram it in and expect it
701
00:36:40.280 --> 00:36:42.480
to give you answers on the other end, or are
702
00:36:42.480 --> 00:36:44.800
you going to use something that already exists like the
703
00:36:44.880 --> 00:36:48.320
chat GPTs or some of the you know, the GPT
704
00:36:48.440 --> 00:36:52.079
force or the OLAMAS or whatever, right, and then build
705
00:36:52.119 --> 00:36:54.880
on top of it. Because once you're building on top
706
00:36:54.960 --> 00:36:56.800
of it and you're not worried about, Okay, how do
707
00:36:56.840 --> 00:37:00.760
I put this together, then it's essentially okay, Like you're saying,
708
00:37:00.840 --> 00:37:04.440
I understand how the machine works, and then I understand
709
00:37:04.440 --> 00:37:06.159
how to talk to it. Right, so I understand what
710
00:37:06.199 --> 00:37:09.760
the APIs are and the rest of it is, then okay,
711
00:37:09.920 --> 00:37:10.920
what do I want from this?
712
00:37:11.000 --> 00:37:12.519
And how do I validate that I got it?
713
00:37:13.599 --> 00:37:17.599
Yeah? Actually, that's a really important point. The number one
714
00:37:17.679 --> 00:37:22.440
skill may not be understanding every single detail of the calculation.
715
00:37:22.880 --> 00:37:25.320
The number one skill when dealing with an AI model
716
00:37:25.440 --> 00:37:27.079
is that last thing you talked about, how do I
717
00:37:27.119 --> 00:37:30.719
evaluate it? So the name that you hear in the
718
00:37:30.760 --> 00:37:35.239
AI community is evals, But as a you know web developer,
719
00:37:35.280 --> 00:37:37.960
you can think of these as tests. And the key
720
00:37:37.960 --> 00:37:42.119
difference between you know AI evals and tests is that
721
00:37:42.159 --> 00:37:45.039
you don't expect one hundred percent pass right. These are statistical,
722
00:37:45.119 --> 00:37:47.800
probabilistic machines. But the number one, like when you read
723
00:37:47.840 --> 00:37:50.480
about benchmarks, you know, AJA you talked about benchmarks, You
724
00:37:50.599 --> 00:37:53.320
basically need to build the benchmarks for your particular problem.
725
00:37:53.400 --> 00:37:56.599
The benchmark may say some model is better than another,
726
00:37:56.920 --> 00:37:59.000
but when you actually use it for your problem, you
727
00:37:59.039 --> 00:38:02.519
suddenly discover it's not good. And so the first thing
728
00:38:02.559 --> 00:38:05.239
you should do is come up with your own benchmark,
729
00:38:05.320 --> 00:38:07.840
your own evals for the problem, and then try a
730
00:38:07.840 --> 00:38:10.360
bunch of models and see which one works the best.
731
00:38:10.400 --> 00:38:13.639
And then you can start iterating whether that iterating is
732
00:38:13.760 --> 00:38:17.079
changing the prompt, whether it's changing the model or saying
733
00:38:17.079 --> 00:38:18.840
I'm going to go off and find tune my own model.
734
00:38:18.880 --> 00:38:21.079
But you won't be able to make a judgment until
735
00:38:21.079 --> 00:38:24.559
you're able to look across the distribution of your task,
736
00:38:24.800 --> 00:38:27.239
all the different ways your task happens, and whether it's
737
00:38:27.320 --> 00:38:31.480
successful or not, because these are you're dealing with highly
738
00:38:31.559 --> 00:38:36.679
variable machines.
One of you know, the folks who was
739
00:38:36.679 --> 00:38:38.519
in the audience for one of my past talks, hit
740
00:38:38.559 --> 00:38:41.760
a really good analogy. He's like, imagine a database that
741
00:38:41.960 --> 00:38:45.159
was wrong five percent of the time. Like, as developers,
742
00:38:45.199 --> 00:38:49.480
we are not used to having levels of uncertainty like
743
00:38:49.519 --> 00:38:52.639
this within our systems, unless maybe you're using distributed systems
744
00:38:52.760 --> 00:38:54.920
where there's all sorts of race conditions and stuff like that.
745
00:38:55.000 --> 00:38:58.719
But we're used to sanitizing the user input and then
746
00:38:58.760 --> 00:39:01.159
once we get the user input. Everything is predictable after that.
747
00:39:01.519 --> 00:39:04.039
But here it's like suddenly we've got a database that
748
00:39:04.159 --> 00:39:06.000
sometimes it's wrong, and so that's where you need to
749
00:39:06.000 --> 00:39:08.159
put all sorts of checks and guardrails. And you're dealing
750
00:39:08.239 --> 00:39:12.719
with this really smart but sometimes fallible thing like a human,
751
00:39:12.760 --> 00:39:16.519
I hate to say, anthropomorphizing it, and see how you
752
00:39:16.519 --> 00:39:18.760
build a system around that is going to be different
753
00:39:18.880 --> 00:39:20.840
than how you build a regular system. But it all
754
00:39:20.880 --> 00:39:23.000
starts with that key idea that you just talked about,
755
00:39:23.000 --> 00:39:27.599
which is about being able to evaluate mathematically how well
756
00:39:27.639 --> 00:39:30.639
your your model or your system is doing, and the
757
00:39:30.719 --> 00:39:32.599
question about whether you should build your own model or not.
758
00:39:33.159 --> 00:39:37.880
The usual hierarchy of needs is first start with an
759
00:39:37.880 --> 00:39:41.000
off the shelf model. It could be open source, it
760
00:39:41.039 --> 00:39:44.239
could be one of the providers. It's actually probably easiest
761
00:39:44.280 --> 00:39:46.920
to start with a hosted model and just see if
762
00:39:46.960 --> 00:39:48.599
you can get it to work because there'll be state
763
00:39:48.639 --> 00:39:50.159
of the art and you don't have to worry about
764
00:39:50.199 --> 00:39:52.880
all the stuff around hosting and inference and seeing if
765
00:39:52.920 --> 00:39:55.519
it works. And then next thing to try is try
766
00:39:55.559 --> 00:39:58.840
tuning it. Sorry, try try tuning your prompts. So try
767
00:39:58.880 --> 00:40:01.320
prompt engineering your way. Give it some examples try some
768
00:40:01.440 --> 00:40:05.679
variety of prompt engineering, and then maybe consider fine tuning it.
769
00:40:05.719 --> 00:40:07.719
And again you can fine tune you know, most of
770
00:40:07.719 --> 00:40:09.559
the hosted models, you don't have to go to an
771
00:40:09.599 --> 00:40:11.920
open source model, but you could do that as well.
772
00:40:12.199 --> 00:40:13.760
And then the idea of building your own model is
773
00:40:13.800 --> 00:40:18.239
extremely hard. You know, the amount of dollars that go
774
00:40:18.320 --> 00:40:22.440
into building your own models from scratch is now, you know,
775
00:40:23.000 --> 00:40:26.440
over one hundred million. So the estimates for say Lama
776
00:40:26.519 --> 00:40:29.400
are were, I think over one hundred million to build
777
00:40:29.400 --> 00:40:32.360
that model, and so it's a lot of work and
778
00:40:33.400 --> 00:40:35.239
that's lovely best of the frontier labs.
779
00:40:35.280 --> 00:40:38.960
Yeah, is that GPU cost or where is that number
780
00:40:38.960 --> 00:40:39.559
coming from?
781
00:40:40.840 --> 00:40:44.360
That's a great question because these are all estimates. You know,
782
00:40:44.400 --> 00:40:46.320
we don't know for sure, but obviously some of it
783
00:40:46.400 --> 00:40:51.599
is the GPU cost, some of it is the infrastructure cost,
784
00:40:51.639 --> 00:40:53.960
some of it's the talent. The other key thing to
785
00:40:54.039 --> 00:40:57.760
keep in mind is when you're training a model, you
786
00:40:57.800 --> 00:41:00.119
don't always know how it's going to turn out. What
787
00:41:00.119 --> 00:41:02.039
they actually do is they do a large series of
788
00:41:02.079 --> 00:41:06.719
smaller runs to establish some type of pattern or scaling
789
00:41:06.800 --> 00:41:09.440
law to figure out how they're going to design the model,
790
00:41:09.519 --> 00:41:12.960
which architecture seem to work better, which parameters matter more.
791
00:41:13.000 --> 00:41:15.239
So there's something called a learning rate, for example, that
792
00:41:15.239 --> 00:41:17.119
they have to adjust, and they have a schedule for it,
793
00:41:17.119 --> 00:41:20.079
and they're trying to figure out against evals against the
794
00:41:20.119 --> 00:41:22.920
benchmarks we talked about, like which one seems to make
795
00:41:22.960 --> 00:41:26.400
the model smarter. And so there's a lot of trials
796
00:41:26.480 --> 00:41:29.519
and attempts. So it's not just necessarily one whole shot
797
00:41:29.679 --> 00:41:33.199
of training. It's a lot of experiments that they have
798
00:41:33.280 --> 00:41:35.239
to do. A lot of how the model is going
799
00:41:35.239 --> 00:41:40.000
to behave is surprisingly empirical, and so they're doing experiments
800
00:41:40.039 --> 00:41:41.960
and they're trying that again, so there's a variety of things,
801
00:41:42.000 --> 00:41:45.920
and empower is non trivial. Another thing that's important to
802
00:41:45.960 --> 00:41:50.719
understand is the level of scale of data that these
803
00:41:50.719 --> 00:41:53.960
frontier labs are dealing with. And there's a really good
804
00:41:54.559 --> 00:41:58.519
analogy from the anthropic guys actually, and one of the
805
00:41:58.519 --> 00:41:59.800
things I have to do is you have to randomize
806
00:41:59.800 --> 00:42:02.400
the day data so it
doesn't learn arbitrary patterns and
807
00:42:02.440 --> 00:42:04.400
the order of the data you gave it. And so
808
00:42:05.280 --> 00:42:08.599
one of their research engineers gave this great example is like, okay,
809
00:42:08.840 --> 00:42:11.480
randomizing sounds like it should be easy, Like take a
810
00:42:11.519 --> 00:42:13.159
deck of cards. If I tell you to shuffle it,
811
00:42:13.159 --> 00:42:15.960
it's fairly easy. But imagine I gave you like seven
812
00:42:16.000 --> 00:42:19.960
warehouses full of decks of cards and you need to
813
00:42:20.000 --> 00:42:22.719
shuffle them by hand. It's not quite clear how what
814
00:42:22.920 --> 00:42:25.199
policy or process you're going to use to make sure
815
00:42:25.199 --> 00:42:28.039
you hit all of them and you've evenly shuffled them.
816
00:42:28.280 --> 00:42:30.599
And the size of the data that these guys are
817
00:42:30.679 --> 00:42:32.599
using with is it's almost like that to the CPU,
818
00:42:32.679 --> 00:42:35.480
it's like seven warehouses of data to it like for
819
00:42:35.599 --> 00:42:38.360
you compared to manually you know, shuffle your your deck.
820
00:42:38.800 --> 00:42:40.719
And so when you're dealing with data at this large
821
00:42:40.719 --> 00:42:45.360
infrastructure scale scale itself makes every little thing harder, and
822
00:42:45.400 --> 00:42:50.079
so that also adds some difficulty to this. So should
823
00:42:50.119 --> 00:42:53.320
I you know, walk through just like a little more
824
00:42:53.360 --> 00:42:56.039
detail of what's happening in that mathematical calculation or happy
825
00:42:56.039 --> 00:42:57.199
to answer additional questions?
826
00:42:57.320 --> 00:42:58.800
Yeah, that's what I was going to ask.
827
00:42:58.880 --> 00:43:02.199
Next is Yeah, because you've mentioned you've got different layers
828
00:43:02.280 --> 00:43:05.440
or different steps in the process you explain in the video.
829
00:43:05.760 --> 00:43:07.679
The video is a little longer, I guess than we
830
00:43:07.800 --> 00:43:10.280
have to go over at this point, but yeah, if
831
00:43:10.320 --> 00:43:12.599
you can give people kind of an overview of how
832
00:43:12.599 --> 00:43:14.880
the LM system actually works.
833
00:43:15.239 --> 00:43:18.800
Yeah, So while you're doing that, if you distinguish between
834
00:43:19.119 --> 00:43:23.719
the different types of training, like the RAG versus the
835
00:43:23.760 --> 00:43:27.199
fine tuning versus the.
836
00:43:26.239 --> 00:43:32.039
Base Yeah, okay, I don't think of RAG as training,
837
00:43:33.320 --> 00:43:35.840
but maybe we should step back and explain to the
838
00:43:35.840 --> 00:43:39.519
audience who isn't familiar with RAG what it is. So
839
00:43:39.960 --> 00:43:43.239
you can kind of think of RAG as like a
840
00:43:43.360 --> 00:43:46.960
sort of prompt engineering technique. So you want the model
841
00:43:47.000 --> 00:43:49.400
to answer questions about something that wasn't trained on. So
842
00:43:49.480 --> 00:43:53.320
imagine I'm you know, I'm a smart home electronics company,
843
00:43:53.599 --> 00:43:57.519
and all of the documentation about my product was behind
844
00:43:57.800 --> 00:44:00.119
you know, some firewall or behind a log in, and
845
00:44:00.159 --> 00:44:02.679
so I know that, let's call it chatchy. He was
846
00:44:02.719 --> 00:44:04.320
never trained on it. But I want to build a
847
00:44:04.400 --> 00:44:07.000
chatpot where customers come and say, hey, I can't configure
848
00:44:07.159 --> 00:44:08.920
this setting on it. How am I going to get
849
00:44:09.039 --> 00:44:12.039
a chatbot to do that without having to retrain it
850
00:44:12.079 --> 00:44:15.559
specifically on my data. So what you can do is
851
00:44:15.639 --> 00:44:18.679
when a request comes in and somebody's like, well, how
852
00:44:18.679 --> 00:44:21.199
do I change the color on my smart light bulb,
853
00:44:21.880 --> 00:44:25.360
It'll go and it will search through my data. I
854
00:44:25.400 --> 00:44:27.840
can take that request from my user on my chatbot,
855
00:44:28.079 --> 00:44:30.159
and they say, I take I see the words light bulb,
856
00:44:30.199 --> 00:44:33.199
I see change color, and I'll search all my documentation
857
00:44:33.400 --> 00:44:35.000
and I'm not going to search it just a plain
858
00:44:35.079 --> 00:44:37.440
tech search. I'll use it called a semantic search. So
859
00:44:37.519 --> 00:44:40.559
it'll find things that are similar to the word light,
860
00:44:40.760 --> 00:44:44.079
like the word bright, even though it's not anywhere close
861
00:44:44.079 --> 00:44:46.039
to the same character. So it'll find all the similar
862
00:44:46.039 --> 00:44:49.840
passages and it will pull those out, and then it
863
00:44:49.840 --> 00:44:54.400
will give the model, here are relevant passages. Here's the
864
00:44:54.480 --> 00:44:57.159
user's question, how do I change the color on my
865
00:44:57.199 --> 00:44:59.840
smart light bulb? And then it will give it paragraphs
866
00:45:00.119 --> 00:45:03.559
chunks of text from my documentation, and it will put
867
00:45:03.559 --> 00:45:05.159
those at the beginning of the prompt. So you've got
868
00:45:05.159 --> 00:45:07.119
a prompt that structure at the start with the user's question.
869
00:45:07.639 --> 00:45:09.800
Then it's got some chunks of data that came right
870
00:45:09.800 --> 00:45:12.800
for my documentation. And then we tell the model, you know,
871
00:45:13.000 --> 00:45:15.280
come up with an answer to the user's question using
872
00:45:15.320 --> 00:45:17.719
these chunks of data I gave you, and it will
873
00:45:17.960 --> 00:45:20.239
be able to think over those passages and find the
874
00:45:20.239 --> 00:45:23.480
ones that are relevant and then give the answer out.
875
00:45:23.599 --> 00:45:27.760
And that's called retrieval augmented generation. So retrieval because you're
876
00:45:27.800 --> 00:45:30.400
taking the user's query, you're pulling a data that wasn't
877
00:45:30.480 --> 00:45:32.920
the model didn't have during training, and you're passing it
878
00:45:32.960 --> 00:45:35.599
into the prompt and then asking the model to answer it.
879
00:45:35.880 --> 00:45:39.280
And so it's a very low friction way to take
880
00:45:39.320 --> 00:45:42.159
a model off the shelf and make it understand all
881
00:45:42.159 --> 00:45:44.000
your stuff even though it wasn't in the training data.
882
00:45:44.719 --> 00:45:48.239
So that's that's uh, that's that's what RAG is.
883
00:45:48.679 --> 00:45:51.320
The version is is you're building context out of a
884
00:45:51.400 --> 00:45:53.880
database that you already.
885
00:45:53.599 --> 00:45:58.000
Have, great, great summary, thank you. Uh so, yeah, you're
886
00:45:58.000 --> 00:46:00.360
giving it the context it didn't have during training to
887
00:46:00.440 --> 00:46:08.320
answer the question on training. So there's a variety of
888
00:46:08.400 --> 00:46:11.199
steps in the model where it's trained. Mainly, well, let
889
00:46:11.199 --> 00:46:12.800
me think of the best way to explain this. So
890
00:46:14.559 --> 00:46:17.920
I'll discuss training when I get to the call it
891
00:46:17.960 --> 00:46:20.199
the fourth step of the model, and I'll talk about
892
00:46:20.199 --> 00:46:22.920
how they gets trained in a second. But let me
893
00:46:22.960 --> 00:46:24.960
walk through the five steps of what happens when you
894
00:46:24.960 --> 00:46:27.400
input text into the model. So the first thing you
895
00:46:27.440 --> 00:46:29.480
do when you input a passage so that one I
896
00:46:29.599 --> 00:46:31.480
like to uses Mike as quick he moves. And then
897
00:46:31.599 --> 00:46:33.679
the completion we leave it to the model filling the
898
00:46:33.960 --> 00:46:36.599
blank quickly. The first thing it's gonna do is going
899
00:46:36.679 --> 00:46:40.360
to break the text into subword units. So you might
900
00:46:40.400 --> 00:46:42.639
think it would break it into characters, and you might
901
00:46:42.679 --> 00:46:44.360
think it might break it into words. So break into
902
00:46:44.440 --> 00:46:47.280
characters would be like ASKI, and breaking into words would
903
00:46:47.280 --> 00:46:49.800
be just giving every word it's entry in a dictionary.
904
00:46:50.400 --> 00:46:53.440
It turns out you can't handle if you break it
905
00:46:53.480 --> 00:46:57.480
into words. You can't handle unknown words very well, and
906
00:46:57.519 --> 00:47:00.000
you can't handle spellings you weren't planning on. So especially
907
00:47:00.119 --> 00:47:02.920
if you're going across multiple languages, and if you break
908
00:47:02.920 --> 00:47:05.679
it into characters, it turns out it's really hard and
909
00:47:05.719 --> 00:47:07.880
a lot of compute for the model to learn purely
910
00:47:07.880 --> 00:47:11.199
from characters, although some research has been able to do it.
911
00:47:11.559 --> 00:47:13.519
So they do is they can do a Goldilocks and
912
00:47:13.519 --> 00:47:16.159
they say, okay, let's break it into these little pieces
913
00:47:16.159 --> 00:47:18.760
of words, and if you think about it as a human,
914
00:47:18.800 --> 00:47:21.719
you actually do this. So one of the examples, I
915
00:47:22.039 --> 00:47:24.559
use the word flavor eye. It's actually not a word
916
00:47:24.559 --> 00:47:26.280
in the dictionary, but you know what it means because
917
00:47:26.280 --> 00:47:28.039
you know what flavor means, and you know what ize
918
00:47:28.119 --> 00:47:30.599
is a suffix means, and so the model kind of
919
00:47:30.679 --> 00:47:32.960
does that. Now. I want to be clear, the tokens
920
00:47:32.960 --> 00:47:35.480
that comes up with, as they're called these subword pieces.
921
00:47:35.639 --> 00:47:39.639
Word pieces don't map to any human sense of the meaning.
922
00:47:39.679 --> 00:47:41.719
There are some that, like ice turns out to be
923
00:47:42.000 --> 00:47:44.960
a token, but it's by coincidence or correlation, not like
924
00:47:45.000 --> 00:47:47.920
it's trying to understand human English at this stage. So
925
00:47:47.960 --> 00:47:49.360
it breaks it into these Yeah.
926
00:47:49.159 --> 00:47:51.119
Gosh, can I just say that in a different way
927
00:47:51.119 --> 00:47:53.800
too effectively? What it does is it breaks it up
928
00:47:53.840 --> 00:47:58.639
into pieces that have meaning, right, because when we're looking
929
00:47:58.719 --> 00:48:00.679
for the output, we're looking for out it has meaning,
930
00:48:00.719 --> 00:48:04.920
and we group words or ideas together that give it meaning.
931
00:48:05.519 --> 00:48:08.360
And so it's doing the same thing. It's breaking it up, right,
932
00:48:08.480 --> 00:48:11.639
Like you said, flavor has a meaning, eyes has a meaning.
933
00:48:12.000 --> 00:48:14.480
You know, the other words in there have meaning. And
934
00:48:14.519 --> 00:48:17.320
so that that's the approach that it kind of takes
935
00:48:17.320 --> 00:48:20.599
when it's breaking it into tokens of kind.
936
00:48:20.400 --> 00:48:24.039
Of sort of kind of. It's a decent mental model.
937
00:48:24.639 --> 00:48:26.519
But the reason I stress that it's not trying to
938
00:48:26.519 --> 00:48:30.400
match human meaning is because it's actually not trying at
939
00:48:30.400 --> 00:48:32.960
this stage of the model. It's not trying to assign meaning.
940
00:48:33.480 --> 00:48:37.000
In fact, what it's really trying to do is take
941
00:48:37.119 --> 00:48:39.079
all the text on the Internet that it's trying to
942
00:48:39.119 --> 00:48:43.199
train on and compress it to the most efficient representation
943
00:48:43.440 --> 00:48:45.719
so that the training can be as efficient as possible.
944
00:48:46.440 --> 00:48:49.760
And that's why the tokens don't always map to what
945
00:48:49.800 --> 00:48:54.559
you'd expect and why So this is this.
946
00:48:54.599 --> 00:48:56.920
Is different than what a full text search database would
947
00:48:56.920 --> 00:48:59.000
do because a full text search database, like the example
948
00:48:59.039 --> 00:49:02.119
you gave flavor, Yeah, full text search database is going
949
00:49:02.159 --> 00:49:04.239
to break it up that same way. But this is
950
00:49:04.280 --> 00:49:06.280
different than the way a full text search database would
951
00:49:06.280 --> 00:49:06.760
break it up.
952
00:49:07.119 --> 00:49:09.599
Yes, it is different, and it's very dependent on the
953
00:49:09.679 --> 00:49:13.800
data it was trained on. And so a good example
954
00:49:13.880 --> 00:49:18.400
is I use the word re injury. Right as a human,
955
00:49:18.400 --> 00:49:21.199
you would think it was it was re an injury,
956
00:49:21.639 --> 00:49:24.239
but if you actually put it through the GPT two tokenizer,
957
00:49:24.719 --> 00:49:28.760
it puts it as rain injury. And the reason it
958
00:49:28.800 --> 00:49:31.159
decided to do that is simply because of the greater
959
00:49:31.960 --> 00:49:35.000
you know, occurrence of the word jury on itself by
960
00:49:35.039 --> 00:49:38.559
itself than injury, and so that decided that was the
961
00:49:38.559 --> 00:49:40.679
more efficient representation. And I want to be clear, this
962
00:49:40.719 --> 00:49:44.719
isn't about representing your prompt efficiently. This is about representing
963
00:49:45.039 --> 00:49:48.519
all the training it's going to do on the text efficiently,
964
00:49:48.559 --> 00:49:50.440
the stuff you don't see, the stuff that you know
965
00:49:50.480 --> 00:49:53.400
you're talking about nobody releases. That's what it's really based on.
966
00:49:53.599 --> 00:49:57.519
And it's really a compression of all the text. So
967
00:49:57.559 --> 00:49:59.840
it just got a really efficient So think about it this.
968
00:50:00.159 --> 00:50:02.480
If it's going to compress all the text, then you know,
969
00:50:02.559 --> 00:50:04.760
if it gets down to say ten thousand or fifty
970
00:50:04.760 --> 00:50:08.280
thousand tokens, then it only has to learn ten thousand
971
00:50:08.360 --> 00:50:11.679
or fifty thousand concepts in a sense, Although that's a
972
00:50:11.760 --> 00:50:14.440
gross oversimplification, but that's what it's trying to do, is
973
00:50:14.440 --> 00:50:15.960
trying to reduce the number of things it needs to
974
00:50:16.039 --> 00:50:18.880
learn on essentially the number of combinations and variations.
975
00:50:19.280 --> 00:50:22.159
Hey, a couple of quick questions with that flavorized example.
976
00:50:22.239 --> 00:50:24.599
Here's hoping they don't pick up flavor flav the rapper's
977
00:50:24.679 --> 00:50:26.480
lyrics right and throw that in there. That could get
978
00:50:26.519 --> 00:50:32.159
really confusing. But then when you're talking about like the
979
00:50:32.199 --> 00:50:35.079
re injury for example, Yeah, and how I as soon
980
00:50:35.119 --> 00:50:37.599
as you saw that said that, I was thinking, Okay,
981
00:50:37.639 --> 00:50:39.719
I can see where that's going. How you get rained
982
00:50:40.199 --> 00:50:42.639
just throwing stuff like and this might be getting into
983
00:50:42.679 --> 00:50:45.199
the weeds too much. But just throwing stuff like hyphens
984
00:50:45.239 --> 00:50:48.400
into words make a difference. So if you were to
985
00:50:48.400 --> 00:50:51.840
do redash injury, would it see that and maybe just
986
00:50:52.039 --> 00:50:55.440
categorize the re as separate from injury? Does that help
987
00:50:55.559 --> 00:50:57.119
or is that a non issue? Does it sort of
988
00:50:57.119 --> 00:50:59.639
filter that stuff out and just focus on the letters?
989
00:51:00.000 --> 00:51:03.639
And that's a great question. It's actually implementation dependent on
990
00:51:03.880 --> 00:51:09.800
the tokenizer. In practice, you usually separate. You create boundaries,
991
00:51:09.800 --> 00:51:13.199
hard boundaries between tokens or words, so one of them
992
00:51:13.320 --> 00:51:15.760
is the space character. In most of these, the hyphen
993
00:51:15.840 --> 00:51:18.159
is considered also a boundary, and so it would see
994
00:51:18.159 --> 00:51:21.880
it separately. The important thing to understand, though about the
995
00:51:21.920 --> 00:51:25.960
tokenizer is that the model doesn't see words the same
996
00:51:25.960 --> 00:51:28.960
way you do. So a great example of this is
997
00:51:29.880 --> 00:51:33.679
how many letters are in the word strawberry. It does
998
00:51:33.719 --> 00:51:36.960
not see the word as S T R A, W
999
00:51:37.559 --> 00:51:40.440
B E R R y. In fact, when you read it,
1000
00:51:40.480 --> 00:51:43.440
you really don't either. When you read words, you typically
1001
00:51:43.440 --> 00:51:45.480
aren't paying attention to every single character. When you have
1002
00:51:45.480 --> 00:51:46.840
to count the words in strawberry, you kind of have
1003
00:51:46.880 --> 00:51:49.119
to change your mental state and think, oh wait, let
1004
00:51:49.159 --> 00:51:51.840
me think what are the characters? And you walk through it.
1005
00:51:51.840 --> 00:51:54.960
It just sees it as it might see it as
1006
00:51:54.960 --> 00:51:56.920
the token strawberry. It might see it as the token
1007
00:51:56.960 --> 00:51:59.159
straw and berry. But the key thing is it has
1008
00:51:59.199 --> 00:52:01.199
no idea. It doesn't the ability to see the letters.
1009
00:52:01.239 --> 00:52:05.039
In fact, if you capitalize, the tokenization is case sensitive,
1010
00:52:05.320 --> 00:52:07.199
so if you change the capitalization, it looks like a
1011
00:52:07.199 --> 00:52:10.840
different it. So to it, the word strawberry with like
1012
00:52:10.880 --> 00:52:12.719
a space in front of it is different than the
1013
00:52:12.760 --> 00:52:15.480
strawberry without a space. Strawberry with the capital in front
1014
00:52:15.480 --> 00:52:18.320
of it
is different than strawberry with a capital. You know,
1015
00:52:18.400 --> 00:52:20.559
if you put quotes around it, it's a different word.
1016
00:52:21.039 --> 00:52:24.039
So it doesn't see text the same way you do.
1017
00:52:24.119 --> 00:52:27.239
Another great example of this is numbers. So they've fixed
1018
00:52:27.239 --> 00:52:30.400
this in most modern tokenizers, but the early ones would
1019
00:52:30.400 --> 00:52:33.000
just take examples of numbers and those would be a
1020
00:52:33.000 --> 00:52:35.360
whole take token. So two fifty six right power of
1021
00:52:35.400 --> 00:52:38.280
two fairly common gets a token. But it sees that
1022
00:52:38.320 --> 00:52:40.079
as a single token. It sees that as a single thing.
1023
00:52:40.079 --> 00:52:41.480
It doesn't even see it as the numbers two, five
1024
00:52:41.519 --> 00:52:43.639
and six. It doesn't break it apart. And so that's
1025
00:52:43.679 --> 00:52:46.039
why it was really hard for these guys or these guys,
1026
00:52:46.079 --> 00:52:48.039
these things to it was part of the contributing reason
1027
00:52:48.039 --> 00:52:49.400
why it was hard to do math, it's not the
1028
00:52:49.440 --> 00:52:52.559
sole reason. So the key lesson, you know, on the
1029
00:52:52.559 --> 00:52:55.559
tokenizer before we leave it, and the algorithm that's commonly
1030
00:52:55.639 --> 00:52:58.199
used something called byte pair and coding, and in my
1031
00:52:58.280 --> 00:53:00.280
classes is something we walk through. In fact, we do
1032
00:53:00.320 --> 00:53:02.760
it by hand so you can understand, and we talk
1033
00:53:02.800 --> 00:53:05.360
through the training process. But the key thing to understand
1034
00:53:05.400 --> 00:53:07.800
is that the model doesn't always see text the same
1035
00:53:07.840 --> 00:53:09.960
way you do. So that's the first step that's tokenizer.
1036
00:53:10.679 --> 00:53:13.440
Then the next thing is we we map each of
1037
00:53:13.480 --> 00:53:16.079
these tokens, but you can think of them as words
1038
00:53:16.119 --> 00:53:19.840
into a list of numbers. So I talked to earlier
1039
00:53:19.880 --> 00:53:21.760
like we map each word to a number, but it's
1040
00:53:21.800 --> 00:53:23.719
really we map it to a large list of numbers.
1041
00:53:23.760 --> 00:53:26.920
And this is called an embedding. And the way to
1042
00:53:26.960 --> 00:53:30.119
think about this is where we're taking all the words
1043
00:53:30.519 --> 00:53:33.320
or in this case, tokens technically, and we're putting them
1044
00:53:33.360 --> 00:53:36.920
on a map. But instead of like a two dimensional map,
1045
00:53:37.239 --> 00:53:39.519
this is many, many dimensions. So in the case of
1046
00:53:39.639 --> 00:53:42.199
GPT two, it's seven hundred and sixty eight I think
1047
00:53:42.360 --> 00:53:45.679
LAMA four or five B it's like sixteen thousand list
1048
00:53:45.760 --> 00:53:48.360
of numbers per every single word. In fact, like in
1049
00:53:48.400
--> 00:53:51.360
the sentence phrase Mike is quick, period he moves, the
1050
00:53:51.400 --> 00:53:54.079
period itself gets seven hundred and sixty eight numbers to
1051
00:53:54.079 --> 00:53:58.079
represent it. And you can think about like on a map,
1052
00:53:58.159 --> 00:54:00.639
you have like you know, coordinates. This is just a
1053
00:54:00.760 --> 00:54:05.679
very very multidimensional list of coordinates. And a good embedding
1054
00:54:05.880 --> 00:54:08.760
puts words that are related to each other closer to
1055
00:54:08.800 --> 00:54:10.760
each other. So in my class, they use the words
1056
00:54:10.920 --> 00:54:16.960
like happy, sad, joyful, glad, dog, cat, rabbit. The first
1057
00:54:17.000 --> 00:54:20.159
set of those are emotions happy, sad, joyful, right, and
1058
00:54:20.199 --> 00:54:22.719
would expect happy and joyful to be close to each other,
1059
00:54:22.840 --> 00:54:26.280
same with glad, and then dogcat and rabbit are totally unrelated,
1060
00:54:26.320 --> 00:54:28.440
so would expect them to be further apart on the map.
1061
00:54:29.119 --> 00:54:31.960
And the word sad is an emotion, but it's not
1062
00:54:32.280 --> 00:54:34.599
quite the same emotion as being happy, so it'd be
1063
00:54:34.599 --> 00:54:38.320
somewhere in between. And if you actually visualize this, you
1064
00:54:38.360 --> 00:54:40.880
see that this happens. It's actually putting words closer together.
1065
00:54:40.960 --> 00:54:44.239
And you might hear this paper or it's a series
1066
00:54:44.239 --> 00:54:47.920
of papers or algorithms called word to vec which pioneered
1067
00:54:47.920 --> 00:54:51.119
this model, And if you go to projector dot TensorFlow
1068
00:54:51.159 --> 00:54:53.760
dot org, you can actually see a three D map
1069
00:54:53.800 --> 00:54:55.519
of various words, and you click on it and it
1070
00:54:55.559 --> 00:54:57.039
will show you the words that are close to it,
1071
00:54:57.280 --> 00:54:59.519
and they all tend to be related words. So the
1072
00:54:59.559 --> 00:55:01.760
first thing we start with is, you know, the next
1073
00:55:01.760 --> 00:55:05.480
step after we break the text into tokens, as we
1074
00:55:05.519 --> 00:55:07.639
map each of those tokens to a position on a
1075
00:55:07.679 --> 00:55:10.880
map where close words are related to each other. Let
1076
00:55:10.880 --> 00:55:13.199
me pause and see if that made any sense. I'm
1077
00:55:13.280 --> 00:55:16.440
usually doing this all visually, so over pair audio. It's
1078
00:55:16.519 --> 00:55:19.559
it's a bit of a challenge, but yeah, good.
1079
00:55:19.960 --> 00:55:22.719
So my question is is, since it's predicting the next word,
1080
00:55:24.000 --> 00:55:26.400
I would imagine that, yeah, some of the words that
1081
00:55:26.400 --> 00:55:28.719
appear close to it are going to be words that
1082
00:55:30.079 --> 00:55:32.519
mean kind of the same thing or you know, have
1083
00:55:32.599 --> 00:55:35.519
a related meaning. But does it also group words together
1084
00:55:35.559 --> 00:55:38.800
that commonly appear together or is that a different Does
1085
00:55:38.800 --> 00:55:40.079
it not wait things that way at all?
1086
00:55:40.760 --> 00:55:45.719
Uh, it's actually kind of doing both. The way it's
1087
00:55:45.719 --> 00:55:48.880
grouping related words together, it doesn't actually group words together.
1088
00:55:49.800 --> 00:55:53.920
It's grouping words together that have the same meaning based
1089
00:55:53.960 --> 00:55:57.400
on the idea that they appear in the same places. Oh,
1090
00:55:57.440 --> 00:56:03.400
so the word ice and cold commonly occur together, probably
1091
00:56:03.440 --> 00:56:06.679
in text on the internet, right, like the I put
1092
00:56:06.840 --> 00:56:08.679
ice in the drink to make it cold, you're as
1093
00:56:08.719 --> 00:56:11.360
cold as ice? Right, Those would be common phrases you
1094
00:56:11.480 --> 00:56:15.519
usually don't see, you know, like steam and cold together.
1095
00:56:16.039 --> 00:56:20.119
And so the model is able to understand that ice
1096
00:56:20.199 --> 00:56:23.679
is colder than steam because it sees cold closer to
1097
00:56:23.920 --> 00:56:27.840
ice more often than seems cold close to steam. It's
1098
00:56:27.880 --> 00:56:33.960
the relative occurrence of how often. And there's a phrase
1099
00:56:34.039 --> 00:56:37.119
that's often used by JR. Firth. It's called you will
1100
00:56:37.119 --> 00:56:39.559
know a word by the company it keeps, which is
1101
00:56:39.599 --> 00:56:41.559
the idea you don't really know what a word means.
1102
00:56:41.599 --> 00:56:42.880
You could look it up in the dictionary, but you
1103
00:56:42.880 --> 00:56:45.360
really understand it through how it's used by multiple people,
1104
00:56:45.679 --> 00:56:48.119
and that you can look at, you know, the distribution
1105
00:56:48.199 --> 00:56:50.000
of how it's used to really understand what it means.
1106
00:56:50.000 --> 00:56:52.760
So good example is the word bad right, although it's
1107
00:56:52.840 --> 00:56:55.239
less in fashioned, bad at one time meant good, right,
1108
00:56:55.719 --> 00:56:58.960
And so how do you really understand whether it means
1109
00:56:58.960 --> 00:57:01.480
good in one context versus another? You learn that through
1110
00:57:01.480 --> 00:57:05.719
all the various contexts that it is used. And if
1111
00:57:05.719 --> 00:57:08.079
you want to understand, you know how word really is used.
1112
00:57:08.079 --> 00:57:10.119
You see it in usage many, many, men times. So
1113
00:57:10.159 --> 00:57:11.800
if you want a model to understand how what a
1114
00:57:11.840 --> 00:57:14.400
word means, you just see it used in many, many, many,
1115
00:57:14.480 --> 00:57:17.639
many sentences and eventually pick up on those differences.
1116
00:57:17.960 --> 00:57:19.920
So then the word baby could be seen as cold
1117
00:57:19.960 --> 00:57:25.519
because of vanilla ice, then right, ice ice baby.
1118
00:57:24.280 --> 00:57:26.719
If you train it, I'd be really interested to see
1119
00:57:26.719 --> 00:57:29.400
a model trained only on song lyrics. That would be that.
1120
00:57:29.519 --> 00:57:30.480
Would be fascinating.
1121
00:57:30.760 --> 00:57:34.119
Yeah, yeah, it's interesting because the way you're talking about it.
1122
00:57:35.039 --> 00:57:37.639
We were driving home from my mom's house the other
1123
00:57:37.760 --> 00:57:41.440
night and my wife put on an audiobook that she's
1124
00:57:41.440 --> 00:57:43.360
been listening to with my nine year old and they
1125
00:57:43.440 --> 00:57:46.280
used the word satisfaction and my daughter asked, what does
1126
00:57:46.320 --> 00:57:50.199
satisfaction mean? And we basically did that it's kind of
1127
00:57:50.280 --> 00:57:51.880
like this, and it's kind of like that, right, It's
1128
00:57:51.960 --> 00:57:55.920
it's in this area of meaning, right, yeah, and it's
1129
00:57:55.960 --> 00:57:57.199
related to these other words.
1130
00:57:57.239 --> 00:57:57.360
Right.
1131
00:57:57.400 --> 00:57:59.480
We used other words to explain it, and then yeah,
1132
00:57:59.480 --> 00:58:02.079
we did we context, so you could use it like this,
1133
00:58:02.239 --> 00:58:04.239
or you could use it like this, and it's you know,
1134
00:58:04.559 --> 00:58:07.599
another form of the word is satisfy or satisfied, and
1135
00:58:07.639 --> 00:58:10.000
so you know, this is what it means to satisfy
1136
00:58:10.079 --> 00:58:13.880
something and you know, more context and more sentences and okay,
1137
00:58:13.920 --> 00:58:16.239
I understand, right, And I think she may have even
1138
00:58:16.400 --> 00:58:18.320
said it's so it's kind of like this and kind
1139
00:58:18.320 --> 00:58:21.639
of like that, yep, using examples that we didn't use, and.
1140
00:58:21.599 --> 00:58:25.159
You told her that Nick couldn't get any right, that's right.
1141
00:58:26.960 --> 00:58:29.320
But yeah, that's that's basically what the model is going through.
1142
00:58:29.480 --> 00:58:33.440
Is it's like, uh, you know, basically saying seeing all
1143
00:58:33.440 --> 00:58:35.159
these examples and it's like, oh, it's kind of like this,
1144
00:58:35.320 --> 00:58:37.440
but it's like like in some context, I see it
1145
00:58:37.440 --> 00:58:40.360
being used in this other way, and so it's it's
1146
00:58:40.400 --> 00:58:42.760
basically putting that all together. And then it's putting all
1147
00:58:42.800 --> 00:58:46.599
the words on this map and it's saying, okay, you know,
1148
00:58:46.639 --> 00:58:48.639
the ones that are related are here, and the ones
1149
00:58:48.679 --> 00:58:51.960
and it's this multidimensional map. It's you know, hundreds to
1150
00:58:52.079 --> 00:58:57.119
thousands of dimensions long. And that's that's the embedding step.
1151
00:58:57.199 --> 00:58:59.679
Is we've basically mapped them to you know, I say
1152
00:58:59.679 --> 00:59:02.400
a numb but it's really a point in space, right,
1153
00:59:02.559 --> 00:59:06.000
It's basically taken. So there we're at the second step.
1154
00:59:06.280 --> 00:59:08.400
We base first step was we took the passage and
1155
00:59:08.440 --> 00:59:11.000
we broke it into tokens, which you can think of
1156
00:59:11.000 --> 00:59:13.360
as like words, but smaller, and then we took each
1157
00:59:13.400 --> 00:59:16.880
of those tokens and we put them into a point
1158
00:59:16.920 --> 00:59:19.480
on a map and a point in space, and we
1159
00:59:19.599 --> 00:59:21.400
just we know that point is going to be close
1160
00:59:21.440 --> 00:59:24.880
to other things that are related to it. So that's
1161
00:59:25.000 --> 00:59:31.679
the second step, and then the third step is called attention.
1162
00:59:32.840 --> 00:59:34.719
I'm going to skip over that in a second, and
1163
00:59:34.760 --> 00:59:36.320
I'm going to go to the fourth step, which is
1164
00:59:36.760 --> 00:59:40.519
the neural network or the multi layer perceptron. And this
1165
00:59:40.559 --> 00:59:43.320
gets to the training question. The key thing that it's
1166
00:59:43.360 --> 00:59:45.199
really great about neural networks is that thing I talked
1167
00:59:45.239 --> 00:59:47.360
about earlier, which is you don't have to give them
1168
00:59:47.360 --> 00:59:50.480
the rules. You just give them the answers, and they
1169
00:59:50.519 --> 00:59:54.159
figured out the rules. So we basically feed it in
1170
00:59:54.320 --> 00:59:59.320
these points in space from our prompt, and then we
1171
00:59:59.360 --> 01:00:01.280
can take a pasth message on the internet, like maybe
1172
01:00:01.280 --> 01:00:03.119
the passage on the internet is Mike is quick, he
1173
01:00:03.119 --> 01:00:07.360
moves quickly. We remove the word quickly, and then we
1174
01:00:07.480 --> 01:00:11.320
give the model the phrase Mike is quick he moves
1175
01:00:11.719 --> 01:00:13.199
and then we ask it to make a prediction, and
1176
01:00:13.239 --> 01:00:14.800
it's going to get it wrong because it hasn't done
1177
01:00:14.840 --> 01:00:17.599
any training at all, and when it gets it wrong,
1178
01:00:17.639 --> 01:00:20.159
maybe it says, you know, Mike's quick he moves bicycle
1179
01:00:20.559 --> 01:00:22.679
and you're like, no, that's wrong. The right answer is quickly.
1180
01:00:23.400 --> 01:00:27.599
It mathematically learns how to change itself to get closer
1181
01:00:27.599 --> 01:00:29.960
to that answer. So we go through a lot of
1182
01:00:29.960 --> 01:00:32.320
iterations of getting lots of passages where we take off
1183
01:00:32.320 --> 01:00:35.239
the last word and then we give it. We ask
1184
01:00:35.320 --> 01:00:37.400
it to predict it, and if it's good, we say okay, great,
1185
01:00:37.480 --> 01:00:40.199
you're fine, and if it's wrong, we say, okay, you're
1186
01:00:40.239 --> 01:00:41.880
off by this amount. It's kind of like
when you
1187
01:00:41.920 --> 01:00:44.639
throw darts out of board. Right, if you're far from
1188
01:00:44.719 --> 01:00:48.199
the bullseye, you'll move a lot more closer to correct
1189
01:00:48.199 --> 01:00:50.480
your position, But if you're close but slightly off, you're
1190
01:00:50.480 --> 01:00:52.679
going to move slightly subtly. So that's what it does.
1191
01:00:52.920 --> 01:00:56.079
It changes the model parameters, the numbers inside the model
1192
01:00:56.679 --> 01:00:59.599
slightly if it's close, or a lot if it's far away.
1193
01:00:59.719 --> 01:01:03.000
Does us you know, trillions of times over lots of
1194
01:01:03.039 --> 01:01:06.480
different pieces of data. And the key thing about the
1195
01:01:07.320 --> 01:01:10.280
neural network is it can learn to imitate basically from
1196
01:01:10.639 --> 01:01:15.360
answers and data, and so we basically give it the
1197
01:01:15.400 --> 01:01:19.519
known passage that we gave like Mike's queaky moves, and
1198
01:01:19.519 --> 01:01:22.000
we knew quickly was the right answer. And then we
1199
01:01:22.119 --> 01:01:24.320
basically asked the neural network to make a prediction from
1200
01:01:24.360 --> 01:01:28.960
these points in space. And so that's the that's the
1201
01:01:29.960 --> 01:01:32.480
basic version of what's happening inside the train. Let me
1202
01:01:32.480 --> 01:01:34.159
pause because I jump to the fourth Layer'll come back
1203
01:01:34.199 --> 01:01:37.079
to the third one in a second. But does that
1204
01:01:37.320 --> 01:01:38.719
let me see if there are any questions there on
1205
01:01:38.840 --> 01:01:45.360
what's happening inside the neural network. Okay, so yeah, okay,
1206
01:01:45.559 --> 01:01:48.239
So if we're good there, then the next thing that
1207
01:01:48.280 --> 01:01:51.320
happens is I'll jump back to the third Like we
1208
01:01:51.360 --> 01:01:53.199
could give it a point in space and say, hey,
1209
01:01:53.239 --> 01:01:56.320
guess what the next word is. But the best thing
1210
01:01:56.400 --> 01:01:57.960
to do is to not just give it a single
1211
01:01:57.960 --> 01:01:59.719
point in space. It's to give it all the points
1212
01:01:59.719 --> 01:02:01.840
that can before, so all the words that came before.
1213
01:02:01.920 --> 01:02:05.000
So in the case Mike is quick, he moves. Knowing
1214
01:02:05.039 --> 01:02:07.519
that you know we're talking about movement helps it know
1215
01:02:07.679 --> 01:02:09.880
that the word quick here is moving around in a
1216
01:02:09.880 --> 01:02:13.320
physical space versus the quick of your fingernail. And so
1217
01:02:13.360 --> 01:02:15.920
we give it the hints of all the words that
1218
01:02:15.960 --> 01:02:18.599
came before it. So this is what's called attention, where
1219
01:02:18.599 --> 01:02:21.599
we say, okay, don't just predict from one single word
1220
01:02:21.599 --> 01:02:23.320
you're looking at. This gets back to what we talked
1221
01:02:23.320 -->
01:02:26.519
about at the beginning, like instead of looking at, you know, statistically,
1222
01:02:26.519 --> 01:02:28.239
what's the next word after the given word, let me
1223
01:02:28.239 --> 01:02:30.320
look two words back, let me look three words back,
1224
01:02:30.360 --> 01:02:33.079
let me look forwards back. It will look at all
1225
01:02:33.159 --> 01:02:35.760
the words that came before it and try to figure
1226
01:02:35.760 --> 01:02:38.800
out what is the next predicted We're giving these hints
1227
01:02:38.800 --> 01:02:42.239
from the entire passage to make its prediction, and that's
1228
01:02:42.400 --> 01:02:45.719
what's called attention. And so that's the third step in
1229
01:02:45.760 --> 01:02:48.559
the middle. And then the last step is we do this,
1230
01:02:48.960 --> 01:02:51.280
you know, we get a prediction out of the neural network.
1231
01:02:51.400 --> 01:02:53.599
So jumping back to the fourth step, which was the
1232
01:02:53.880 --> 01:02:57.119
neural network that makes the prediction, and it gives a number,
1233
01:02:57.559 --> 01:03:00.480
and that number we need is a long list of numbers,
1234
01:03:00.519 --> 01:03:02.960
and we need to map that back to one of
1235
01:03:02.960 --> 01:03:05.639
our tokens, one of our words. But it's sitting in
1236
01:03:05.679 --> 01:03:08.599
a point in space that it may land right on
1237
01:03:08.679 --> 01:03:10.679
the word quickly, but more than likely it's going to
1238
01:03:10.760 --> 01:03:13.559
land somewhere close to the word quickly, like the predicted
1239
01:03:13.599 --> 01:03:16.039
token that comes up with it, that the model comes
1240
01:03:16.119 --> 01:03:18.719
up with, and so it will interpret that point in
1241
01:03:18.760 --> 01:03:21.039
space that it gave us back to those embeddings in
1242
01:03:21.039 --> 01:03:22.800
that map. It picked us some at the end of
1243
01:03:22.800 --> 01:03:25.320
the number crunching, took all the words and the points
1244
01:03:25.360 --> 01:03:27.559
in space we gave it and said the predicted word
1245
01:03:27.599 --> 01:03:31.360
is right here in this point in space, and it says, okay.
1246
01:03:31.639 --> 01:03:34.559
We then interpret that, and we look, what are the
1247
01:03:34.679 --> 01:03:37.400
words or tokens that are close to that predicted point
1248
01:03:37.440 --> 01:03:39.679
in space. And it's probably going to be closer to
1249
01:03:39.679 --> 01:03:42.760
the word fast, closer to the word around. Like Mike
1250
01:03:42.840 --> 01:03:47.559
moves quickly, he moves around, he moves fast, he moves speedily.
1251
01:03:47.760 --> 01:03:49.559
Those words are going to be close to it, and
1252
01:03:49.599 --> 01:03:52.320
so we give them a higher probability and we run
1253
01:03:52.360 --> 01:03:54.760
a random number generator and we say, okay, let me
1254
01:03:54.800 --> 01:03:57.800
pick one according to this probability distribution, and that's how
1255
01:03:57.840 --> 01:04:00.679
we get the predicted word out. And that last step
1256
01:04:00.760 --> 01:04:03.119
of running that random number generator and looking at what
1257
01:04:03.159 --> 01:04:05.079
words are closer it is the piece that's called the
1258
01:04:05.119 --> 01:04:10.880
language head. So the key thing about the language head
1259
01:04:10.960 --> 01:04:14.280
is that is where most of the uncertainty or unpredictability
1260
01:04:14.320 --> 01:04:17.199
of your model comes from. So if we decide not
1261
01:04:17.400 --> 01:04:19.599
to run the random number generator and we just always
1262
01:04:19.639 --> 01:04:22.280
pick the word that is closest in space to the prediction,
1263
01:04:22.679 --> 01:04:25.480
that's what's called temperature zero, and it will always be consistent.
1264
01:04:25.480 --> 01:04:27.599
It will always be predictable for the most part. There's
1265
01:04:27.639 --> 01:04:32.920
some other very small orders of randomness in the process,
1266
01:04:32.960 --> 01:04:35.400
but for the most part, it'll be very consistent, and
1267
01:04:35.440 --> 01:04:38.079
that's that's called temperature zero. So most of the randomness
1268
01:04:38.400 --> 01:04:42.920
inside the model is entirely in some sense imposed by us.
1269
01:04:43.039 --> 01:04:44.920
We decided, oh, we're not just going to always take
1270
01:04:44.960 --> 01:04:48.400
the thing that's closest. We're gonna probabilistically take some of
1271
01:04:48.400 --> 01:04:51.360
the other ones that are also close, and we can
1272
01:04:51.400 --> 01:04:55.519
control those parameters and control how we do that probability.
1273
01:04:55.519 --> 01:04:58.679
So if you're in o LLAMA or you know an API,
1274
01:04:59.239 --> 01:05:01.800
you'll see things like top P and top K or
1275
01:05:01.840 --> 01:05:05.440
temperature and these are tools we are given, you know,
1276
01:05:05.480 --> 01:05:08.119
the API user of a model on how they can
1277
01:05:08.199 --> 01:05:12.559
shape the probability distribution of the model. And that's probably
1278
01:05:12.599 --> 01:05:16.000
the most important to understand of the components in the model.
1279
01:05:16.239 --> 01:05:19.000
After you understand what tokens are and embeddings. The next
1280
01:05:19.000 --> 01:05:21.239
one is probably the language head because that's where the
1281
01:05:21.840 --> 01:05:23.960
randomness comes from. So let me pause. I know I
1282
01:05:24.039 --> 01:05:28.599
just talked for quite a while. See if there are
1283
01:05:28.599 --> 01:05:29.239
any questions.
1284
01:05:30.519 --> 01:05:31.760
I think, so far, so good.
1285
01:05:32.239 --> 01:05:38.159
Okay, So what the Excel spreadsheet does or the website
1286
01:05:38.199 --> 01:05:40.159
I have that's built in you know, web components in
1287
01:05:40.280 --> 01:05:45.519
pure JavaScript is it runs through the entire process using
1288
01:05:45.559 --> 01:05:49.039
the very same weights that open Ay released for a
1289
01:05:49.079 --> 01:05:52.880
model called GPT two GPT two small, and it steps
1290
01:05:52.880 --> 01:05:55.719
through every single one of those processes and it takes
1291
01:05:55.719 --> 01:05:57.880
you You enter a prompt and then what it does.
1292
01:05:58.039 --> 01:05:59.800
It doesn't it's not like chat GPT where you can
1293
01:05:59.800 --> 01:06:01.880
have a conversation with it. It just predicts the next word,
1294
01:06:01.920 --> 01:06:03.960
but it walks you through every single step. That's the
1295
01:06:03.960 --> 01:06:07.360
same thing I do inside the class. But that's basically
1296
01:06:07.480 --> 01:06:10.320
you know, how your model works under the hood is
1297
01:06:10.760 --> 01:06:15.159
it's basically taking your words, your input prompt, breaking it
1298
01:06:15.199 --> 01:06:17.960
into units that are called tokens that are slightly smaller
1299
01:06:18.039 --> 01:06:20.360
than a word, that it maps it to points in space,
1300
01:06:20.800 --> 01:06:22.719
does a bunch of number crunching on it through the
1301
01:06:22.760 --> 01:06:24.719
things I talked about, using a neural network and this
1302
01:06:24.760 --> 01:06:27.159
other attention that looks at all the other words, and
1303
01:06:27.199 --> 01:06:29.880
then it takes that prediction and it says, Okay, what
1304
01:06:29.920 --> 01:06:32.079
words is it close to in our points in space,
1305
01:06:32.280 --> 01:06:34.960
and then let me pick one that's relatively close to that.
1306
01:06:35.880 --> 01:06:38.360
So I know one of the things, Chuck, you would
1307
01:06:38.360 --> 01:06:41.360
want to talk about is building it and the use
1308
01:06:41.400 --> 01:06:42.840
of web components in the web version.
1309
01:06:43.280 --> 01:06:47.039
Yeah, at this point, given our time constraints, Yeah, we
1310
01:06:47.119 --> 01:06:48.639
might have you come back and do that, because I
1311
01:06:48.679 --> 01:06:50.639
think it'd be interesting to dive into the project and
1312
01:06:50.639 --> 01:06:51.360
how it went together.
1313
01:06:51.800 --> 01:06:54.679
Okay, I will. I will just say the reason I
1314
01:06:54.800 --> 01:06:59.199
built it in web components was to make it as
1315
01:06:59.280 --> 01:07:03.320
portable and easy to use and as easy to step through.
1316
01:07:03.440 --> 01:07:06.320
I wanted to make it as accessible. I did think about,
1317
01:07:06.400 --> 01:07:08.840
like say, using React, but then you need to know React,
1318
01:07:08.880 --> 01:07:12.079
and I really wanted this to be as approachable for
1319
01:07:12.199 --> 01:07:15.719
somebody who knows just vanilla JavaScript and web components was
1320
01:07:15.760 --> 01:07:16.480
the easiest.
1321
01:07:16.199 --> 01:07:16.719
Way to do that.
1322
01:07:17.079 --> 01:07:19.599
So that's the main reason why I did it that way. Yeah,
1323
01:07:19.679 --> 01:07:23.559
So is it open source then or I actually haven't
1324
01:07:23.559 --> 01:07:25.360
put a license on it. And what I've said is
1325
01:07:25.360 --> 01:07:28.480
if people feel like I help me decide, like, tell
1326
01:07:28.480 --> 01:07:31.239
me which licenses you prefer, but the code is right
1327
01:07:31.280 --> 01:07:33.159
there for people to look at. I mean you can
1328
01:07:33.199 --> 01:07:36.280
practically step through it, and it's written so that people
1329
01:07:36.320 --> 01:07:38.639
can understand it. I just haven't figured out what license,
1330
01:07:38.719 --> 01:07:42.639
but you know, let me know, and I'm all ears
1331
01:07:43.320 --> 01:07:45.039
all right. The goal is to make its a teaching tool.
1332
01:07:45.880 --> 01:07:48.559
Yeah, all right? Cool?
1333
01:07:49.800 --> 01:07:51.480
Well, yeah, like I said, we're kind of at the
1334
01:07:51.880 --> 01:07:54.960
end of our time and so I'm I'm going to
1335
01:07:55.000 --> 01:07:57.920
push this into picks. But you want to just give
1336
01:07:57.920 --> 01:08:01.039
out information on your course again, let people know what
1337
01:08:01.079 --> 01:08:02.440
that coupon code was.
1338
01:08:02.480 --> 01:08:05.039
I just if people are digging this as much as I.
1339
01:08:04.960 --> 01:08:07.239
Am, I think they may want to just go pick
1340
01:08:07.320 --> 01:08:08.920
up the course and go, okay, we can go into
1341
01:08:08.960 --> 01:08:09.440
more depth.
1342
01:08:10.440 --> 01:08:14.440
Uh yeah, thank you. So the website for the project
1343
01:08:14.559 --> 01:08:17.239
is called Spreadsheets Are All You Need with hyphens in between.
1344
01:08:17.640 --> 01:08:23.359
So spreadsheets hyphen are hyphen All hyphen you hyphenneed dot ai.
1345
01:08:24.279 --> 01:08:26.760
It is a very long domain name, and that will
1346
01:08:26.920 --> 01:08:29.039
link to where you can download the excel file. You
1347
01:08:29.079 --> 01:08:32.079
can try this out in the browser yourself, and then
1348
01:08:32.079 --> 01:08:33.960
there's a link to my class that I teach on
1349
01:08:34.039 --> 01:08:38.239
the Maven platform. It basically has five lectures over two
1350
01:08:38.239 --> 01:08:41.479
weeks and we walk through this for anybody who understands
1351
01:08:41.520 --> 01:08:45.520
spreadsheets or vanilla JavaScript. And I have a promo code
1352
01:08:45.680 --> 01:08:48.520
js jabber, so just use the promo code during checkout
1353
01:08:48.520 --> 01:08:53.720
and you get twenty percent off the courses taught live
1354
01:08:53.800 --> 01:08:56.159
but also is available on demand. So my last cohort
1355
01:08:56.239 --> 01:08:58.640
just wrapped up earlier this month. But if you sign up,
1356
01:08:58.720 --> 01:09:01.039
you'll get to watch all the recordings. I'll answer all
1357
01:09:01.079 --> 01:09:03.199
the questions over email you have. You'll be in the
1358
01:09:03.199 --> 01:09:05.439
same private discord as the same as the rest of
1359
01:09:05.479 --> 01:09:09.239
the cohort. And if for some reason you're watching it
1360
01:09:09.239 -->
01:09:11.279
on demand and you'd say I'd rather have the live version,
1361
01:09:12.079 --> 01:09:14.199
I offer that if you want to attend a future
1362
01:09:14.239 --> 01:09:16.159
live version, you can do that for free, even if
1363
01:09:16.159 --> 01:09:19.239
you signed up for the on demand, so you know,
1364
01:09:20.000 --> 01:09:22.399
feel free to check it out. It's on maven. It's
1365
01:09:22.560 --> 01:09:24.439
got some long URL, but if you go to spreadsheets
1366
01:09:24.439 --> 01:09:26.239
at all, you need dot ai you can check it
1367
01:09:26.279 --> 01:09:30.319
out and then to find me. I'm on Twitter I
1368
01:09:30.760 --> 01:09:32.640
A N A N D so my first initial with
1369
01:09:32.680 --> 01:09:34.840
my last name, and of course on LinkedIn. I'm also
1370
01:09:34.840 --> 01:09:37.239
on blue Sky as well. If people want to reach me,
1371
01:09:37.319 --> 01:09:37.720
happy to.
1372
01:09:37.680 --> 01:09:43.680
Answer questions awesome. Well, yeah, I definitely want to dig
1373
01:09:43.720 --> 01:09:45.560
into it. I'm probably gonna go watch your video on
1374
01:09:45.600 --> 01:09:49.680
YouTube a few more times, just you know, getting all
1375
01:09:49.720 --> 01:09:51.760
those little pieces in my head. So that yeah, I
1376
01:09:51.800 --> 01:09:54.079
think you said at the beginning, the model that matters,
1377
01:09:54.199 --> 01:09:57.399
matters most is your mental model. Yes, and so yeah,
1378
01:09:57.560 --> 01:09:59.960
just knowing how to think about Okay, I'm dropping this
1379
01:10:00.159 --> 01:10:02.479
in right. This is how it goes through the Plinko
1380
01:10:02.640 --> 01:10:04.479
machine to give me the output on the other end.
1381
01:10:05.479 --> 01:10:07.600
That that's the thing that really helps me out.
1382
01:10:07.680 --> 01:10:09.479
So yeah, that's a great analogy. And one thing that
1383
01:10:09.520 --> 01:10:12.279
maybe worth highlighting is I've had some feedback that people saw,
1384
01:10:12.960 --> 01:10:16.680
Oh you're using spreadsheets or using JavaScript. The real models
1385
01:10:16.680 --> 01:10:18.880
are in Python, and then you're using GPT two, which
1386
01:10:18.920 --> 01:10:22.520
is an older model. What I teach in the class
1387
01:10:22.680 --> 01:10:27.800
are essentially the timeless technical fundamentals of how these models work.
1388
01:10:27.920 --> 01:10:30.840
And it's worth remembering that all the major models you've
1389
01:10:30.840 --> 01:10:35.920
heard of, you know, Claude Chat, GPT, Gemini, they all
1390
01:10:36.039 --> 01:10:39.319
are inheriting from GPT two. So if you understand GPT two,
1391
01:10:39.359 --> 01:10:41.399
you understand eighty percent. You're eighty percent of the way
1392
01:10:41.399 --> 01:10:44.520
to understanding any of the model or Lama model architectures.
1393
01:10:45.600 --> 01:10:49.279
So it's it's not like toy. It is essentially as
1394
01:10:49.640 --> 01:10:51.439
very close to how the real models work. It's a
1395
01:10:51.479 --> 01:10:54.920
really good stepping stone to getting that really sharp mental
1396
01:10:54.920 --> 01:10:56.319
model of what's happening under the hood.
1397
01:10:57.399 --> 01:10:59.800
Yeah, that's true of most technologies, right, I mean, if
1398
01:10:59.840 --> 01:11:02.479
you were using I don't know, let's pick one my
1399
01:11:02.720 --> 01:11:07.319
SQL ten years ago, probably sixty seventy percent of the
1400
01:11:07.319 --> 01:11:09.760
stuff is fundamentally the same.
1401
01:11:09.920 --> 01:11:12.239
The engine works mostly the same.
1402
01:11:12.600 --> 01:11:16.479
They've probably optimized some pieces, they've probably add some features,
1403
01:11:16.680 --> 01:11:19.359
but for the most part, if you understand understood what
1404
01:11:19.720 --> 01:11:22.720
it was doing back, then you get it now. And
1405
01:11:22.760 --> 01:11:24.800
to be honest, the other thing is is you'll also
1406
01:11:24.880 --> 01:11:28.159
see as we get more variations on things, you also
1407
01:11:28.199 --> 01:11:30.880
have a decent understanding of how to use something like
1408
01:11:30.920 --> 01:11:32.680
SQL light or post PRESSQL as well.
1409
01:11:32.880 --> 01:11:35.920
So yeah, I really like that Algiae. It's a good one.
1410
01:11:36.000 --> 01:11:37.720
I M a bar of that, thank you.
1411
01:11:38.720 --> 01:11:40.640
Yeah, no problem. All right, Well let's go and do
1412
01:11:40.680 --> 01:11:43.600
our picks. AJ Do you want to start us with picks?
1413
01:11:44.720 --> 01:11:45.119
Sure?
1414
01:11:46.000 --> 01:11:52.159
Okay, so Civilization. I've still been playing that, not as
1415
01:11:52.199 --> 01:11:55.119
much as I was the other week, but enjoying it.
1416
01:11:55.279 --> 01:11:57.720
It turns out you can run it on the Mac
1417
01:11:58.479 --> 01:12:02.239
if you turn the you have to go into settings
1418
01:12:02.319 --> 01:12:05.880
and turn its performance mode completely off to all the
1419
01:12:05.880 --> 01:12:07.800
way down. I think it's something to do with multi threading,
1420
01:12:07.880 --> 01:12:10.279
was why it crashes, Like if he uses more than
1421
01:12:10.319 --> 01:12:12.960
one core, it just crashes every five minutes or something. Anyway,
1422
01:12:13.199 --> 01:12:16.479
so there's that civilization still going strong. But wanted to correct.
1423
01:12:16.520 --> 01:12:18.760
You can get it running on the Mac. It just
1424
01:12:18.800 --> 01:12:21.279
won't run on the Mac with the default settings, and
1425
01:12:21.319 --> 01:12:27.399
it's not abundantly clear why. But anyway, other thing was
1426
01:12:28.399 --> 01:12:31.520
with the announcement of the switch to I just got
1427
01:12:31.720 --> 01:12:35.920
angry because I still can't play Tiers of the Kingdom
1428
01:12:36.079 --> 01:12:40.600
or Super Mario RPG or Spiro or you know, any
1429
01:12:40.640 --> 01:12:43.840
game that's basically released in the last three years without
1430
01:12:43.960
--> 01:12:48.000
massive stuttering. And you know with Tiers of the Kingdom,
1431
01:12:48.039 --> 01:12:49.840
you know how they have it go into bullet time
1432
01:12:49.920 --> 01:12:55.279
whenever it gets overloaded instead of getting choppy, although it
1433
01:12:55.319 --> 01:12:59.840
still does that, it does dynamic resolution and bullet times,
1434
01:13:00.039 --> 01:13:05.159
so your swings will slow down and stuff, which you know, whatever.
1435
01:13:05.520 --> 01:13:10.000
So I decided to mod my switches and I did
1436
01:13:10.000 --> 01:13:13.119
the hardware mod of the switch and it was super easy.
1437
01:13:14.039 --> 01:13:17.199
Now I've done mods in the past, So saying that
1438
01:13:17.239 --> 01:13:20.840
it was super easy, No, if you're not familiar with soldering,
1439
01:13:20.880 --> 01:13:24.560
if you haven't you know, fixed a phone or or
1440
01:13:24.720 --> 01:13:27.079
you know, done something with that before, no it's not.
1441
01:13:27.159 --> 01:13:29.399
It's not super easy. Getting all the screws out, getting
1442
01:13:29.479 --> 01:13:31.840
to the actual part that you're getting the heat sync off,
1443
01:13:32.039 --> 01:13:36.119
that's super easy. Anybody that has a precision precision toolkit
1444
01:13:36.159 --> 01:13:39.079
for like phone repair, game repair or whatever, can can
1445
01:13:39.119 --> 01:13:43.920
get get in there and do that. I actually couldn't
1446
01:13:44.000 --> 01:13:47.520
see the soldering that I was doing because the components
1447
01:13:47.520 --> 01:13:50.039
are so small. Now I've learned some tricks because I've
1448
01:13:50.039 --> 01:13:52.720
practiced a little bit of micro soldering in the past
1449
01:13:52.760 --> 01:13:57.560
and failed at repairing a three DS, but the pieces
1450
01:13:57.560 --> 01:13:59.600
are so small that I literally can't see that. I
1451
01:13:59.640 --> 01:14:02.239
mean I can see them, but I can't see them.
1452
01:14:02.439 --> 01:14:05.000
I mean like I can see them as like I
1453
01:14:05.039 --> 01:14:09.439
can see a grain of sand, but I cannot see
1454
01:14:09.439 --> 01:14:12.319
them in terms of like actually accurately predicted. So what
1455
01:14:12.359 --> 01:14:15.720
I did was I used my phone to zoom in,
1456
01:14:15.960 --> 01:14:18.960
take a picture of it, see that I had bridged
1457
01:14:19.000 --> 01:14:22.520
two capacitors, and then just kind of blindly, you know,
1458
01:14:22.600 --> 01:14:24.720
move the soldering iron next to them. And kind of
1459
01:14:24.800 --> 01:14:26.840
sweeped it the same way that I would when i'm,
1460
01:14:26.960 --> 01:14:29.479
you know, soldering on a bigger component. Then use the
1461
01:14:29.520 --> 01:14:33.640
phone again zoomed in, and so I was able. I
1462
01:14:33.760 --> 01:14:36.199
was able to get the peace on there. And I
1463
01:14:36.399 --> 01:14:39.279
should have had some sort of magnifying glass set up,
1464
01:14:40.359 --> 01:14:43.239
but whatever, so I was. I was actually able to
1465
01:14:43.239 --> 01:14:45.520
do it blind in a way like I could. I
1466
01:14:45.520 --> 01:14:48.479
could see my tip was there, but I couldn't actually
1467
01:14:48.520 --> 01:14:51.079
see what, you know, what was because I mean the
1468
01:14:51.119 --> 01:14:53.720
things they're they're smaller than a grain of salt. They're
1469
01:14:53.760 --> 01:14:58.039
they're small anyway, the capacitors they're they're like wicked small.
1470
01:14:58.600 --> 01:15:02.159
But even with that, I was able to do it.
1471
01:15:02.239 --> 01:15:05.520
And that's my my So. So modding the switches is
1472
01:15:05.600 --> 01:15:09.800
one is one pick there. There's the Pico Fly mod kit.
1473
01:15:09.880 --> 01:15:11.520
You can do it if you've done if you've done
1474
01:15:11.560 --> 01:15:13.960
other soldering, if you haven't done micro soldering, buy a
1475
01:15:13.960 --> 01:15:19.000
couple of practice kits from eBay or or Ali Express
1476
01:15:19.119 --> 01:15:21.079
or or something and you can get there. But my
1477
01:15:21.119 --> 01:15:25.199
third pick is the soldering station that I used is
1478
01:15:25.359 --> 01:15:29.920
actually a custom made soldering station. So a few years ago,
1479
01:15:30.079 --> 01:15:32.720
these Chinese companies came out with the T twelve tips,
1480
01:15:32.800 --> 01:15:35.479
or they cloned the T twelve tips. They turn out
1481
01:15:35.479 --> 01:15:39.439
to be really, really good. One reason is that they
1482
01:15:39.520 --> 01:15:42.640
double over as a temperature sensor because they have two
1483
01:15:42.680 --> 01:15:45.560
different types of metals, and anytime you have two different
1484
01:15:45.600 --> 01:15:47.439
types of metals, you have a thermal couple. So they
1485
01:15:47.439 --> 01:15:50.279
have an inner metal and an outer metal, so they
1486
01:15:50.359 --> 01:15:55.279
double over as their own temperature sensor. And so people
1487
01:15:55.640 --> 01:15:59.159
created software for these and put them on micro controllers.
1488
01:15:59.159 --> 01:16:02.359
In the software, and being literal when I say this,
1489
01:16:02.479 --> 01:16:06.319
it rivals three thousand dollars professional workstations because the way
1490
01:16:06.319 --> 01:16:09.720
that it switches back and forth between monitoring the temperature
1491
01:16:09.720 --> 01:16:13.600
of the tip. And so I'll put a link and
1492
01:16:13.640 --> 01:16:15.920
you can get these. You can get clones on all
1493
01:16:15.960 --> 01:16:19.680
the express but I prefer the original one that's made
1494
01:16:19.680 --> 01:16:21.800
by this guy in Australia because I know it comes
1495
01:16:21.800 --> 01:16:24.600
with the right firmware on it, and the firmware is
1496
01:16:24.640 --> 01:16:28.359
really where the magic happens. Any idiot, can you know?
1497
01:16:28.479 --> 01:16:32.840
Three D print some leads onto a rigid battery or
1498
01:16:32.880 --> 01:16:37.039
a Milwaukee
battery and connect it to a T twelve tip.
1499
01:16:37.079 --> 01:16:39.840
But the real smarts of it is in the firmware
1500
01:16:39.840 --> 01:16:42.039
where it manages the heat to make sure it gets
1501
01:16:42.119 --> 01:16:46.000
up to temperature quickly that it actually does the sensing
1502
01:16:46.039 --> 01:16:48.840
thing where if you shake it, it turns back on
1503
01:16:48.960 --> 01:16:53.560
and heats up anytime. Anytime the temperature is cooling down rapidly,
1504
01:16:53.600 --> 01:16:56.840
it sends more power. So anyway, it's just a really
1505
01:16:56.880 --> 01:16:59.560
great soldering iron. And because I have that, like I've
1506
01:16:59.560 --> 01:17:03.039
got a and I've got a couple of cheap ones,
1507
01:17:03.039 --> 01:17:06.479
but that one it cost a hundred bucks, and I'm
1508
01:17:06.520 --> 01:17:09.199
considering trying out one of the knockoffs. It's only like
1509
01:17:09.239 --> 01:17:13.279
thirty five because now everybody, even Craftsman, is selling one
1510
01:17:13.279 --> 01:17:14.920
of these at Low's now. But I don't know if
1511
01:17:14.920 --> 01:17:18.159
the Craftsman one is just like the cheap idiot kind
1512
01:17:18.199 --> 01:17:20.760
where it's just connecting the leads together, or if it's
1513
01:17:20.760 --> 01:17:23.560
actually got a I have a hard time believing that
1514
01:17:23.600 --> 01:17:26.479
they would have gotten an illegal copy of the firmware,
1515
01:17:26.479 --> 01:17:28.920
whereas the Chinese companies on expression you know that anyway
1516
01:17:29.239 --> 01:17:34.760
saw that. Yeah, so I had a good time soldering
1517
01:17:35.079 --> 01:17:38.600
because of the rigid powered which you give them Mount
1518
01:17:38.680 --> 01:17:42.119
Milwaukee or Iobi or whatever battery brand you like, soldering iron.
1519
01:17:42.199 --> 01:17:44.840
They're super fast. There's so much better than a weller
1520
01:17:45.039 --> 01:17:49.079
or a Hako or all the traditional ones that cost
1521
01:17:49.239 --> 01:17:54.520
hundreds of dollars. So anyway, and of course I'll pick
1522
01:17:54.520 --> 01:17:57.760
a LLAMA because I really do enjoy running my own
1523
01:17:58.520 --> 01:18:05.079
local lams. Actually, since the thirty two billion parameter model
1524
01:18:05.119 --> 01:18:08.279
of Quinn two point five coder has come out, that
1525
01:18:08.319 --> 01:18:10.600
one I just find to be the best of the best.
1526
01:18:10.720 --> 01:18:14.039
It rivals GPT four oh if not is better than
1527
01:18:14.039 --> 01:18:16.279
four to oh, and you can run it on an
1528
01:18:16.359 --> 01:18:19.039
Apple silken back boom, all the things.
1529
01:18:19.479 --> 01:18:21.920
Okay, I have a question, what are you modeling your
1530
01:18:21.960 --> 01:18:22.960
switch to do?
1531
01:18:23.720 --> 01:18:23.920
Oh?
1532
01:18:24.159 --> 01:18:28.880
Sorry, I skipped over that part over well, not overclock
1533
01:18:29.159 --> 01:18:38.800
to the native CPU freak speed of the tegra x one,
1534
01:18:38.920 --> 01:18:42.840
because the switch is an Android tablet running a custom
1535
01:18:42.880 --> 01:18:47.520
operating system rather than Android is two point two gigahertz.
1536
01:18:47.760 --> 01:18:51.359
That's the native clock speed. The clock speed that it
1537
01:18:51.479 --> 01:18:55.319
runs at is something like a thousand or seven hundred,
1538
01:18:55.359 --> 01:18:58.359
depending on whether it's doctor handheld. Same thing with the GPU.
1539
01:18:58.399 --> 01:19:01.079
The native GPU speed is like fifty one point five
1540
01:19:01.159 --> 01:19:04.279
gigahertz or something like that, but they clock it down
1541
01:19:04.319 --> 01:19:06.199
to five hundred or seven hundred.
1542
01:19:06.560 --> 01:19:07.159
Gotcha.
1543
01:19:07.720 --> 01:19:12.279
So when you mod it, you can then and this
1544
01:19:12.399 --> 01:19:14.560
you can do without getting banned or at least this
1545
01:19:14.600 --> 01:19:17.000
is what people are reporting, and I'm I'm doing this
1546
01:19:17.079 --> 01:19:18.640
so you know, if you mod it and you want
1547
01:19:18.640 --> 01:19:21.880
to run pirated games or something, you have to set
1548
01:19:21.960 --> 01:19:24.159
up more stuff and make sure that you don't get banned,
1549
01:19:24.199 --> 01:19:25.880
although a lot of that stuff is built in now.
1550
01:19:26.119 --> 01:19:28.079
But if all you want to do is overclock it,
1551
01:19:28.159 --> 01:19:31.399
the overclock system runs in a layer that's kind of
1552
01:19:31.920 --> 01:19:35.720
protected from the main switch system, so the main switch
1553
01:19:35.800 --> 01:19:39.800
system can't detect that it's rooted while it's running, So
1554
01:19:39.880 --> 01:19:44.159
you can you if you just install hakat what does it?
1555
01:19:46.279 --> 01:19:51.319
Hikata Atmosphere and cisclock. If those are the only things
1556
01:19:51.319 --> 01:19:55.000
you install, then you should be able to run your
1557
01:19:55.000 --> 01:19:58.439
switch modded on the original firmware, be able to play online,
1558
01:19:58.600 --> 01:20:02.760
et cetera, without any risk of banning or anything, because
1559
01:20:02.800 --> 01:20:07.720
it's the where it's not modifying the switch operating system
1560
01:20:07.800 --> 01:20:10.920
or the game. It's just modifying the CPU clock.
1561
01:20:12.359 --> 01:20:14.600
Gotcha cool?
1562
01:20:15.039 --> 01:20:17.680
So now, so now my friend asked me, well, can
1563
01:20:17.720 --> 01:20:20.760
you notice any difference? And my answer is no. And
1564
01:20:20.960 --> 01:20:23.439
the reason my answer is no is because when you're
1565
01:20:23.439 --> 01:20:27.039
not playing, when you're playing it underclocked, you notice the
1566
01:20:27.119 --> 01:20:32.039
stuttering all the time, and like you notice the resolution changing,
1567
01:20:32.119 --> 01:20:34.079
Like you know, you turn and there's a bad guy
1568
01:20:34.079 --> 01:20:36.840
and you
shoot him, and then the resolution Like, but
1569
01:20:36.920 --> 01:20:39.800
when you're playing it closer to native speeds, you can't
1570
01:20:39.840 --> 01:20:41.560
get it all the way up to native speeds because
1571
01:20:41.560 --> 01:20:44.159
the power delivery on the board isn't actually capable of
1572
01:20:44.520 --> 01:20:48.000
playing it at native speeds without also draining the battery
1573
01:20:48.000 --> 01:20:50.560
at the same time. But when you're playing it at
1574
01:20:50.640 --> 01:20:54.399
near native speeds, you don't notice it because you like,
1575
01:20:54.439 --> 01:20:57.920
the things that are annoying aren't there. The resolution's not changing,
1576
01:20:57.960 --> 01:21:00.239
it's not stuttering, it's not going into bullet time. As
1577
01:21:00.279 --> 01:21:02.720
much I haven't, I did not notice it at all
1578
01:21:02.760 --> 01:21:04.920
going into bullet time since I've been playing it at
1579
01:21:04.960 --> 01:21:07.960
near native speeds, and I did some things where I
1580
01:21:08.000 --> 01:21:10.880
was blowing up rocks and things that I thought would
1581
01:21:10.920 --> 01:21:12.680
normally make it go to bullet time, and it didn't.
1582
01:21:12.960 --> 01:21:16.079
So like five star stories, so far.
1583
01:21:16.840 --> 01:21:20.840
Cool, very cool, All right, Steve, what are your picks?
1584
01:21:21.720 --> 01:21:22.079
All right?
1585
01:21:22.199 --> 01:21:27.279
Kind for my twenty minute picks. So before I get
1586
01:21:27.319 --> 01:21:29.680
in picks, one note, I will make it sort of
1587
01:21:29.720 --> 01:21:33.239
circles back to what I asked at the beginning. You know,
1588
01:21:33.319 --> 01:21:35.720
as someone who has spent a lot of time doing
1589
01:21:35.720 --> 01:21:40.960
search indexing, you know, with lucine type search indexes. A
1590
01:21:41.039 --> 01:21:43.560
lot of this sounds really familiar. And to me, I've
1591
01:21:43.600 --> 01:21:46.640
always said that the I and AI is a misnomer.
1592
01:21:47.319 --> 01:21:51.119
I think it's it's not necessarily intelligent. It's just basically
1593
01:21:51.840 --> 01:21:55.520
better using of training and fancy or using of existing
1594
01:21:55.600 --> 01:22:00.680
data to answer things, not necessarily intelligence that and create
1595
01:22:00.760 --> 01:22:06.199
new things. That's just my two cents for what it's worth. Interesting,
1596
01:22:06.279 --> 01:22:09.720
pick Eean, you mentioned this earlier today and as of
1597
01:22:09.800 --> 01:22:12.079
today that you know, this will come out a little later,
1598
01:22:12.479 --> 01:22:16.640
but deep seek is like disrupting in a huge way.
1599
01:22:17.239 --> 01:22:17.439
You know.
1600
01:22:17.560 --> 01:22:21.960
For instance, if you go look on Hacker News, both
1601
01:22:22.039 --> 01:22:25.479
on the top page and on the new page, there
1602
01:22:25.479 --> 01:22:31.159
are multiple articles from NPR from CNBC about what it's
1603
01:22:31.159
--> 01:22:35.119
doing to the stock market, and the gist is basically
1604
01:22:35.239 --> 01:22:38.640
that they've been able to create these fantastic models with
1605
01:22:39.000 --> 01:22:44.000
much less investment, with less powerful chips. And there's a
1606
01:22:44.039 --> 01:22:47.800
whole story behind this, and so that's wreaking have at
1607
01:22:47.920 --> 01:22:50.239
least in the stock market and with people like Nvidia,
1608
01:22:50.840 --> 01:22:54.359
just because of supposedly how how much cheaper and more
1609
01:22:54.359 --> 01:22:58.720
efficient deep seek is compared to some of the other models.
1610
01:22:58.760 --> 01:23:02.560
So today's I've the first day and know it remains
1611
01:23:02.600 --> 01:23:05.319
to be seen how accurate this is, especially coming from
1612
01:23:05.319 --> 01:23:10.479
the Chinese, but sort of a disruptive thing going on
1613
01:23:10.520 --> 01:23:12.600
this morning, at least as of the time of recording.
1614
01:23:13.720 --> 01:23:15.600
Yeah, do you mind if I jump in there a
1615
01:23:15.640 --> 01:23:19.760
little bit. Deep Seek is an utterly fascinating story and model.
1616
01:23:22.119 --> 01:23:27.920
I'll say one thing is that the training cost might
1617
01:23:27.960 --> 01:23:33.840
have been apples to oranges. Like they stated what the
1618
01:23:33.880 --> 01:23:36.800
cost was for the best run or the final run.
1619
01:23:37.680 --> 01:23:39.640
There's a lot of other costs that go into I
1620
01:23:39.680 --> 01:23:41.720
talked earlier when AJ was asking what goes into it.
1621
01:23:41.720 --> 01:23:43.359
One of the key things I was thinking actually about
1622
01:23:43.399 --> 01:23:45.800
Deep Seak is like they're going to do a lot
1623
01:23:45.800 --> 01:23:47.960
of other experiments and runs. There's a lot of stuff
1624
01:23:48.000 --> 01:23:50.840
that gets built upon. So I think some people are
1625
01:23:51.920 --> 01:23:56.119
comparing apples to oranges, but it is, you know, a
1626
01:23:57.039 --> 01:23:59.680
impressive model in a lot of ways. The other thing
1627
01:23:59.720 --> 01:24:02.760
that I find the most fascinating about it is the
1628
01:24:02.800 --> 01:24:10.560
training process is remarkably simple. And you know, I'm trying
1629
01:24:10.560 --> 01:24:13.760
to think of an analogy. It's like, normally when they
1630
01:24:13.840 --> 01:24:16.600
do this part of the training process called reinforcement learning.
1631
01:24:16.800 --> 01:24:21.319
It's a lot more complex, and it's kind of like,
1632
01:24:21.560 --> 01:24:23.000
you know, you think about a car and you're like, well,
1633
01:24:23.000 --> 01:24:24.319
if you want to go from point A to point B,
1634
01:24:24.439 --> 01:24:28.239
you need an internal combustion engine. It's basically doing you know,
1635
01:24:28.600 --> 01:24:30.439
having a little fire and you got these pistons and
1636
01:24:30.439 --> 01:24:34.359
the cylinders, really complex piece of mechanics. And then somebody's
1637
01:24:34.359 --> 01:24:36.000
like the electric car, and so you know that little
1638
01:24:36.039 --> 01:24:38.399
toy motor you had, Well, let's just scale that thing up.
1639
01:24:38.840 --> 01:24:42.039
And so they tried this really simple, relatively simple technique
1640
01:24:42.079 --> 01:24:44.680
and just scaled it up and it worked. And I
1641
01:24:44.720 --> 01:24:48.479
think different people are reacting to this model differently. Some
1642
01:24:48.520 --> 01:24:51.680
people it's about the price, other people it's about the
1643
01:24:51.720 --> 01:24:54.840
training setup, and it's how did we miss this? It's
1644
01:24:54.920 --> 01:25:00.319
just remarkably simple. So it's definitely a worthy to bring
1645
01:25:00.399 --> 01:25:02.159
up as it's just a really interesting model.
1646
01:25:02.560 --> 01:25:04.000
O Coom's razor strikes again.
1647
01:25:04.239 --> 01:25:04.479
Right.
1648
01:25:05.119 --> 01:25:08.720
Well, there's something they call in AI the bitter lesson,
1649
01:25:09.359 --> 01:25:12.760
which is stop trying to put into the model how
1650
01:25:12.840 --> 01:25:17.359
you think you think. Instead just give really general compute
1651
01:25:17.640 --> 01:25:19.680
and just throw more and more data and more and
1652
01:25:19.720 --> 01:25:23.079
more compute at the problem, and the model will figure
1653
01:25:23.119 --> 01:25:25.479
it out, so don't try to be too smart about it.
1654
01:25:26.000 --> 01:25:28.439
And people are like, this was a bitter lesson all
1655
01:25:28.439 --> 01:25:30.600
over again. It's like, oh, we thought, you know, we
1656
01:25:30.640 --> 01:25:34.520
had to do this really complex, you know, reinforcement learning setup,
1657
01:25:34.600 --> 01:25:37.800
and these guys showed, well, maybe you don't now. In
1658
01:25:37.840 --> 01:25:40.600
the end, their production model actually still had some complex
1659
01:25:40.600 --> 01:25:43.359
training pipeline. But one of the interesting results is this
1660
01:25:43.439 --> 01:25:47.319
model called zero, where it kind of like, you know,
1661
01:25:47.399 --> 01:25:49.960
Alpha zero learned how to become a really good go
1662
01:25:50.039 --> 01:25:52.560
player just by playing against itself. In this case, it
1663
01:25:52.600 --> 01:25:54.680
wasn't the model playing against itself. It was just trying
1664
01:25:54.720 --> 01:25:57.920
out ideas and they just told it whether it was
1665
01:25:58.000 --> 01:26:02.600
right or wrong, and it started automatically emergently figuring out
1666
01:26:02.680 --> 01:26:05.560
how to improve its thinking. And it starts getting these
1667
01:26:05.600 --> 01:26:08.600
eureka moments where it suddenly realizes it can backtrack and
1668
01:26:08.640 --> 01:26:10.960
it's like, oh, wait, I made a mistake, and it's
1669
01:26:11.319 --> 01:26:13.119
you're watching the model like we didn't train it to
1670
01:26:13.159 --> 01:26:15.640
do this, and it suddenly figures
out how to like
1671
01:26:15.720 --> 01:26:20.399
get smarter. So it's really really fascinating. I could we
1672
01:26:20.439 --> 01:26:24.359
could talk another hour on the model, but yeah, lots
1673
01:26:24.399 --> 01:26:27.760
of people are pouring over it. It's It's fascinating in many dimensions.
1674
01:26:28.399 --> 01:26:28.680
Cool.
1675
01:26:30.239 --> 01:26:32.880
And then, last, but not least certainly, the high point
1676
01:26:32.880 --> 01:26:34.760
of any episode is the dad jokes of the week.
1677
01:26:37.479 --> 01:26:40.319
So what did one pie say to the other pie
1678
01:26:40.680 --> 01:26:44.000
before being put in the oven? You know this is
1679
01:26:44.039 --> 01:26:47.840
a musical answer. All we are is crusted in the tin.
1680
01:26:49.760 --> 01:26:55.279
For anybody that knows Kansas, here's an Australian version. My
1681
01:26:55.319 --> 01:26:57.439
mate was bitten by a snake, so I told him
1682
01:26:57.439 --> 01:27:00.520
an amusing story. If I know the difference between anecdote
1683
01:27:00.560 --> 01:27:07.119
and antidote, he'd still be alive. And finally, when I
1684
01:27:07.159 --> 01:27:09.880
was in school, my teachers told me I would never
1685
01:27:09.880 --> 01:27:12.880
amount to anything because I procrastinated too much. I told him,
1686
01:27:13.119 --> 01:27:18.159
you just wait, those are the dad jokes of the week.
1687
01:27:19.960 --> 01:27:21.920
All right, well, I'm going to jump in here and
1688
01:27:22.640 --> 01:27:26.800
save us from the high point of the episode. I've
1689
01:27:26.800 --> 01:27:28.399
got a couple of picks. I always do a board
1690
01:27:28.399 --> 01:27:30.520
game pick. So the game I'm gonna pick I learned
1691
01:27:30.520 --> 01:27:34.680
this game last week is called Cascadero. I'm gonna put
1692
01:27:34.920 --> 01:27:37.920
links in for both board game Geek, which kind of
1693
01:27:37.920 --> 01:27:40.000
gives you information about the board game, and then an
1694
01:27:40.000 --> 01:27:46.960
Amazon affiliate link, just because then you know where to
1695
01:27:46.960 --> 01:27:52.640
go buy it if you want it anyway, So Cascadero,
1696
01:27:52.760 --> 01:27:56.880
the premise of the game is that the kingdom's breaking up,
1697
01:27:57.560 --> 01:28:02.680
and so you all play a different faction, I guess,
1698
01:28:03.359 --> 01:28:07.520
and you're trying to connect towns and send your people
1699
01:28:07.520 --> 01:28:12.399
through the towns to pull the kingdom back together. And
1700
01:28:13.439 --> 01:28:15.319
so you put your little guys out there, and then
1701
01:28:15.359 --> 01:28:18.239
you score points based on whether you're the first person
1702
01:28:18.279 --> 01:28:20.239
of the town or the second person of the town.
1703
01:28:22.000 --> 01:28:24.000
If you have a group, you have to have a
1704
01:28:24.000 --> 01:28:26.840
group of your little horsemen.
1705
01:28:27.159 --> 01:28:30.479
I can't remember what they call them, heralds. No, the
1706
01:28:30.520 --> 01:28:31.159
heralds were the.
1707
01:28:31.079 --> 01:28:33.319
Other things anyway, So if you connect to a town
1708
01:28:33.359 --> 01:28:36.119
with a herald, then you get an extra point for
1709
01:28:36.159 --> 01:28:40.119
connecting to it, and then there are bonuses that you get.
1710
01:28:40.159 --> 01:28:43.359
So if because when you connect, when you get the points,
1711
01:28:43.399 --> 01:28:47.800
you actually move a marker up the technology or progress
1712
01:28:47.840 --> 01:28:52.239
track in whatever color you connected to. And so I
1713
01:28:52.239 --> 01:28:54.960
guess they're not points, they're just movements. But anyway, so
1714
01:28:55.079 --> 01:29:00.319
once you move past certain points, you get certain rewards,
1715
01:29:00.680 --> 01:29:04.000
and I mean effectively, what you're trying to do is
1716
01:29:04.039 --> 01:29:09.079
you're trying to score the most points, and you also
1717
01:29:09.119 --> 01:29:10.600
want to get to the end of the track and
1718
01:29:10.640 --> 01:29:13.159
whatever color you're playing. So if you're playing pink, you
1719
01:29:13.159 --> 01:29:15.600
want your pink marker to get all the way to
1720
01:29:15.640 --> 01:29:19.239
the end. And yeah, like I said, you get bonus
1721
01:29:19.239 --> 01:29:22.159
points for getting all five of your markers past the
1722
01:29:22.239 --> 01:29:26.039
first space.
1723
01:29:25.680 --> 01:29:26.680
That's marked for that.
1724
01:29:27.840 --> 01:29:30.439
You get more bonus points if you get three past
1725
01:29:30.520 --> 01:29:33.680
the second spacing, and then if you're the first one
1726
01:29:33.720 --> 01:29:34.840
to get one all the way to the end, then
1727
01:29:34.840 --> 01:29:37.439
you get bonus points and only one person can get those,
1728
01:29:37.479 --> 01:29:39.479
and then the other ones are if you connect two
1729
01:29:39.520 --> 01:29:41.560
cities of the same color and there are five colors,
1730
01:29:42.600 --> 01:29:45.359
you get bonus points for each color, and if you
1731
01:29:45.359 --> 01:29:48.319
get all five colors, then you get ten bonus points.
1732
01:29:49.760 --> 01:29:52.079
And so you're just moving your marker around a track
1733
01:29:52.439 --> 01:29:54.279
when you get the points. As soon as somebody gets
1734
01:29:54.319 --> 01:29:59.159
fifty points, the game ends. And so essentially, if you
1735
01:29:59.199 --> 01:30:01.479
want to win, you want to be the first person
1736
01:30:01.520 --> 01:30:03.760
to get your marker all the way to the end
1737
01:30:03.800 --> 01:30:05.800
of the track of your color and then be the
1738
01:30:05.840 --> 01:30:10.079
person that gets that fiftieth point. We played it, and
1739
01:30:10.399 --> 01:30:12.399
it was our first time any of us playing it,
1740
01:30:12.760 --> 01:30:18.239
and so nobody crossed that fiftieth point before we all
1741
01:30:18.319 --> 01:30:21.800
ran out of Little Horseman, and
so when somebody runs out,
1742
01:30:21.800 --> 01:30:24.319
everybody gets one more turn, or if somebody crosses that
1743
01:30:24.760 --> 01:30:28.600
fifty point marker, everybody else gets one more turn, and
1744
01:30:28.640 --> 01:30:32.640
then the game's over. It's reasonably simple. The scoring is
1745
01:30:32.680 --> 01:30:35.399
a little bit complicated as far.
1746
01:30:35.319 --> 01:30:36.199
As like moving.
1747
01:30:36.279 --> 01:30:38.079
You know, when you get moves and how many moves
1748
01:30:38.119 --> 01:30:40.960
and things like that. So board game geek waits at
1749
01:30:40.960 --> 01:30:43.479
at two point five to three, right, So it's a
1750
01:30:43.520 --> 01:30:48.039
little more complicated than kind of your casual gamer who's
1751
01:30:48.079 --> 01:30:51.960
just going to play gravel with their friends. But my
1752
01:30:52.079 --> 01:30:54.800
feeling is is that it was only just getting used
1753
01:30:54.840 --> 01:30:59.399
to when I put my horseman down what happens. And
1754
01:30:59.439 --> 01:31:01.079
as soon as you get used to I put my
1755
01:31:01.119 --> 01:31:03.159
horsemen down, I can move something up the track so
1756
01:31:03.199 --> 01:31:05.960
many spaces, and then how to get the rewards. Once
1757
01:31:06.000 --> 01:31:10.119
you figure that out, it's a relatively simple game. We
1758
01:31:10.199 --> 01:31:13.640
played it in what an hour, maybe a little longer,
1759
01:31:14.439 --> 01:31:16.600
I think once. If we'd known what we were doing,
1760
01:31:17.439 --> 01:31:19.399
we could probably play it in forty five minutes. There
1761
01:31:19.399 --> 01:31:24.119
were three of us playing it, so anyway, Coscadero.
1762
01:31:25.319 --> 01:31:27.119
Fun, fun game. I liked it.
1763
01:31:27.279 --> 01:31:29.039
I want to play it again now that we I
1764
01:31:29.119 --> 01:31:30.840
know how to play it and my friends and how
1765
01:31:30.880 --> 01:31:32.920
to play it, because there were a couple of.
1766
01:31:32.840 --> 01:31:35.279
Things I would have done differently as I gotten into it.
1767
01:31:37.520 --> 01:31:44.680
As far as other picks, go.
1768
01:31:41.680 --> 01:31:43.840
Go to jsgeniuses dot com and sign up.
1769
01:31:43.880 --> 01:31:47.199
We're gonna start doing the meetups and I'm gonna start
1770
01:31:47.199 --> 01:31:53.239
posting videos. The videos I'm posting the I'm kind of
1771
01:31:53.279 --> 01:31:55.520
building an entire app. I don't know if I'm going
1772
01:31:55.560 --> 01:31:58.079
to show you writing all the code because some of
1773
01:31:58.079 --> 01:32:00.800
the stuff gets repetitive. Oh, I have to connect another
1774
01:32:00.960 --> 01:32:03.800
data model to this database. Right, It's like, Okay, you
1775
01:32:03.800 --> 01:32:06.399
don't need to see that eighteen times, but you know,
1776
01:32:06.640 --> 01:32:08.600
we'll get kind of the major pieces in and then
1777
01:32:08.680 --> 01:32:12.960
anything bonus extra that I do the
app I'm gonna build.
1778
01:32:13.359 --> 01:32:16.880
I decided I need to learn next JS. So it's
1779
01:32:16.920 --> 01:32:18.520
going to be an NEXTJS app and I'm going to
1780
01:32:18.600 --> 01:32:21.079
be putting it on cloud Flare workers.
1781
01:32:22.399 --> 01:32:23.199
And the reason is.
1782
01:32:23.159 --> 01:32:25.880
Is because just to give you an idea of what
1783
01:32:25.920 --> 01:32:27.800
the app is, it's relatively simple.
1784
01:32:27.800 --> 01:32:31.359
But last year when we.
1785
01:32:31.560 --> 01:32:36.279
Ran Caucus Night for the Utah Republican Party, we had
1786
01:32:36.319 --> 01:32:39.840
an online registration and we got d doast because there
1787
01:32:39.880 --> 01:32:45.840
were people out there who didn't like us, and it's
1788
01:32:45.920 --> 01:32:48.600
internal politics to Utah. It wasn't the Democrats, it with
1789
01:32:48.640 --> 01:32:55.000
somebody else, but anyway, So because of that, I'm looking to,
1790
01:32:55.159 --> 01:32:57.319
you know, put it on a system where I know
1791
01:32:57.399 --> 01:33:00.279
it'll just kind of expand to whatever comes at it.
1792
01:33:01.119 --> 01:33:03.760
Cloud Flare is also usually pretty good at you hit
1793
01:33:03.800 --> 01:33:07.039
me eighteen times. Now, I'm just going to say drop
1794
01:33:07.079 --> 01:33:09.279
it and drop you unless you can prove your human
1795
01:33:09.920 --> 01:33:11.439
and so I feel like I can get some of
1796
01:33:11.439 --> 01:33:14.239
those benefits. But I'm also curious to see how cloud
1797
01:33:14.239 --> 01:33:17.600
flair workers work. So it's going to be basically a registration.
1798
01:33:18.760 --> 01:33:20.680
There's going to be a little bit of site automation
1799
01:33:20.800 --> 01:33:25.960
because the Utah State voter registration database where you verify
1800
01:33:26.000 --> 01:33:29.640
that your voter registration doesn't have an API, which means
1801
01:33:29.640 --> 01:33:31.720
that I have to go and have my program use
1802
01:33:31.800 --> 01:33:35.800
something like puppeteer to fill in the fields and then
1803
01:33:35.880 --> 01:33:38.520
scrape data off the response to make sure that you're
1804
01:33:38.520 --> 01:33:42.439
registered to go to caucus night. I'm thinking I may
1805
01:33:42.520 --> 01:33:44.920
also offer this same kind of thing to the Democrats
1806
01:33:45.600 --> 01:33:47.560
and anybody else who wants to run.
1807
01:33:48.439 --> 01:33:49.800
A caucus night that night.
1808
01:33:49.800 --> 01:33:53.039
I think Libertarians and Utah do it too, right, so
1809
01:33:53.079 --> 01:33:56.840
that they can just hey, you've got online registration and
1810
01:33:56.880 --> 01:33:59.560
then you've got an app that will verify it on
1811
01:33:59.560 --> 01:34:00.000
the other end.
1812
01:34:00.600 --> 01:34:02.800
So anyway, that's what I'm looking at.
1813
01:34:02.880 --> 01:34:05.840
So there may also be React native app or something
1814
01:34:05.880 --> 01:34:08.960
like that that on the other end, you know, people
1815
01:34:09.000 --> 01:34:12.079
can show up with a QR code that says I
1816
01:34:12.159 --> 01:34:13.960
registered and this is who I am, and people can
1817
01:34:14.000 --> 01:34:15.800
just verify that way instead of having to look them
1818
01:34:15.880 --> 01:34:17.439
up in a paper list or something like that.
1819
01:34:17.840 --> 01:34:21.359
So that's what to be building. You jass geniuses.
1820
01:34:21.359 --> 01:34:23.119
You get access to the videos, you get access to
1821
01:34:23.199 --> 01:34:27.680
the weekly meetups, and a bunch of other stuff. I'm
1822
01:34:27.720 --> 01:34:30.359
also looking at starting a new podcast on doing AI
1823
01:34:30.439 --> 01:34:33.960
with JavaScript, and it's gonna be at this level, right
1824
01:34:34.039 --> 01:34:36.840
We're not building our own models. We're gonna be using
1825
01:34:36.920 --> 01:34:39.800
the existing models that are out there, the open source models,
1826
01:34:39.840 --> 01:34:43.800
if you will, and showing how to build things on
1827
01:34:43.840 --> 01:34:46.840
top of those, or using some of the cloud services
1828
01:34:46.840 --> 01:34:49.960
that generate images, or you know, using something like Whisper
1829
01:34:50.000 --> 01:34:51.319
for transcriptions and things like that.
1830
01:34:51.439 --> 01:34:54.720
So anyway, keep an eye out for that. That'll be free.
1831
01:34:54.840 --> 01:34:57.079
I'll probably drop the first two or three weeks worth
1832
01:34:57.119 --> 01:35:01.640
of episodes onto this RSSV and then From there.
1833
01:35:01.439 --> 01:35:04.039
You'll be able to just subscribe to the other feed.
1834
01:35:04.520 --> 01:35:09.119
So that's what I'm I've got going. Yeah, those are
1835
01:35:09.119 --> 01:35:10.880
my picks, e Shan, what are your picks?
1836
01:35:11.479 --> 01:35:15.239
So I've got two picks. The first one both are
1837
01:35:15.239 --> 01:35:17.560
going to be a I related. The first one is
1838
01:35:18.199 --> 01:35:21.680
notebook LM, but everyone knows about notebook LM with like
1839
01:35:21.720 --> 01:35:25.159
the fake podcasters. My pick is not book LM without
1840
01:35:25.199 --> 01:35:27.399
that feature. I think that feature is great and really compelling.
1841
01:35:28.039 --> 01:35:30.920
Lets me consume you know, material on the go in
1842
01:35:31.000 --> 01:35:35.079
podcast form. But I like the other parts of noepook LM,
1843
01:35:35.119 --> 01:35:37.239
which is like it's a great way to stick a
1844
01:35:37.319 --> 01:35:40.520
variety of sources together and then ask questions about it.
1845
01:35:40.960 --> 01:35:43.560
So one example is I like to go to y Combinator,
1846
01:35:43.640 --> 01:35:45.560
hacker news to see what the comments are, but I
1847
01:35:45.600 --> 01:35:47.880
don't read through every single one. So I will stick
1848
01:35:47.880 --> 01:35:49.319
it in there and say, well, what are the most
1849
01:35:49.319
--> 01:35:51.800
insightful comments? What are people saying? I did this actually
1850
01:35:51.800 --> 01:35:53.760
with deep Seek, I say what are the comments people
1851
01:35:53.760 --> 01:35:55.680
are saying about deep Seek? What are they seeing for performance?
1852
01:35:55.720 --> 01:35:57.960
What are the issues where it's not working? And what's
1853
01:35:57.960 --> 01:36:00.399
great is it doesn't just summarize it. You've got where
1854
01:36:00.399 --> 01:36:02.680
it can go to each part of it and say, okay,
1855
01:36:02.680 --> 01:36:04.760
this is like, oh that sounds interesting, let me go
1856
01:36:04.800 --> 01:36:06.359
click on it, and I can go right to the
1857
01:36:06.399 --> 01:36:09.439
citation of that that comment. The formatting is a little
1858
01:36:09.439 --> 01:36:12.039
off when you when you stick it in there, so
1859
01:36:12.079 --> 01:36:14.800
there's and you're only limited to thirty sources in each notebook,
1860
01:36:15.119 --> 01:36:17.000
but check out the other parts of notebook.
1861
01:36:17.079 --> 01:36:17.359
LM.
1862
01:36:17.399 --> 01:36:19.439
I think it's it's really interesting. I expect to see
1863
01:36:19.439 --> 01:36:22.840
a lot of other applications follow the similar type of
1864
01:36:22.840 --> 01:36:27.560
of ux paradigm or inspiration. The second one is I
1865
01:36:27.560 --> 01:36:29.560
don't know if you guys have been watching it, but
1866
01:36:29.840 --> 01:36:34.199
Star Wars has a new show, Skeleton Crew that they
1867
01:36:34.239 --> 01:36:38.079
have on Disney Plus. And first of all, I think
1868
01:36:38.079 --> 01:36:40.880
it's it's good. I don't think it's you know, Mandalorian
1869
01:36:41.159 --> 01:36:44.000
or and Or, which was my personal favorite level, but
1870
01:36:44.039 --> 01:36:46.960
it's still pretty good. But the other reason I bring
1871
01:36:46.960 --> 01:36:49.479
it up is I liked some of the elements of
1872
01:36:49.520 --> 01:36:53.239
how they handled AI and droids. So in one episode
1873
01:36:54.039 --> 01:36:56.439
there's something that could be akin to jail breaking the
1874
01:36:56.479 --> 01:36:58.960
droid where somebody uses the equivalent of prompt you know,
1875
01:36:59.159 --> 01:37:02.039
prompt hacking to jail break a droid, and I don't
1876
01:37:02.039 --> 01:37:04.800
think we've ever seen that in Star Wars's reflection of
1877
01:37:05.159 --> 01:37:08.640
Droids before. There's another one where it reminded me of
1878
01:37:08.680 --> 01:37:11.520
this paper called alignment faking, where the model has to
1879
01:37:11.560 --> 01:37:14.600
decide between its original training or the thing it's being
1880
01:37:14.640 --> 01:37:16.600
asked to do right now, and it kind of goes
1881
01:37:16.640 --> 01:37:18.239
back and forth and it gets over in by its
1882
01:37:18.279 --> 01:37:21.680
original training. And so there's one thing the very last
1883
01:37:21.720 --> 01:37:23.800
episode that I also thought it was fascinating. But I
1884
01:37:23.840 --> 01:37:26.159
really liked those interesting bits of how they handled AI
1885
01:37:26.279 --> 01:37:29.079
that I think we wouldn't have seen in a show
1886
01:37:29.159 --> 01:37:32.640
like this without understanding of chat GPT that I think
1887
01:37:32.680 --> 01:37:35.600
probably the writers were inspired by. So those are my picks.
1888
01:37:35.800 --> 01:37:38.039
Awesome, Yeah, Skeleton Crews on my list of things I
1889
01:37:38.039 --> 01:37:40.680
want to watch. So I like the recommendation. Thanks for that,
1890
01:37:41.279 --> 01:37:44.439
all right, Well, just a reminder go look on maven
1891
01:37:44.479 --> 01:37:45.079
dot com.
1892
01:37:45.800 --> 01:37:46.800
The code was jays.
1893
01:37:46.640 --> 01:37:50.680
Jabber for twenty percent off, and so if you're interested
1894
01:37:50.680 --> 01:37:53.560
in the course, go check it out. I'm not hard
1895
01:37:53.600 --> 01:37:57.439
sell guy. I just think it sounds fascinating. So anyway,
1896
01:37:58.039 --> 01:37:59.960
let's go ahead and wrap it up here until next time.
1897
01:38:00.079 --> 01:38:01.279
Him Max Ow
1898
01:38:04.920 --> 01:38:05.199
Mhm