1
00:00:00,680 --> 00:00:02,560
Tom Mitchell:
Welcome to machine learning.
2
00:00:02,560 --> 00:00:03,919
Tom Mitchell:
How did we get here?
3
00:00:03,919 --> 00:00:05,799
Tom Mitchell:
I'm Tom Mitchell.
4
00:00:05,799 --> 00:00:08,339
Tom Mitchell:
Today's episode is an interview
5
00:00:08,339 --> 00:00:10,320
Tom Mitchell:
with Geoff Hinton, one of the
6
00:00:10,320 --> 00:00:13,519
Tom Mitchell:
pioneers in the field of neural
7
00:00:13,519 --> 00:00:14,919
Tom Mitchell:
network learning.
8
00:00:14,919 --> 00:00:18,660
Tom Mitchell:
Geoff started out early, as
you'll hear, in the seventies,
9
00:00:18,660 --> 00:00:22,199
Tom Mitchell:
nineteen seventies, and has
continued working in neural
10
00:00:22,199 --> 00:00:24,519
Tom Mitchell:
networks ever since.
11
00:00:24,519 --> 00:00:26,000
Tom Mitchell:
During the period of the
12
00:00:26,000 --> 00:00:28,280
Tom Mitchell:
nineteen nineties and early two
13
00:00:28,280 --> 00:00:30,480
Tom Mitchell:
thousand, when neural networks
14
00:00:30,480 --> 00:00:33,039
Tom Mitchell:
were really in disfavor in the
15
00:00:33,039 --> 00:00:35,119
Tom Mitchell:
field of machine learning, Geoff
16
00:00:35,119 --> 00:00:38,460
Tom Mitchell:
nevertheless persisted, and he
17
00:00:38,460 --> 00:00:41,600
Tom Mitchell:
co-led the triumphant return of
18
00:00:41,600 --> 00:00:43,159
Tom Mitchell:
neural networks in the form of
19
00:00:43,159 --> 00:00:46,500
Tom Mitchell:
deep networks in the twenty ten
20
00:00:46,500 --> 00:00:48,600
Tom Mitchell:
ish period.
21
00:00:48,600 --> 00:00:53,840
Tom Mitchell:
In twenty eighteen, Geoff, along
with Joshua Bengio and Yann
22
00:00:53,840 --> 00:00:58,600
Tom Mitchell:
LeCun, received the Turing Award
in Computer Science.
23
00:00:58,600 --> 00:01:00,899
Tom Mitchell:
That's the highest award given
24
00:01:00,899 --> 00:01:03,259
Tom Mitchell:
in the field of computer science
25
00:01:03,259 --> 00:01:06,359
Tom Mitchell:
to researchers in twenty twenty
26
00:01:06,359 --> 00:01:07,180
Tom Mitchell:
four.
27
00:01:07,180 --> 00:01:09,719
Tom Mitchell:
Jeff, along with John Hopcroft,
28
00:01:09,719 --> 00:01:12,439
Tom Mitchell:
were awarded the twenty twenty
29
00:01:12,439 --> 00:01:15,859
Tom Mitchell:
four Nobel Prize in Physics for
30
00:01:15,859 --> 00:01:18,540
Tom Mitchell:
their work on artificial neural
31
00:01:18,540 --> 00:01:20,019
Tom Mitchell:
networks.
32
00:01:20,019 --> 00:01:22,019
Tom Mitchell:
I hope you enjoy the episode.
33
00:01:37,420 --> 00:01:41,140
Tom Mitchell:
I'm pleased to have with me
today Geoff Hinton, one of the
34
00:01:41,140 --> 00:01:43,659
Tom Mitchell:
pioneers of machine learning.
35
00:01:43,659 --> 00:01:45,540
Tom Mitchell:
Geoff, great to see you again.
36
00:01:45,700 --> 00:01:47,019
Geoffrey Hinton:
Thanks for inviting me.
37
00:01:47,379 --> 00:01:52,579
Tom Mitchell:
What I'd like to do today is get
two things.
38
00:01:52,579 --> 00:01:54,659
Tom Mitchell:
Two types of things from you.
39
00:01:54,659 --> 00:01:59,400
Tom Mitchell:
One is your own personal history
and how you got into this field
40
00:01:59,400 --> 00:02:02,159
Tom Mitchell:
and what happened after you did.
41
00:02:02,159 --> 00:02:06,640
Tom Mitchell:
And the second is kind of your
perspective on the whole field
42
00:02:06,640 --> 00:02:11,280
Tom Mitchell:
of machine learning, AI, and how
things are turning out.
43
00:02:11,120 --> 00:02:13,599
Geoffrey Hinton:
So when I was in high school, I
had a very smart friend who was
44
00:02:13,599 --> 00:02:18,080
Geoffrey Hinton:
a very good mathematician and
read widely, unlike me.
45
00:02:18,080 --> 00:02:20,520
Geoffrey Hinton:
And he came into school one day
46
00:02:20,520 --> 00:02:22,039
Geoffrey Hinton:
and talked about how memories
47
00:02:22,039 --> 00:02:23,400
Geoffrey Hinton:
might be distributed over the
48
00:02:23,400 --> 00:02:24,800
Geoffrey Hinton:
brain rather than localized in a
49
00:02:24,800 --> 00:02:26,840
Geoffrey Hinton:
place like a hologram, because
50
00:02:26,840 --> 00:02:27,759
Geoffrey Hinton:
this would have been nineteen
51
00:02:27,759 --> 00:02:29,240
Geoffrey Hinton:
sixty six and holograms had just
52
00:02:29,240 --> 00:02:30,199
Geoffrey Hinton:
come out.
53
00:02:30,199 --> 00:02:32,680
Geoffrey Hinton:
And that got me interested in
54
00:02:32,680 --> 00:02:34,000
Geoffrey Hinton:
how our memories represented in
55
00:02:34,000 --> 00:02:35,400
Geoffrey Hinton:
the brain.
56
00:02:35,400 --> 00:02:37,960
Geoffrey Hinton:
And I've been interested in that
ever since.
57
00:02:39,639 --> 00:02:43,680
Tom Mitchell:
Now, when I met you, we were
both at Carnegie Mellon.
58
00:02:43,680 --> 00:02:46,500
Tom Mitchell:
It was nineteen eighty six when,
59
00:02:46,500 --> 00:02:48,240
Tom Mitchell:
uh, we really got to do some
60
00:02:48,240 --> 00:02:49,919
Tom Mitchell:
work together or teach a course
61
00:02:49,919 --> 00:02:50,759
Tom Mitchell:
together.
62
00:02:50,759 --> 00:02:51,400
Tom Mitchell:
What?
63
00:02:51,400 --> 00:02:53,039
Tom Mitchell:
How did you get from nineteen
64
00:02:53,039 --> 00:02:55,120
Tom Mitchell:
sixty six up till Eighteen
65
00:02:55,120 --> 00:02:55,939
Tom Mitchell:
eighty six.
66
00:02:55,939 --> 00:02:58,219
Tom Mitchell:
What was the path?
67
00:02:58,780 --> 00:02:59,780
Geoffrey Hinton:
Slightly
68
00:02:59,780 --> 00:03:03,159
Geoffrey Hinton:
rocky. So I went to
69
00:03:03,159 --> 00:03:07,340
Geoffrey Hinton:
university. I studied physics,
chemistry and physiology, and in
70
00:03:07,340 --> 00:03:11,460
Geoffrey Hinton:
physiology. In the last term,
they're going to teach us, um,
how the
71
00:03:11,460 --> 00:03:12,419
Geoffrey Hinton:
central nervous system
72
00:03:12,419 --> 00:03:13,780
Geoffrey Hinton:
worked. And I was very
73
00:03:13,780 --> 00:03:17,900
Geoffrey Hinton:
excited. And they taught us how
action potentials are conducted
74
00:03:17,900 --> 00:03:20,740
Geoffrey Hinton:
along an axon, which wasn't what
I meant by how it
75
00:03:20,740 --> 00:03:23,300
Geoffrey Hinton:
worked. And so I switched to
76
00:03:23,300 --> 00:03:26,219
Geoffrey Hinton:
philosophy. That was even less
77
00:03:26,219 --> 00:03:28,060
Geoffrey Hinton:
useful. And then I switched
78
00:03:28,060 --> 00:03:29,819
Geoffrey Hinton:
to psychology, which was
79
00:03:29,819 --> 00:03:30,300
Geoffrey Hinton:
completely
80
00:03:30,300 --> 00:03:32,219
Geoffrey Hinton:
hopeless. Um, and then I became
a
81
00:03:32,219 --> 00:03:36,020
Geoffrey Hinton:
carpenter. And after I'd been a
82
00:03:36,020 --> 00:03:38,520
Geoffrey Hinton:
carpenter for about nine months,
I met
83
00:03:38,520 --> 00:03:38,840
Geoffrey Hinton:
a
84
00:03:38,840 --> 00:03:42,020
Geoffrey Hinton:
carpenter. And he was so much
better than me, I decided it'd
be easier
85
00:03:42,020 --> 00:03:42,379
Geoffrey Hinton:
to be an
86
00:03:42,379 --> 00:03:46,180
Geoffrey Hinton:
academic. Um, so I went to
graduate
87
00:03:46,180 --> 00:03:48,259
Geoffrey Hinton:
school in Edinburgh, um,
88
00:03:48,259 --> 00:03:49,520
Geoffrey Hinton:
with Longuet-higgins, who
89
00:03:49,520 --> 00:03:50,979
Geoffrey Hinton:
had published interesting stuff
90
00:03:50,979 --> 00:03:53,759
Geoffrey Hinton:
on, um, using neural nets for
91
00:03:53,759 --> 00:03:53,919
Geoffrey Hinton:
a
92
00:03:53,919 --> 00:03:59,360
Geoffrey Hinton:
memory. Unfortunately, around
the time I arrived, Winograd's
thesis
93
00:03:59,360 --> 00:04:02,759
Geoffrey Hinton:
came out and he switched his
allegiance to symbolic AI
94
00:04:02,759 --> 00:04:03,800
Geoffrey Hinton:
and gave up on neural
95
00:04:03,800 --> 00:04:07,280
Geoffrey Hinton:
nets. And so I spent five years
as
96
00:04:07,280 --> 00:04:08,960
Geoffrey Hinton:
his graduate student with
97
00:04:08,960 --> 00:04:10,159
Geoffrey Hinton:
him, trying to persuade me to
give
98
00:04:10,159 --> 00:04:10,639
Geoffrey Hinton:
up neural
99
00:04:10,639 --> 00:04:12,280
Geoffrey Hinton:
nets. And he never
100
00:04:12,280 --> 00:04:16,439
Geoffrey Hinton:
succeeded. Um, in the end, he
was very helpful to
101
00:04:16,439 --> 00:04:18,980
Geoffrey Hinton:
me. But for a long time, there
was a lot of argument about how
102
00:04:18,980 --> 00:04:21,279
Geoffrey Hinton:
I should really be doing
symbolic AI, and all this neural
103
00:04:21,279 --> 00:04:22,079
Geoffrey Hinton:
net stuff was complete
104
00:04:22,079 --> 00:04:24,319
Geoffrey Hinton:
nonsense. And everybody else in
105
00:04:24,319 --> 00:04:26,019
Geoffrey Hinton:
Edinburgh believed that neural
nets
106
00:04:26,019 --> 00:04:26,199
Geoffrey Hinton:
were
107
00:04:26,199 --> 00:04:29,600
Geoffrey Hinton:
nonsense. Um, we actually a
couple of
108
00:04:29,600 --> 00:04:32,480
Geoffrey Hinton:
exceptions. There was a post-doc
called David Willshaw who'd
109
00:04:32,480 --> 00:04:35,199
Geoffrey Hinton:
done associative memory, and he
basically done something
110
00:04:35,199 --> 00:04:36,000
Geoffrey Hinton:
quite like Hopfield
111
00:04:36,000 --> 00:04:37,459
Geoffrey Hinton:
nets. But a long time before
112
00:04:37,459 --> 00:04:41,360
Geoffrey Hinton:
Hopfield and Aaron Sloman was a
113
00:04:41,360 --> 00:04:42,680
Geoffrey Hinton:
visitor for a while, and he was
114
00:04:42,680 --> 00:04:42,879
Geoffrey Hinton:
more
115
00:04:42,879 --> 00:04:47,360
Geoffrey Hinton:
sympathetic. Um, but basically
they all knew it was
116
00:04:47,360 --> 00:04:50,639
Geoffrey Hinton:
rubbish. And they would explain
to me
117
00:04:50,639 --> 00:04:52,470
Geoffrey Hinton:
how neural nets can't even
118
00:04:52,470 --> 00:04:52,610
Geoffrey Hinton:
do
119
00:04:52,610 --> 00:04:56,089
Geoffrey Hinton:
recursion. So because
everything,
120
00:04:56,089 --> 00:04:58,189
Geoffrey Hinton:
everybody believed in recursion,
then,
121
00:04:58,189 --> 00:05:00,930
Geoffrey Hinton:
um, I actually figured out how
to
122
00:05:00,930 --> 00:05:02,009
Geoffrey Hinton:
do true recursion in a
123
00:05:02,009 --> 00:05:04,889
Geoffrey Hinton:
neural network and implemented
it on
124
00:05:04,889 --> 00:05:05,370
Geoffrey Hinton:
a machine
125
00:05:05,370 --> 00:05:07,910
Geoffrey Hinton:
with. I think by then it had one
hundred and ninety two
126
00:05:07,910 --> 00:05:11,050
Geoffrey Hinton:
kilobytes of memory, and it was
only shared by forty
127
00:05:11,050 --> 00:05:14,410
Geoffrey Hinton:
people. Um, but it had a huge
disk that had two
128
00:05:14,410 --> 00:05:15,970
Geoffrey Hinton:
megabytes. So you never ran out
of your
129
00:05:15,970 --> 00:05:18,250
Geoffrey Hinton:
memory? Um, because you used
virtual
130
00:05:18,250 --> 00:05:20,449
Geoffrey Hinton:
memory. And I actually
implemented
131
00:05:20,449 --> 00:05:23,389
Geoffrey Hinton:
a little neural net that did
132
00:05:23,389 --> 00:05:23,569
Geoffrey Hinton:
true
133
00:05:23,569 --> 00:05:29,569
Geoffrey Hinton:
recursion. That is, in the
recursive call, it used the same
neurons and
134
00:05:29,569 --> 00:05:32,769
Geoffrey Hinton:
the same connection strings for
the recursive call as it did for
135
00:05:32,769 --> 00:05:33,290
Geoffrey Hinton:
the high level
136
00:05:33,290 --> 00:05:38,470
Geoffrey Hinton:
call. Now, if you do that, of
course it had to offload all
137
00:05:38,470 --> 00:05:41,889
Geoffrey Hinton:
the parameters of the high level
call into some short term
138
00:05:41,889 --> 00:05:43,569
Geoffrey Hinton:
memory onto a
139
00:05:43,569 --> 00:05:45,170
Geoffrey Hinton:
stack. Eventually, and I figured
140
00:05:45,170 --> 00:05:46,410
Geoffrey Hinton:
out how to implement a stack
141
00:05:46,410 --> 00:05:48,050
Geoffrey Hinton:
with associative memory in a
142
00:05:48,050 --> 00:05:48,410
Geoffrey Hinton:
neural
143
00:05:48,410 --> 00:05:52,350
Geoffrey Hinton:
net. Um, so I had this little
neural net running that was
doing
144
00:05:52,350 --> 00:05:56,189
Geoffrey Hinton:
full recursion in neural nets,
and that was the first talk I
145
00:05:56,189 --> 00:05:58,029
Geoffrey Hinton:
gave. And people were very
146
00:05:58,029 --> 00:06:01,670
Geoffrey Hinton:
puzzled. They said, why would
you want to do recursion in a
neural
147
00:06:01,670 --> 00:06:05,490
Geoffrey Hinton:
net? I mean, it's so easy to do
in pop two, which was our our
148
00:06:05,490 --> 00:06:11,910
Geoffrey Hinton:
sort of, uh, unfortunate bastard
child of Pascal and
149
00:06:11,910 --> 00:06:15,589
Geoffrey Hinton:
Lisp. Um, although I don't think
Pascal existed
150
00:06:15,589 --> 00:06:17,009
Geoffrey Hinton:
then. Um,
151
00:06:17,009 --> 00:06:18,990
Geoffrey Hinton:
so.
152
00:06:18,990 --> 00:06:22,209
Geoffrey Hinton:
Yeah. So I keep meaning to go
back to
153
00:06:22,629 --> 00:06:24,629
Tom Mitchell:
I was going to ask, is there a
154
00:06:24,629 --> 00:06:26,509
Tom Mitchell:
future to recursion for neural
155
00:06:26,509 --> 00:06:27,189
Tom Mitchell:
nets?
156
00:06:27,829 --> 00:06:28,670
Geoffrey Hinton:
Oh, yes.
157
00:06:28,670 --> 00:06:31,189
Geoffrey Hinton:
I mean, to do true recursion,
158
00:06:31,189 --> 00:06:32,389
Geoffrey Hinton:
you have to use the same neurons
159
00:06:32,389 --> 00:06:33,490
Geoffrey Hinton:
and weights for the recursive
160
00:06:33,490 --> 00:06:34,069
Geoffrey Hinton:
call.
161
00:06:34,069 --> 00:06:36,389
Geoffrey Hinton:
That means you have to have a
stack, something like a stack,
162
00:06:36,389 --> 00:06:38,949
Geoffrey Hinton:
to store the parameters of the
high level call.
163
00:06:38,949 --> 00:06:40,990
Geoffrey Hinton:
That all works if you have fast
weights.
164
00:06:40,990 --> 00:06:42,670
Geoffrey Hinton:
So that was the first thing I
did with fast weights in
165
00:06:42,670 --> 00:06:44,470
Geoffrey Hinton:
nineteen seventy three.
166
00:06:44,470 --> 00:06:47,029
Geoffrey Hinton:
I should say fast weights were
invented by Schmidhuber in
167
00:06:47,029 --> 00:06:48,269
Geoffrey Hinton:
nineteen ninety, something.
168
00:06:50,449 --> 00:06:51,370
Tom Mitchell:
Fair enough.
169
00:06:51,370 --> 00:06:56,569
Tom Mitchell:
Okay, so then, um, you moved on
from Edinburgh.
170
00:06:56,569 --> 00:06:58,470
Tom Mitchell:
Did you come directly to
171
00:06:58,470 --> 00:07:00,269
Tom Mitchell:
Carnegie Mellon from there, or
172
00:07:00,269 --> 00:07:01,009
Tom Mitchell:
how did.
173
00:07:00,850 --> 00:07:05,170
Geoffrey Hinton:
Oh, no. No. Um, I dropped out
174
00:07:05,170 --> 00:07:07,569
Geoffrey Hinton:
again after I finished my
175
00:07:07,569 --> 00:07:08,610
Geoffrey Hinton:
thesis.
176
00:07:08,610 --> 00:07:10,250
Geoffrey Hinton:
I dropped out and became a
177
00:07:10,250 --> 00:07:11,529
Geoffrey Hinton:
teacher in a free school in
178
00:07:11,529 --> 00:07:12,569
Geoffrey Hinton:
London.
179
00:07:12,569 --> 00:07:15,529
Geoffrey Hinton:
Um, it was voluntary, I was
unpaid.
180
00:07:15,529 --> 00:07:19,410
Geoffrey Hinton:
They were rough, emotionally
disturbed inner city kids.
181
00:07:19,410 --> 00:07:20,529
Geoffrey Hinton:
And after a few months of that,
182
00:07:20,529 --> 00:07:22,009
Geoffrey Hinton:
I again decided academia might
183
00:07:22,009 --> 00:07:23,129
Geoffrey Hinton:
be easier.
184
00:07:23,129 --> 00:07:28,410
Geoffrey Hinton:
Um, so I went back to a post-doc
with Aaron Sloman in Sussex.
185
00:07:28,410 --> 00:07:33,610
Geoffrey Hinton:
Higgins had moved from Edinburgh
to Sussex and um, as I was
186
00:07:33,610 --> 00:07:36,810
Geoffrey Hinton:
finishing my PhD, I got a
post-doc with Aaron Sloman.
187
00:07:36,810 --> 00:07:40,850
Geoffrey Hinton:
Um, and there were no proper
faculty jobs in Britain.
188
00:07:40,850 --> 00:07:42,889
Geoffrey Hinton:
Then there was one job in the
189
00:07:42,889 --> 00:07:43,970
Geoffrey Hinton:
whole of Britain which Alan
190
00:07:43,970 --> 00:07:49,310
Geoffrey Hinton:
Bundy got, um, and so I applied
191
00:07:49,310 --> 00:07:52,889
Geoffrey Hinton:
for jobs in the States, and I
192
00:07:52,889 --> 00:07:56,329
Geoffrey Hinton:
got a job as a postdoc in UCSD,
193
00:07:56,329 --> 00:07:57,790
Geoffrey Hinton:
um, with Don Norman and Dave
194
00:07:57,790 --> 00:07:59,149
Geoffrey Hinton:
Rumelhart.
195
00:07:59,149 --> 00:08:02,230
Geoffrey Hinton:
And I really got along very well
with Dave Rumelhart, and that
196
00:08:02,230 --> 00:08:03,550
Geoffrey Hinton:
made a huge difference.
197
00:08:03,550 --> 00:08:08,029
Geoffrey Hinton:
So I moved from a country where
it was a sort of small country,
198
00:08:08,029 --> 00:08:13,430
Geoffrey Hinton:
Britain, and there was only room
for one ideology, and the
199
00:08:13,430 --> 00:08:17,589
Geoffrey Hinton:
ideology was symbolic AI and
neural nets was just rubbish.
200
00:08:18,709 --> 00:08:22,189
Geoffrey Hinton:
And I moved to the states where,
um, on the West coast, on the
201
00:08:22,189 --> 00:08:24,930
Geoffrey Hinton:
East Coast, it was symbolic AI
but on the West Coast they were
202
00:08:24,930 --> 00:08:26,230
Geoffrey Hinton:
kind of more open.
203
00:08:26,230 --> 00:08:27,310
Geoffrey Hinton:
And in particular, Don Norman
204
00:08:27,310 --> 00:08:28,529
Geoffrey Hinton:
and Dave Rumelhart thought
205
00:08:28,529 --> 00:08:29,550
Geoffrey Hinton:
neural nets were worth
206
00:08:29,550 --> 00:08:30,790
Geoffrey Hinton:
considering.
207
00:08:30,790 --> 00:08:33,909
Geoffrey Hinton:
Um, so it was a huge liberation
208
00:08:33,909 --> 00:08:36,309
Geoffrey Hinton:
to be in a place where neural
209
00:08:36,309 --> 00:08:37,570
Geoffrey Hinton:
nets were regarded as not
210
00:08:37,570 --> 00:08:38,870
Geoffrey Hinton:
obvious nonsense.
211
00:08:40,350 --> 00:08:41,110
Geoffrey Hinton:
And.
212
00:08:43,149 --> 00:08:44,850
Geoffrey Hinton:
Um, I went there.
213
00:08:44,850 --> 00:08:48,769
Geoffrey Hinton:
I, I got to meet Terry Sinofsky,
who I invited to a conference.
214
00:08:48,769 --> 00:08:51,490
Geoffrey Hinton:
And we'd been sort of lifelong
friends and collaborators.
215
00:08:51,490 --> 00:08:55,409
Geoffrey Hinton:
Um, I got to meet Francis Crick
later on, who was there.
216
00:08:55,409 --> 00:08:56,610
Geoffrey Hinton:
So I was there for a couple of
years.
217
00:08:56,610 --> 00:08:57,529
Geoffrey Hinton:
And then I got a job in
218
00:08:57,529 --> 00:08:59,250
Geoffrey Hinton:
Cambridge in the Applied
219
00:08:59,250 --> 00:09:02,110
Geoffrey Hinton:
Psychology Research Unit, um,
220
00:09:02,110 --> 00:09:03,129
Geoffrey Hinton:
where I was meant to do applied
221
00:09:03,129 --> 00:09:04,769
Geoffrey Hinton:
psychology.
222
00:09:04,769 --> 00:09:08,370
Geoffrey Hinton:
And I was strongly reminded of
William James's comment about
223
00:09:08,370 --> 00:09:12,409
Geoffrey Hinton:
applied psychology, which is to
do applied psychology, you have
224
00:09:12,409 --> 00:09:14,210
Geoffrey Hinton:
to have something to apply.
225
00:09:14,210 --> 00:09:19,009
Geoffrey Hinton:
Um. But I actually did some
interesting stuff.
226
00:09:19,009 --> 00:09:21,049
Geoffrey Hinton:
It was just around the time some
227
00:09:21,049 --> 00:09:23,470
Geoffrey Hinton:
workstations were coming out,
228
00:09:23,470 --> 00:09:26,549
Geoffrey Hinton:
and they had a contract with the
229
00:09:26,549 --> 00:09:29,370
Geoffrey Hinton:
British Telephone Company to
230
00:09:29,370 --> 00:09:31,330
Geoffrey Hinton:
help with network management and
231
00:09:31,330 --> 00:09:32,330
Geoffrey Hinton:
network management.
232
00:09:32,330 --> 00:09:34,490
Geoffrey Hinton:
Then was all done by hand.
233
00:09:35,490 --> 00:09:38,190
Geoffrey Hinton:
And you had information about,
234
00:09:38,190 --> 00:09:39,850
Geoffrey Hinton:
um, the loads on various
235
00:09:39,850 --> 00:09:41,929
Geoffrey Hinton:
switching centers, and the
236
00:09:41,929 --> 00:09:44,230
Geoffrey Hinton:
information was on a huge wall
237
00:09:44,230 --> 00:09:46,990
Geoffrey Hinton:
that was twenty feet high and
238
00:09:46,990 --> 00:09:48,470
Geoffrey Hinton:
worked like you sometimes see at
239
00:09:48,470 --> 00:09:49,669
Geoffrey Hinton:
train stations.
240
00:09:49,669 --> 00:09:52,549
Geoffrey Hinton:
It was little flaps with white
letters on that come down.
241
00:09:52,549 --> 00:09:54,990
Geoffrey Hinton:
They sort of rotate around until
you get the right flap.
242
00:09:54,990 --> 00:09:58,730
Geoffrey Hinton:
And so you could see all these
numbers, um, that said how busy
243
00:09:58,730 --> 00:10:00,909
Geoffrey Hinton:
each switching station was.
244
00:10:00,909 --> 00:10:04,350
Geoffrey Hinton:
Um, and I figured a sun
245
00:10:04,350 --> 00:10:05,590
Geoffrey Hinton:
workstation could do that and
246
00:10:05,590 --> 00:10:08,309
Geoffrey Hinton:
would be a lot cheaper, and the
247
00:10:08,309 --> 00:10:09,590
Geoffrey Hinton:
resolution wasn't that good
248
00:10:09,590 --> 00:10:10,389
Geoffrey Hinton:
then.
249
00:10:10,389 --> 00:10:13,409
Geoffrey Hinton:
So I had to figure out if you
could display the states of all
250
00:10:13,409 --> 00:10:16,110
Geoffrey Hinton:
the switching stations in
Britain on the screen of a sun
251
00:10:16,110 --> 00:10:20,309
Geoffrey Hinton:
workstation, and you couldn't
type the names, but if you had
252
00:10:20,309 --> 00:10:24,750
Geoffrey Hinton:
two letters for each name, you
could get two letters there.
253
00:10:24,750 --> 00:10:28,509
Geoffrey Hinton:
And I worked on a display with
254
00:10:28,509 --> 00:10:30,389
Geoffrey Hinton:
two letter names, and there were
255
00:10:30,389 --> 00:10:31,470
Geoffrey Hinton:
a large number of switching
256
00:10:31,470 --> 00:10:32,470
Geoffrey Hinton:
stations.
257
00:10:32,470 --> 00:10:33,710
Geoffrey Hinton:
There were hundreds.
258
00:10:33,710 --> 00:10:34,950
Geoffrey Hinton:
And the question was could could
259
00:10:34,950 --> 00:10:36,429
Geoffrey Hinton:
an operator remember which they
260
00:10:36,429 --> 00:10:37,190
Geoffrey Hinton:
were?
261
00:10:37,190 --> 00:10:38,490
Geoffrey Hinton:
So I actually taught myself to
262
00:10:38,490 --> 00:10:39,710
Geoffrey Hinton:
remember all those two letter
263
00:10:39,710 --> 00:10:41,210
Geoffrey Hinton:
names.
264
00:10:41,210 --> 00:10:45,169
Geoffrey Hinton:
Um, they were in very small type
to fit on, and I actually got a
265
00:10:45,169 --> 00:10:47,370
Geoffrey Hinton:
serious migraine from looking at
it too long.
266
00:10:47,370 --> 00:10:52,809
Geoffrey Hinton:
Um, that was my interaction with
human factors, and then I.
267
00:10:53,690 --> 00:10:58,169
Tom Mitchell:
It does remind me of that was
around the time that Unix was
268
00:10:58,169 --> 00:11:02,690
Tom Mitchell:
getting invented, and all the
commands had no vowels in them.
269
00:11:02,690 --> 00:11:04,289
Tom Mitchell:
So there was a theme there?
270
00:11:04,610 --> 00:11:08,730
Geoffrey Hinton:
Yes. Um, so I wrote a report on
271
00:11:08,730 --> 00:11:09,769
Geoffrey Hinton:
it and they said it was a very
272
00:11:09,769 --> 00:11:10,169
Geoffrey Hinton:
nice report.
273
00:11:10,169 --> 00:11:11,450
Geoffrey Hinton:
Thank you very much.
274
00:11:11,450 --> 00:11:12,169
Geoffrey Hinton:
And they weren't going to
275
00:11:12,169 --> 00:11:13,970
Geoffrey Hinton:
implement it, even though it
276
00:11:13,970 --> 00:11:14,649
Geoffrey Hinton:
would have been much more
277
00:11:14,649 --> 00:11:15,610
Geoffrey Hinton:
efficient and much easier to
278
00:11:15,610 --> 00:11:16,330
Geoffrey Hinton:
update.
279
00:11:16,330 --> 00:11:18,250
Geoffrey Hinton:
And I said, why not?
280
00:11:18,250 --> 00:11:21,490
Geoffrey Hinton:
And they confidentially
explained to me that, well, when
281
00:11:21,490 --> 00:11:26,090
Geoffrey Hinton:
people come and visit the
network control center, um, or
282
00:11:26,090 --> 00:11:28,090
Geoffrey Hinton:
when actually when they visit
the headquarters of British
283
00:11:28,090 --> 00:11:31,370
Geoffrey Hinton:
Telecom, um, they have to have
something to show them, like the
284
00:11:31,370 --> 00:11:34,210
Geoffrey Hinton:
politicians have to see
something, and they would always
285
00:11:34,210 --> 00:11:37,529
Geoffrey Hinton:
show them this huge wall that
displayed the state of all the
286
00:11:37,529 --> 00:11:40,629
Geoffrey Hinton:
networks, of all the switching
stations, and they were very
287
00:11:40,629 --> 00:11:41,750
Geoffrey Hinton:
impressed by that.
288
00:11:41,750 --> 00:11:43,029
Geoffrey Hinton:
And if they got rid of the huge
289
00:11:43,029 --> 00:11:45,230
Geoffrey Hinton:
wall and had just some
290
00:11:45,230 --> 00:11:46,789
Geoffrey Hinton:
workstations, they were very
291
00:11:46,789 --> 00:11:48,870
Geoffrey Hinton:
worried that network management
292
00:11:48,870 --> 00:11:49,830
Geoffrey Hinton:
would get less funds from
293
00:11:49,830 --> 00:11:50,909
Geoffrey Hinton:
British Telecom.
294
00:11:50,909 --> 00:11:53,710
Geoffrey Hinton:
So they were going to keep their
huge wall.
295
00:11:53,710 --> 00:11:56,070
Geoffrey Hinton:
I learned a lot then about
applied research.
296
00:11:57,629 --> 00:11:59,070
Geoffrey Hinton:
It's not about whether it works,
297
00:11:59,070 --> 00:12:00,350
Geoffrey Hinton:
it's whether about the company
298
00:12:00,350 --> 00:12:01,269
Geoffrey Hinton:
likes it.
299
00:12:03,429 --> 00:12:04,990
Tom Mitchell:
Fair enough, fair enough.
300
00:12:05,389 --> 00:12:11,870
Geoffrey Hinton:
Then after that, um, I went back
to sea to San Diego for six
301
00:12:11,870 --> 00:12:15,950
Geoffrey Hinton:
months, and that's when we
worked on the PDP books with
302
00:12:15,950 --> 00:12:18,350
Geoffrey Hinton:
Dave Rumelhart and McClelland.
303
00:12:18,350 --> 00:12:20,750
Geoffrey Hinton:
Um, I was one of the authors
304
00:12:20,750 --> 00:12:22,350
Geoffrey Hinton:
until almost when they were
305
00:12:22,350 --> 00:12:22,750
Geoffrey Hinton:
published.
306
00:12:22,750 --> 00:12:23,909
Geoffrey Hinton:
And the last minute I dropped
307
00:12:23,909 --> 00:12:25,309
Geoffrey Hinton:
out because at that point I
308
00:12:25,309 --> 00:12:26,629
Geoffrey Hinton:
decided Boltzmann machines were
309
00:12:26,629 --> 00:12:27,629
Geoffrey Hinton:
the future.
310
00:12:27,629 --> 00:12:30,370
Geoffrey Hinton:
Boltzmann machines was just a
much better idea than back prop,
311
00:12:30,370 --> 00:12:32,110
Geoffrey Hinton:
and back prop was a silly idea.
312
00:12:32,110 --> 00:12:34,070
Geoffrey Hinton:
Boltzmann machines were a much
better idea.
313
00:12:34,070 --> 00:12:36,470
Geoffrey Hinton:
Um, and there was no point being
an author of a book where the
314
00:12:36,470 --> 00:12:39,850
Geoffrey Hinton:
main thing was, um, back prop.
315
00:12:39,850 --> 00:12:42,210
Geoffrey Hinton:
Uh, that was a mistake.
316
00:12:42,210 --> 00:12:49,610
Geoffrey Hinton:
Um. But in nineteen eighty four,
I figured out.
317
00:12:49,610 --> 00:12:53,730
Geoffrey Hinton:
Yeah, in nineteen eighty two,
then I applied to CMU, and
318
00:12:53,730 --> 00:12:57,350
Geoffrey Hinton:
because it was a private
university, um, you could just
319
00:12:57,350 --> 00:12:59,690
Geoffrey Hinton:
they didn't have to sort of
advertise very widely.
320
00:12:59,690 --> 00:13:05,889
Geoffrey Hinton:
Um, and Scott Fahlman was sort
of my, um, host.
321
00:13:05,889 --> 00:13:07,809
Geoffrey Hinton:
He sort of interacted with him
322
00:13:07,809 --> 00:13:09,610
Geoffrey Hinton:
at many workshops, and we got
323
00:13:09,610 --> 00:13:11,330
Geoffrey Hinton:
along well, and he pushed hard
324
00:13:11,330 --> 00:13:12,629
Geoffrey Hinton:
to get for me to get them to go
325
00:13:12,629 --> 00:13:13,450
Geoffrey Hinton:
there.
326
00:13:13,450 --> 00:13:15,049
Geoffrey Hinton:
And I had a very funny
interview.
327
00:13:15,049 --> 00:13:16,610
Geoffrey Hinton:
So I went there.
328
00:13:16,610 --> 00:13:18,490
Geoffrey Hinton:
On the first day there, I gave a
329
00:13:18,490 --> 00:13:20,289
Geoffrey Hinton:
talk into computer science, and
330
00:13:20,289 --> 00:13:21,570
Geoffrey Hinton:
then Scott Farmer took me out
331
00:13:21,570 --> 00:13:24,649
Geoffrey Hinton:
for lunch at a place it might
332
00:13:24,649 --> 00:13:25,769
Geoffrey Hinton:
have been called the oh, I can't
333
00:13:25,769 --> 00:13:27,769
Geoffrey Hinton:
remember what it was called, but
334
00:13:27,769 --> 00:13:29,250
Geoffrey Hinton:
it had a motto, which is if you
335
00:13:29,250 --> 00:13:30,610
Geoffrey Hinton:
don't get sick, you got a bad
336
00:13:30,610 --> 00:13:31,450
Geoffrey Hinton:
one.
337
00:13:31,450 --> 00:13:35,049
Geoffrey Hinton:
Um, and I got terribly sick.
338
00:13:35,049 --> 00:13:36,870
Geoffrey Hinton:
And the next day I had acute
diarrhea.
339
00:13:36,870 --> 00:13:37,710
Geoffrey Hinton:
I couldn't eat anything.
340
00:13:37,710 --> 00:13:41,029
Geoffrey Hinton:
I was living on coffee and
Coca-Cola.
341
00:13:41,029 --> 00:13:48,230
Geoffrey Hinton:
Um, I gave a talk in psychology,
um, about mental imagery and my
342
00:13:48,230 --> 00:13:49,990
Geoffrey Hinton:
theory of mental imagery.
343
00:13:49,990 --> 00:13:54,350
Geoffrey Hinton:
And then there was someone at
the end who, um, asked a
344
00:13:54,350 --> 00:13:55,990
Geoffrey Hinton:
question which I didn't
understand to begin with.
345
00:13:55,990 --> 00:13:59,389
Geoffrey Hinton:
And then I realized the question
he was asking was, did I believe
346
00:13:59,389 --> 00:14:03,029
Geoffrey Hinton:
the theory of someone called
Marcel just about how you
347
00:14:03,029 --> 00:14:05,870
Geoffrey Hinton:
weren't really rotating an image
in your mind, you were just
348
00:14:05,870 --> 00:14:08,269
Geoffrey Hinton:
looking backwards and forwards
between two things.
349
00:14:08,269 --> 00:14:11,470
Geoffrey Hinton:
Um, and in my reply I said, oh,
350
00:14:11,470 --> 00:14:12,990
Geoffrey Hinton:
I see you mean that silly theory
351
00:14:12,990 --> 00:14:13,629
Geoffrey Hinton:
by Marcel.
352
00:14:13,629 --> 00:14:13,990
Geoffrey Hinton:
Marcel?
353
00:14:13,990 --> 00:14:16,830
Geoffrey Hinton:
Just where I didn't realize it
was Marcel.
354
00:14:16,830 --> 00:14:18,509
Geoffrey Hinton:
Just asking the question.
355
00:14:18,509 --> 00:14:22,269
Geoffrey Hinton:
Um, and after that, I got a
356
00:14:22,269 --> 00:14:23,409
Geoffrey Hinton:
request to go and see Nico
357
00:14:23,409 --> 00:14:24,990
Geoffrey Hinton:
Habermann.
358
00:14:24,990 --> 00:14:26,149
Geoffrey Hinton:
Now, Nico and I were always
359
00:14:26,149 --> 00:14:27,350
Geoffrey Hinton:
great friends, even though we
360
00:14:27,350 --> 00:14:28,750
Geoffrey Hinton:
were politically extremely
361
00:14:28,750 --> 00:14:29,990
Geoffrey Hinton:
different.
362
00:14:29,990 --> 00:14:33,674
Geoffrey Hinton:
I was a sort of leftie, nineteen
sixties radical with long hair
363
00:14:33,674 --> 00:14:35,399
Geoffrey Hinton:
and rather disheveled.
364
00:14:35,399 --> 00:14:39,539
Geoffrey Hinton:
Niko was a European gentleman
who was very nicely dressed,
365
00:14:39,539 --> 00:14:42,399
Geoffrey Hinton:
worked with the Defense
Department, set up an institute
366
00:14:42,399 --> 00:14:44,840
Geoffrey Hinton:
I wasn't allowed to go to
because I was a foreigner.
367
00:14:44,840 --> 00:14:48,080
Geoffrey Hinton:
Um, but we got along very well,
and I think it was because of
368
00:14:48,080 --> 00:14:49,480
Geoffrey Hinton:
our initial interview.
369
00:14:50,320 --> 00:14:54,720
Tom Mitchell:
So Niko was the department head
in computer science?
370
00:14:54,559 --> 00:14:56,639
Geoffrey Hinton:
Yes. And so in the initial
371
00:14:56,639 --> 00:14:58,639
Geoffrey Hinton:
interview, he said, so we've
372
00:14:58,639 --> 00:14:59,840
Geoffrey Hinton:
decided to offer you the
373
00:14:59,840 --> 00:15:00,759
Geoffrey Hinton:
position.
374
00:15:00,759 --> 00:15:03,639
Geoffrey Hinton:
And I said, oh, oh, there's
something you should know.
375
00:15:03,639 --> 00:15:05,240
Geoffrey Hinton:
And he said, oh what's that?
376
00:15:05,240 --> 00:15:06,379
Geoffrey Hinton:
And I said, well, I don't
377
00:15:06,379 --> 00:15:07,440
Geoffrey Hinton:
actually know any computer
378
00:15:07,440 --> 00:15:08,759
Geoffrey Hinton:
science.
379
00:15:08,759 --> 00:15:10,759
Geoffrey Hinton:
And he said, it's okay here.
380
00:15:10,759 --> 00:15:11,519
Geoffrey Hinton:
It's okay.
381
00:15:11,519 --> 00:15:13,559
Geoffrey Hinton:
We have people here who do.
382
00:15:13,559 --> 00:15:15,799
Geoffrey Hinton:
So I said, okay.
383
00:15:15,799 --> 00:15:17,039
Geoffrey Hinton:
So I said, okay.
384
00:15:17,039 --> 00:15:18,799
Geoffrey Hinton:
In that case I accept.
385
00:15:18,799 --> 00:15:20,679
Geoffrey Hinton:
And Niko said, don't you think
386
00:15:20,679 --> 00:15:22,080
Geoffrey Hinton:
perhaps we should talk about the
387
00:15:22,080 --> 00:15:23,559
Geoffrey Hinton:
salary?
388
00:15:23,559 --> 00:15:25,360
Geoffrey Hinton:
And I said, oh no, I'm not
interested in salary.
389
00:15:25,360 --> 00:15:26,360
Geoffrey Hinton:
You can pay me whatever you
like.
390
00:15:26,360 --> 00:15:28,440
Geoffrey Hinton:
I'm not doing it for the money.
391
00:15:28,440 --> 00:15:30,120
Geoffrey Hinton:
And he said, well, how does
392
00:15:30,120 --> 00:15:31,399
Geoffrey Hinton:
twenty six thousand sound to
393
00:15:31,399 --> 00:15:31,679
Geoffrey Hinton:
you?
394
00:15:31,679 --> 00:15:34,279
Geoffrey Hinton:
So if that sounds fine, um, I
395
00:15:34,279 --> 00:15:35,919
Geoffrey Hinton:
discovered I was being paid ten
396
00:15:35,919 --> 00:15:37,419
Geoffrey Hinton:
thousand than the next lowest
397
00:15:37,419 --> 00:15:38,659
Geoffrey Hinton:
paid professor.
398
00:15:38,659 --> 00:15:40,139
Geoffrey Hinton:
Ten thousand less.
399
00:15:40,139 --> 00:15:44,820
Geoffrey Hinton:
Um, but every year I got a big
pay rise, and me and Niko got
400
00:15:44,820 --> 00:15:46,820
Geoffrey Hinton:
along very well after that,
because he knew I wasn't doing
401
00:15:46,820 --> 00:15:47,620
Geoffrey Hinton:
it for the money.
402
00:15:49,980 --> 00:15:50,500
Geoffrey Hinton:
Change.
403
00:15:50,500 --> 00:15:52,340
Geoffrey Hinton:
Things have changed so much.
404
00:15:55,220 --> 00:15:56,940
Tom Mitchell:
That's. That's fantastic.
405
00:15:56,940 --> 00:15:59,480
Tom Mitchell:
Okay, so now we're up to the mid
406
00:15:59,480 --> 00:16:02,820
Tom Mitchell:
eighties when really neural nets
407
00:16:02,820 --> 00:16:04,259
Tom Mitchell:
are reborn.
408
00:16:04,259 --> 00:16:05,940
Tom Mitchell:
Is that the right word?
409
00:16:05,940 --> 00:16:06,539
Tom Mitchell:
How would you.
410
00:16:06,340 --> 00:16:08,899
Geoffrey Hinton:
Yes. We back we back
propagation.
411
00:16:08,899 --> 00:16:10,100
Geoffrey Hinton:
I mean, we didn't invent it.
412
00:16:10,100 --> 00:16:11,659
Geoffrey Hinton:
We invented by several different
413
00:16:11,659 --> 00:16:14,679
Geoffrey Hinton:
groups, but we showed that it
414
00:16:14,679 --> 00:16:15,980
Geoffrey Hinton:
really worked to learn
415
00:16:15,980 --> 00:16:17,259
Geoffrey Hinton:
representations.
416
00:16:17,259 --> 00:16:19,820
Geoffrey Hinton:
And as you know, sort of one of
the big problems in AI is how do
417
00:16:19,820 --> 00:16:22,100
Geoffrey Hinton:
you learn new representations?
418
00:16:22,100 --> 00:16:24,580
Geoffrey Hinton:
How do you avoid having to put
them all in by hand?
419
00:16:24,580 --> 00:16:27,940
Geoffrey Hinton:
Um, and my particular example,
420
00:16:27,940 --> 00:16:29,299
Geoffrey Hinton:
which was the family trees
421
00:16:29,299 --> 00:16:30,639
Geoffrey Hinton:
example, where you take all the
422
00:16:30,639 --> 00:16:31,840
Geoffrey Hinton:
information in some family
423
00:16:31,840 --> 00:16:33,879
Geoffrey Hinton:
trees, you convert it into
424
00:16:33,879 --> 00:16:36,200
Geoffrey Hinton:
triples of symbols like John has
425
00:16:36,200 --> 00:16:37,639
Geoffrey Hinton:
Father Mary.
426
00:16:37,639 --> 00:16:39,840
Geoffrey Hinton:
Um, and then you train a neural
427
00:16:39,840 --> 00:16:41,519
Geoffrey Hinton:
net to predict the last term in
428
00:16:41,519 --> 00:16:42,639
Geoffrey Hinton:
a triple given the first two
429
00:16:42,639 --> 00:16:43,639
Geoffrey Hinton:
terms.
430
00:16:43,639 --> 00:16:45,600
Geoffrey Hinton:
So it's just like the big
language models.
431
00:16:45,600 --> 00:16:48,200
Geoffrey Hinton:
You're predicting the next word
given the context.
432
00:16:48,200 --> 00:16:50,120
Geoffrey Hinton:
It's just much simpler.
433
00:16:50,120 --> 00:16:53,100
Geoffrey Hinton:
I had one hundred and twelve
434
00:16:53,100 --> 00:16:55,000
Geoffrey Hinton:
total examples, of which one
435
00:16:55,000 --> 00:16:56,440
Geoffrey Hinton:
hundred and four were training
436
00:16:56,440 --> 00:16:57,720
Geoffrey Hinton:
examples and eight were test
437
00:16:57,720 --> 00:16:59,759
Geoffrey Hinton:
examples, which is a bit less
438
00:16:59,759 --> 00:17:01,080
Geoffrey Hinton:
than the trillion examples they
439
00:17:01,080 --> 00:17:02,279
Geoffrey Hinton:
have nowadays.
440
00:17:02,279 --> 00:17:04,319
Geoffrey Hinton:
Um, but it was the same idea.
441
00:17:04,319 --> 00:17:08,200
Geoffrey Hinton:
You convert a symbol into a
feature vector.
442
00:17:08,200 --> 00:17:11,200
Geoffrey Hinton:
You then have the feature
vectors of the context interact
443
00:17:11,200 --> 00:17:14,000
Geoffrey Hinton:
um, via a hidden layer.
444
00:17:14,000 --> 00:17:15,640
Geoffrey Hinton:
They then predict the features
445
00:17:15,640 --> 00:17:17,839
Geoffrey Hinton:
of the next symbol, and from
446
00:17:17,839 --> 00:17:18,920
Geoffrey Hinton:
those features you guess what
447
00:17:18,920 --> 00:17:20,440
Geoffrey Hinton:
the next symbol should be, and
448
00:17:20,440 --> 00:17:21,400
Geoffrey Hinton:
you try and maximize the
449
00:17:21,400 --> 00:17:22,640
Geoffrey Hinton:
probability of predicting the
450
00:17:22,640 --> 00:17:23,839
Geoffrey Hinton:
next symbol.
451
00:17:23,839 --> 00:17:28,039
Geoffrey Hinton:
And you then backpropagate
through the feature interactions
452
00:17:28,039 --> 00:17:30,980
Geoffrey Hinton:
and through the process that
converts a symbol into features.
453
00:17:30,980 --> 00:17:34,460
Geoffrey Hinton:
And that way you learn, um,
feature vectors to represent the
454
00:17:34,460 --> 00:17:38,460
Geoffrey Hinton:
symbols and how these vectors
should interact to predict the
455
00:17:38,460 --> 00:17:40,059
Geoffrey Hinton:
features of the next symbol.
456
00:17:40,059 --> 00:17:43,380
Geoffrey Hinton:
And that's what these big
language models do, except it's
457
00:17:43,380 --> 00:17:44,339
Geoffrey Hinton:
a bit more complicated.
458
00:17:44,339 --> 00:17:46,819
Geoffrey Hinton:
The feature interactions are
much more complicated.
459
00:17:46,819 --> 00:17:47,859
Geoffrey Hinton:
They have many more layers of
460
00:17:47,859 --> 00:17:48,940
Geoffrey Hinton:
interaction, so they can
461
00:17:48,940 --> 00:17:51,140
Geoffrey Hinton:
disambiguate ambiguous symbols
462
00:17:51,140 --> 00:17:52,880
Geoffrey Hinton:
and get refine the shade of
463
00:17:52,880 --> 00:17:53,819
Geoffrey Hinton:
meaning of things where the
464
00:17:53,819 --> 00:17:54,740
Geoffrey Hinton:
meaning depends a lot on the
465
00:17:54,740 --> 00:17:55,859
Geoffrey Hinton:
context.
466
00:17:55,859 --> 00:18:01,660
Geoffrey Hinton:
Um, but it's basically an
extremely simple version of the
467
00:18:01,660 --> 00:18:02,819
Geoffrey Hinton:
current large language models.
468
00:18:02,819 --> 00:18:03,900
Geoffrey Hinton:
I called it a tiny language
469
00:18:03,900 --> 00:18:06,140
Geoffrey Hinton:
model, and that convinced the
470
00:18:06,140 --> 00:18:08,140
Geoffrey Hinton:
editors of nature that we really
471
00:18:08,140 --> 00:18:08,980
Geoffrey Hinton:
could learn interesting
472
00:18:08,980 --> 00:18:11,380
Geoffrey Hinton:
representations, because the
473
00:18:11,380 --> 00:18:12,500
Geoffrey Hinton:
vectors I learned for the
474
00:18:12,500 --> 00:18:14,099
Geoffrey Hinton:
symbols, which were people and
475
00:18:14,099 --> 00:18:15,740
Geoffrey Hinton:
relationships, they had six
476
00:18:15,740 --> 00:18:17,019
Geoffrey Hinton:
components.
477
00:18:17,019 --> 00:18:21,140
Geoffrey Hinton:
And if you used weight decay,
you could interpret what all
478
00:18:21,140 --> 00:18:22,220
Geoffrey Hinton:
those components were.
479
00:18:22,220 --> 00:18:24,579
Geoffrey Hinton:
And they were sensible semantic
features.
480
00:18:24,579 --> 00:18:27,400
Geoffrey Hinton:
They were the nationality of the
person and the generation of the
481
00:18:27,400 --> 00:18:30,960
Geoffrey Hinton:
person, and which branch of the
family tree they were in.
482
00:18:30,960 --> 00:18:33,839
Geoffrey Hinton:
And so it would learn things
483
00:18:33,839 --> 00:18:36,980
Geoffrey Hinton:
like the relationship uncle
484
00:18:36,980 --> 00:18:39,240
Geoffrey Hinton:
requires the output person to be
485
00:18:39,240 --> 00:18:40,640
Geoffrey Hinton:
one generation older than the
486
00:18:40,640 --> 00:18:41,839
Geoffrey Hinton:
input person.
487
00:18:42,920 --> 00:18:45,640
Geoffrey Hinton:
And so it would have generations
for people.
488
00:18:45,640 --> 00:18:46,799
Geoffrey Hinton:
And if the input person was a
489
00:18:46,799 --> 00:18:48,920
Geoffrey Hinton:
generation two, it would predict
490
00:18:48,920 --> 00:18:50,279
Geoffrey Hinton:
that the output person would be
491
00:18:50,279 --> 00:18:51,759
Geoffrey Hinton:
generation one.
492
00:18:51,759 --> 00:18:54,160
Geoffrey Hinton:
Um, so it was actually learning
493
00:18:54,160 --> 00:18:55,640
Geoffrey Hinton:
a whole bunch of little rules
494
00:18:55,640 --> 00:18:57,839
Geoffrey Hinton:
just probabilistically.
495
00:18:57,839 --> 00:18:59,920
Geoffrey Hinton:
And the people interested in
rule based induction got
496
00:18:59,920 --> 00:19:02,319
Geoffrey Hinton:
interested in it because they
said, oh, we can do that too.
497
00:19:02,319 --> 00:19:02,799
Geoffrey Hinton:
And it's true.
498
00:19:02,799 --> 00:19:04,319
Geoffrey Hinton:
They could do that too, with
499
00:19:04,319 --> 00:19:04,960
Geoffrey Hinton:
rules that weren't
500
00:19:04,960 --> 00:19:06,319
Geoffrey Hinton:
probabilistic.
501
00:19:06,319 --> 00:19:08,079
Geoffrey Hinton:
The point about neural nets is
502
00:19:08,079 --> 00:19:09,440
Geoffrey Hinton:
they can mimic something that
503
00:19:09,440 --> 00:19:11,440
Geoffrey Hinton:
learns discrete rules, but they
504
00:19:11,440 --> 00:19:11,799
Geoffrey Hinton:
can.
505
00:19:11,799 --> 00:19:14,640
Geoffrey Hinton:
They're also perfectly happy if
the rules are just usually true.
506
00:19:14,640 --> 00:19:17,359
Geoffrey Hinton:
And they use the preponderance
of the evidence then, which is
507
00:19:17,359 --> 00:19:20,960
Geoffrey Hinton:
much harder to do in, um, logic.
508
00:19:20,960 --> 00:19:27,059
Geoffrey Hinton:
And so that it was that example
which, curiously, was a little
509
00:19:27,059 --> 00:19:30,460
Geoffrey Hinton:
language model, um, that
convinced the editors of nature
510
00:19:30,460 --> 00:19:31,779
Geoffrey Hinton:
to publish a paper.
511
00:19:31,779 --> 00:19:33,619
Geoffrey Hinton:
I know because I talked to them
later.
512
00:19:33,619 --> 00:19:35,700
Geoffrey Hinton:
The referees, I talked to the
referee, one of the referees
513
00:19:35,700 --> 00:19:38,700
Geoffrey Hinton:
later, and he said, yeah, it was
that example that did it.
514
00:19:38,700 --> 00:19:40,819
Geoffrey Hinton:
And then we were all very
excited.
515
00:19:40,819 --> 00:19:44,140
Geoffrey Hinton:
We thought, we can solve
everything.
516
00:19:44,140 --> 00:19:45,140
Geoffrey Hinton:
You just have to give it a lot
517
00:19:45,140 --> 00:19:46,460
Geoffrey Hinton:
of training data and run
518
00:19:46,460 --> 00:19:48,180
Geoffrey Hinton:
backprop, and it'll learn all
519
00:19:48,180 --> 00:19:49,940
Geoffrey Hinton:
the representations you need and
520
00:19:49,940 --> 00:19:50,859
Geoffrey Hinton:
it'll learn to do parallel
521
00:19:50,859 --> 00:19:51,900
Geoffrey Hinton:
computation.
522
00:19:51,900 --> 00:19:53,779
Geoffrey Hinton:
Because at that time people were
very interested in parallel
523
00:19:53,779 --> 00:19:56,940
Geoffrey Hinton:
computation, but it was quite
hard to program.
524
00:19:56,940 --> 00:19:59,740
Geoffrey Hinton:
And the idea was, well, this
will have all these neurons
525
00:19:59,740 --> 00:20:02,420
Geoffrey Hinton:
inside and they'll all be
operating in parallel and it'll
526
00:20:02,420 --> 00:20:05,700
Geoffrey Hinton:
figure out how to use them so
there aren't any problems.
527
00:20:05,700 --> 00:20:07,900
Geoffrey Hinton:
At that point, people were very
interested in races and things
528
00:20:07,900 --> 00:20:10,140
Geoffrey Hinton:
like that, and you didn't have
to worry about any of that.
529
00:20:10,140 --> 00:20:11,299
Geoffrey Hinton:
It was all synchronous and you
530
00:20:11,299 --> 00:20:12,660
Geoffrey Hinton:
just they just learned what to
531
00:20:12,660 --> 00:20:13,259
Geoffrey Hinton:
do.
532
00:20:13,259 --> 00:20:18,460
Geoffrey Hinton:
Um, so we thought we'd solved
everything and little did we
533
00:20:18,460 --> 00:20:19,539
Geoffrey Hinton:
know we had.
534
00:20:19,539 --> 00:20:22,099
Geoffrey Hinton:
It's just we needed more data
and more compute.
535
00:20:24,440 --> 00:20:26,440
Tom Mitchell:
So then there's the long period
536
00:20:26,440 --> 00:20:27,880
Tom Mitchell:
of waiting for more data and
537
00:20:27,880 --> 00:20:28,960
Tom Mitchell:
more compute.
538
00:20:29,559 --> 00:20:34,359
Geoffrey Hinton:
And yeah, not realizing that
that was the main problem.
539
00:20:34,359 --> 00:20:36,440
Geoffrey Hinton:
Obviously, with other little
problems, there were more
540
00:20:36,440 --> 00:20:39,859
Geoffrey Hinton:
sensible kinds of neurons to use
than more sensible ways to
541
00:20:39,859 --> 00:20:41,440
Geoffrey Hinton:
regularize it and all that.
542
00:20:41,440 --> 00:20:43,000
Geoffrey Hinton:
Um, and things like transformers
543
00:20:43,000 --> 00:20:44,039
Geoffrey Hinton:
had to be invented to make it
544
00:20:44,039 --> 00:20:45,079
Geoffrey Hinton:
really efficient.
545
00:20:45,079 --> 00:20:50,359
Geoffrey Hinton:
Um, but basically backprop was
the way to do it.
546
00:20:50,359 --> 00:20:52,680
Geoffrey Hinton:
And you couldn't convince
547
00:20:52,680 --> 00:20:53,880
Geoffrey Hinton:
anybody when computers were
548
00:20:53,880 --> 00:20:54,559
Geoffrey Hinton:
slow.
549
00:20:54,559 --> 00:20:56,519
Geoffrey Hinton:
It will work for little
problems.
550
00:20:56,519 --> 00:20:57,759
Geoffrey Hinton:
It will work for slightly bigger
551
00:20:57,759 --> 00:20:59,480
Geoffrey Hinton:
problems, like a few years
552
00:20:59,480 --> 00:20:59,960
Geoffrey Hinton:
later.
553
00:20:59,960 --> 00:21:03,680
Geoffrey Hinton:
Yang got it working for for
mNIST, recognizing digits.
554
00:21:03,680 --> 00:21:05,220
Geoffrey Hinton:
But all the vision people said,
555
00:21:05,220 --> 00:21:06,240
Geoffrey Hinton:
you know, that's not real
556
00:21:06,240 --> 00:21:07,279
Geoffrey Hinton:
vision.
557
00:21:07,279 --> 00:21:10,680
Geoffrey Hinton:
Um, you're never going to do it
with real images that are high
558
00:21:10,680 --> 00:21:12,279
Geoffrey Hinton:
resolution on the web.
559
00:21:13,400 --> 00:21:16,500
Geoffrey Hinton:
And so it wasn't until about
twenty twelve that they had to
560
00:21:16,500 --> 00:21:18,000
Geoffrey Hinton:
eat their words.
561
00:21:19,680 --> 00:21:20,240
Tom Mitchell:
That's right.
562
00:21:20,240 --> 00:21:21,700
Tom Mitchell:
That was the year when.
563
00:21:21,700 --> 00:21:22,980
Tom Mitchell:
Well, you tell this story.
564
00:21:22,980 --> 00:21:24,900
Tom Mitchell:
You were the first person.
565
00:21:25,740 --> 00:21:29,059
Geoffrey Hinton:
Uh, well, I was the advisor of
the first two people.
566
00:21:29,059 --> 00:21:31,099
Geoffrey Hinton:
Now, it's not quite fair,
because Jan had already
567
00:21:31,099 --> 00:21:34,460
Geoffrey Hinton:
basically shown that they worked
for real images.
568
00:21:34,460 --> 00:21:36,700
Geoffrey Hinton:
Um, and Jan realized when Feifei
569
00:21:36,700 --> 00:21:37,660
Geoffrey Hinton:
came up with the ImageNet
570
00:21:37,660 --> 00:21:38,619
Geoffrey Hinton:
dataset.
571
00:21:38,619 --> 00:21:39,819
Geoffrey Hinton:
Jan realized they could win that
572
00:21:39,819 --> 00:21:41,619
Geoffrey Hinton:
competition, and he tried to get
573
00:21:41,619 --> 00:21:43,099
Geoffrey Hinton:
graduate students and postdocs
574
00:21:43,099 --> 00:21:44,579
Geoffrey Hinton:
in his lab to do it, and they
575
00:21:44,579 --> 00:21:45,819
Geoffrey Hinton:
all declined.
576
00:21:47,420 --> 00:21:54,059
Geoffrey Hinton:
Um, and Ilya, Ilya Sutskever
realized that, um, backprop
577
00:21:54,059 --> 00:21:56,140
Geoffrey Hinton:
would just kill ImageNet.
578
00:21:56,140 --> 00:22:00,380
Geoffrey Hinton:
Um, and he wanted Alex to work
on it.
579
00:22:00,380 --> 00:22:02,259
Geoffrey Hinton:
And I didn't really want to work
on it.
580
00:22:02,259 --> 00:22:03,500
Geoffrey Hinton:
Um, Alex had already been
581
00:22:03,500 --> 00:22:04,819
Geoffrey Hinton:
working on small images and
582
00:22:04,819 --> 00:22:06,079
Geoffrey Hinton:
recognizing small images in
583
00:22:06,079 --> 00:22:07,019
Geoffrey Hinton:
c410.
584
00:22:07,019 --> 00:22:09,660
Geoffrey Hinton:
Um, and you pre-processed
585
00:22:09,660 --> 00:22:10,740
Geoffrey Hinton:
everything for Alex to make it
586
00:22:10,740 --> 00:22:11,660
Geoffrey Hinton:
easy.
587
00:22:11,660 --> 00:22:14,119
Geoffrey Hinton:
And I bought Alex two Nvidia
588
00:22:14,119 --> 00:22:16,460
Geoffrey Hinton:
GPUs to have in his bedroom at
589
00:22:16,460 --> 00:22:17,259
Geoffrey Hinton:
home.
590
00:22:17,259 --> 00:22:22,359
Geoffrey Hinton:
Um, And Alex, then get on with
get on with it.
591
00:22:22,359 --> 00:22:24,160
Geoffrey Hinton:
And he was an absolutely wizard
programmer.
592
00:22:24,160 --> 00:22:27,000
Geoffrey Hinton:
He wrote amazing code on
593
00:22:27,000 --> 00:22:29,000
Geoffrey Hinton:
multiple GPUs to do convolution
594
00:22:29,000 --> 00:22:29,839
Geoffrey Hinton:
really efficiently.
595
00:22:29,839 --> 00:22:32,359
Geoffrey Hinton:
Much better code than anybody
else had ever written.
596
00:22:32,359 --> 00:22:40,279
Geoffrey Hinton:
Um, I believe and so it's a
combination of Ilya realizing we
597
00:22:40,279 --> 00:22:43,640
Geoffrey Hinton:
really had to do this, and Ilya
was involved in the design of
598
00:22:43,640 --> 00:22:48,220
Geoffrey Hinton:
the net and so on, but Alex's
programming skills and then I
599
00:22:48,220 --> 00:22:51,920
Geoffrey Hinton:
added a few ideas, like use
rectified linear units instead
600
00:22:51,920 --> 00:22:56,480
Geoffrey Hinton:
of sigmoid units and use little
patches of the images.
601
00:22:56,480 --> 00:22:57,440
Geoffrey Hinton:
I mean big patches of the
602
00:22:57,440 --> 00:22:59,079
Geoffrey Hinton:
images, so you can translate
603
00:22:59,079 --> 00:23:00,559
Geoffrey Hinton:
things around a bit to get some
604
00:23:00,559 --> 00:23:02,480
Geoffrey Hinton:
translation invariance, as well
605
00:23:02,480 --> 00:23:05,799
Geoffrey Hinton:
as using convolution, um, and
606
00:23:05,799 --> 00:23:06,640
Geoffrey Hinton:
use dropout.
607
00:23:06,640 --> 00:23:08,920
Geoffrey Hinton:
So that was one of the first
applications of dropout.
608
00:23:08,920 --> 00:23:11,119
Geoffrey Hinton:
And that helped about one
percent.
609
00:23:11,119 --> 00:23:13,200
Geoffrey Hinton:
It really helped.
610
00:23:13,200 --> 00:23:15,880
Geoffrey Hinton:
And then we beat the best vision
systems.
611
00:23:15,880 --> 00:23:19,069
Geoffrey Hinton:
The best vision systems were
sort of plateauing at twenty
612
00:23:19,069 --> 00:23:21,049
Geoffrey Hinton:
five percent errors.
613
00:23:21,049 --> 00:23:23,410
Geoffrey Hinton:
That's errors for getting the
right answer in the top in your
614
00:23:23,410 --> 00:23:24,769
Geoffrey Hinton:
top five bets.
615
00:23:24,769 --> 00:23:30,089
Geoffrey Hinton:
Um, and we got like fifteen
percent, fifteen or sixteen
616
00:23:30,089 --> 00:23:31,650
Geoffrey Hinton:
depending on how you count it.
617
00:23:31,650 --> 00:23:33,809
Geoffrey Hinton:
So we got almost half the error
rate.
618
00:23:35,049 --> 00:23:37,369
Geoffrey Hinton:
And what happened then was what
619
00:23:37,369 --> 00:23:38,529
Geoffrey Hinton:
ought to happen in science but
620
00:23:38,529 --> 00:23:40,569
Geoffrey Hinton:
seldom does.
621
00:23:40,569 --> 00:23:45,109
Geoffrey Hinton:
So our most vigorous opponents,
like Jitendra Malik and
622
00:23:45,109 --> 00:23:48,609
Geoffrey Hinton:
Zisserman Andrew Zisserman,
looked at these results and
623
00:23:48,609 --> 00:23:50,970
Geoffrey Hinton:
said, okay, you were right.
624
00:23:50,970 --> 00:23:53,049
Geoffrey Hinton:
That never happens in science.
625
00:23:53,049 --> 00:23:56,009
Geoffrey Hinton:
And slightly irritating the
Andrew Zisserman then switched
626
00:23:56,009 --> 00:23:57,650
Geoffrey Hinton:
to doing this.
627
00:23:57,650 --> 00:24:00,849
Geoffrey Hinton:
He had some very good postdocs
or students working with him.
628
00:24:00,849 --> 00:24:07,089
Geoffrey Hinton:
Simonyan um, and um, after about
a year, they were making better
629
00:24:07,089 --> 00:24:08,890
Geoffrey Hinton:
networks than us.
630
00:24:08,890 --> 00:24:11,049
Geoffrey Hinton:
But that was really the.
631
00:24:13,009 --> 00:24:14,289
Geoffrey Hinton:
As far as the general public was
632
00:24:14,289 --> 00:24:15,710
Geoffrey Hinton:
concerned, that was the start of
633
00:24:15,710 --> 00:24:17,549
Geoffrey Hinton:
this big swing towards deep
634
00:24:17,549 --> 00:24:19,390
Geoffrey Hinton:
learning in twenty twelve when
635
00:24:19,390 --> 00:24:20,589
Geoffrey Hinton:
we really nailed computer
636
00:24:20,589 --> 00:24:21,029
Geoffrey Hinton:
vision.
637
00:24:21,029 --> 00:24:22,309
Geoffrey Hinton:
But it actually happened before
that.
638
00:24:22,309 --> 00:24:25,509
Geoffrey Hinton:
It happened in two thousand and
nine when we showed how you
639
00:24:25,509 --> 00:24:28,509
Geoffrey Hinton:
could do speech recognition, or
rather the acoustic modeling
640
00:24:28,509 --> 00:24:29,950
Geoffrey Hinton:
part of speech recognition.
641
00:24:29,950 --> 00:24:31,109
Geoffrey Hinton:
We showed how you could do that
642
00:24:31,109 --> 00:24:32,269
Geoffrey Hinton:
a bit better than the best
643
00:24:32,269 --> 00:24:33,750
Geoffrey Hinton:
technology.
644
00:24:33,750 --> 00:24:36,829
Geoffrey Hinton:
And that influenced all the big
speech groups, the big speech
645
00:24:36,829 --> 00:24:42,829
Geoffrey Hinton:
groups that IBM and Microsoft,
um, and somewhere else.
646
00:24:42,829 --> 00:24:43,150
Geoffrey Hinton:
Google.
647
00:24:43,150 --> 00:24:43,750
Geoffrey Hinton:
Yes.
648
00:24:43,750 --> 00:24:48,190
Geoffrey Hinton:
Um, all switched to doing neural
nets for acoustic modeling.
649
00:24:48,190 --> 00:24:51,069
Geoffrey Hinton:
And so by twenty ten, it was
650
00:24:51,069 --> 00:24:52,230
Geoffrey Hinton:
clear that neural nets were the
651
00:24:52,230 --> 00:24:53,150
Geoffrey Hinton:
right way to do acoustic
652
00:24:53,150 --> 00:24:54,230
Geoffrey Hinton:
modeling.
653
00:24:54,230 --> 00:24:56,230
Geoffrey Hinton:
And we had lots of people
onside.
654
00:24:56,230 --> 00:24:58,109
Geoffrey Hinton:
Um, and but in twenty twelve, it
655
00:24:58,109 --> 00:24:58,930
Geoffrey Hinton:
actually came out for the
656
00:24:58,930 --> 00:25:00,210
Geoffrey Hinton:
Android, and suddenly the
657
00:25:00,210 --> 00:25:02,109
Geoffrey Hinton:
Android caught up with Siri in
658
00:25:02,109 --> 00:25:04,069
Geoffrey Hinton:
speech recognition.
659
00:25:04,069 --> 00:25:07,430
Geoffrey Hinton:
So really we demonstrated it for
speech before that, but that
660
00:25:07,430 --> 00:25:09,390
Geoffrey Hinton:
didn't make a big impact.
661
00:25:09,390 --> 00:25:11,710
Geoffrey Hinton:
The reason it worked for speech
was they had a big data set.
662
00:25:11,710 --> 00:25:13,849
Geoffrey Hinton:
They had millions of examples.
663
00:25:13,849 --> 00:25:14,730
Geoffrey Hinton:
They were one of the areas.
664
00:25:14,730 --> 00:25:16,069
Geoffrey Hinton:
Unlike vision, they had big data
665
00:25:16,069 --> 00:25:17,369
Geoffrey Hinton:
sets because of the DARPA speech
666
00:25:17,369 --> 00:25:18,450
Geoffrey Hinton:
project.
667
00:25:18,450 --> 00:25:21,930
Geoffrey Hinton:
Um, because they really wanted
to be able to benchmark systems.
668
00:25:21,930 --> 00:25:25,089
Geoffrey Hinton:
Um, also, speech is easier than
vision.
669
00:25:25,089 --> 00:25:27,690
Geoffrey Hinton:
Speech is just vision with
either one or two pixels.
670
00:25:27,690 --> 00:25:29,809
Geoffrey Hinton:
It's just they change rather
fast.
671
00:25:29,809 --> 00:25:32,930
Geoffrey Hinton:
Um, and.
672
00:25:35,130 --> 00:25:39,250
Geoffrey Hinton:
So we demonstrated for speech
when we did it for vision.
673
00:25:39,250 --> 00:25:42,410
Geoffrey Hinton:
The big companies already knew
it worked for speech and they
674
00:25:42,410 --> 00:25:44,089
Geoffrey Hinton:
saw it work for vision.
675
00:25:44,089 --> 00:25:46,890
Geoffrey Hinton:
And so they realized it was sort
of universal.
676
00:25:46,890 --> 00:25:51,450
Geoffrey Hinton:
Um, it wasn't just a specific
trick for a specific domain.
677
00:25:51,450 --> 00:25:54,250
Geoffrey Hinton:
It will work for perception in
general.
678
00:25:54,250 --> 00:25:55,210
Geoffrey Hinton:
They didn't realize at that
679
00:25:55,210 --> 00:25:55,930
Geoffrey Hinton:
point it would work for
680
00:25:55,930 --> 00:25:56,410
Geoffrey Hinton:
language.
681
00:25:56,410 --> 00:25:59,529
Geoffrey Hinton:
And nor really did we, even
though our very first impressive
682
00:25:59,529 --> 00:26:01,130
Geoffrey Hinton:
example was for language.
683
00:26:01,130 --> 00:26:01,849
Geoffrey Hinton:
Um.
684
00:26:04,329 --> 00:26:04,690
Geoffrey Hinton:
Yeah.
685
00:26:04,690 --> 00:26:10,849
Geoffrey Hinton:
So in twenty twelve, there was
this big swing to neural
686
00:26:10,849 --> 00:26:17,750
Geoffrey Hinton:
networks And that's when Jensen
at Nvidia realized he finally
687
00:26:17,750 --> 00:26:22,190
Geoffrey Hinton:
realized those Nvidia boards
weren't just for gaming.
688
00:26:22,190 --> 00:26:25,349
Geoffrey Hinton:
They were supercomputers for
doing machine learning.
689
00:26:25,349 --> 00:26:29,750
Geoffrey Hinton:
Now, I actually gave a talk at
NIPS in two thousand and nine
690
00:26:29,750 --> 00:26:33,430
Geoffrey Hinton:
when I told a thousand people
this was about speech, I told a
691
00:26:33,430 --> 00:26:36,589
Geoffrey Hinton:
thousand people, if you want to
do machine learning now, you
692
00:26:36,589 --> 00:26:39,150
Geoffrey Hinton:
have to buy Nvidia GPUs.
693
00:26:39,150 --> 00:26:42,269
Geoffrey Hinton:
Nvidia GPUs will make your
program go about thirty times as
694
00:26:42,269 --> 00:26:47,789
Geoffrey Hinton:
fast because they're relatively
easy to utilize parallelism.
695
00:26:47,789 --> 00:26:49,990
Geoffrey Hinton:
They're just right for neural
nets.
696
00:26:49,990 --> 00:26:52,670
Geoffrey Hinton:
It was Rick Zelinsky who was a
student of mine at CMU, who told
697
00:26:52,670 --> 00:26:57,549
Geoffrey Hinton:
me that in about two thousand
and six, um, and it was true.
698
00:26:58,990 --> 00:27:03,210
Geoffrey Hinton:
And, um, I sent mail to Nvidia
699
00:27:03,210 --> 00:27:04,230
Geoffrey Hinton:
saying, how about giving me a
700
00:27:04,230 --> 00:27:04,589
Geoffrey Hinton:
free one?
701
00:27:04,589 --> 00:27:06,690
Geoffrey Hinton:
Because I told a thousand
machine learning researchers to
702
00:27:06,690 --> 00:27:08,430
Geoffrey Hinton:
buy your boards.
703
00:27:08,430 --> 00:27:10,450
Geoffrey Hinton:
And they declined.
704
00:27:10,450 --> 00:27:15,890
Geoffrey Hinton:
Um. Years later, Jensen came to
Toronto and gave a talk and
705
00:27:15,890 --> 00:27:19,490
Geoffrey Hinton:
mentioned how Toronto, you know,
was the place where they
706
00:27:19,490 --> 00:27:24,250
Geoffrey Hinton:
convinced him that Nvidia GPUs
were good for AI.
707
00:27:24,250 --> 00:27:26,589
Geoffrey Hinton:
Um, and that it all happened in
708
00:27:26,589 --> 00:27:28,250
Geoffrey Hinton:
twenty twelve, and I couldn't
709
00:27:28,250 --> 00:27:28,609
Geoffrey Hinton:
resist it.
710
00:27:28,609 --> 00:27:29,089
Geoffrey Hinton:
At the end.
711
00:27:29,089 --> 00:27:30,130
Geoffrey Hinton:
I said, well, I told you in two
712
00:27:30,130 --> 00:27:31,730
Geoffrey Hinton:
thousand and nine that you
713
00:27:31,730 --> 00:27:32,609
Geoffrey Hinton:
ignored me.
714
00:27:33,650 --> 00:27:38,250
Geoffrey Hinton:
And what he should have said
was, well, you're very silly.
715
00:27:38,250 --> 00:27:40,650
Geoffrey Hinton:
You should have bought stock in
two thousand and nine.
716
00:27:42,569 --> 00:27:44,569
Geoffrey Hinton:
If I'd done that, I'd be a
billionaire.
717
00:27:44,569 --> 00:27:49,289
Geoffrey Hinton:
Um, but, um, instead, he gave
me.
718
00:27:49,289 --> 00:27:52,170
Geoffrey Hinton:
He opened his briefcase and gave
me their very special, very
719
00:27:52,170 --> 00:27:55,730
Geoffrey Hinton:
latest GPU, of which they'd only
made a few that had twice as
720
00:27:55,730 --> 00:27:58,450
Geoffrey Hinton:
much memory as any other GPU.
721
00:27:58,450 --> 00:28:00,410
Geoffrey Hinton:
So that was a nice move by
Jensen.
722
00:28:00,809 --> 00:28:03,650
Tom Mitchell:
That's a great story too.
723
00:28:03,650 --> 00:28:09,390
Tom Mitchell:
So then in the twenty tens,
things really just kind of rapid
724
00:28:09,390 --> 00:28:11,710
Tom Mitchell:
fire started taking off.
725
00:28:11,710 --> 00:28:12,990
Tom Mitchell:
Take us through that.
726
00:28:14,309 --> 00:28:16,549
Geoffrey Hinton:
So speech worked.
727
00:28:16,549 --> 00:28:20,950
Geoffrey Hinton:
Um, I we got a good
collaboration between the
728
00:28:20,950 --> 00:28:24,549
Geoffrey Hinton:
research groups at IBM and
Google and, um.
729
00:28:29,589 --> 00:28:31,990
Geoffrey Hinton:
Toronto and Microsoft.
730
00:28:31,990 --> 00:28:33,069
Geoffrey Hinton:
Yeah.
731
00:28:33,069 --> 00:28:35,470
Geoffrey Hinton:
Um, we actually published a
joint paper, which is sort of
732
00:28:35,470 --> 00:28:39,750
Geoffrey Hinton:
quite rare in this stuff about
the sort of new view of how to
733
00:28:39,750 --> 00:28:41,950
Geoffrey Hinton:
do acoustic modeling.
734
00:28:41,950 --> 00:28:51,470
Geoffrey Hinton:
Um. And then we did vision, and
then, um, I started getting lots
735
00:28:51,470 --> 00:28:54,670
Geoffrey Hinton:
of requests from big companies
who wanted to.
736
00:28:54,670 --> 00:29:01,029
Geoffrey Hinton:
By me or by me and Alex and Ilya
or fund our company or get us to
737
00:29:01,029 --> 00:29:02,509
Geoffrey Hinton:
come work for them.
738
00:29:02,509 --> 00:29:06,450
Geoffrey Hinton:
Um, and I realized this stuff
was probably valuable.
739
00:29:06,450 --> 00:29:08,930
Geoffrey Hinton:
We had no idea how much it was
worth.
740
00:29:08,930 --> 00:29:11,529
Geoffrey Hinton:
Um, so Craig Butler, who was the
741
00:29:11,529 --> 00:29:12,450
Geoffrey Hinton:
chair of the Department of
742
00:29:12,450 --> 00:29:14,170
Geoffrey Hinton:
Computer Science, was an expert
743
00:29:14,170 --> 00:29:15,289
Geoffrey Hinton:
on auctions.
744
00:29:15,289 --> 00:29:17,250
Geoffrey Hinton:
And he said, you know, you
745
00:29:17,250 --> 00:29:18,970
Geoffrey Hinton:
should actually, since you have
746
00:29:18,970 --> 00:29:19,890
Geoffrey Hinton:
no idea what it's worth, but
747
00:29:19,890 --> 00:29:20,970
Geoffrey Hinton:
there's people, many people
748
00:29:20,970 --> 00:29:22,009
Geoffrey Hinton:
interested, you should set up an
749
00:29:22,009 --> 00:29:22,650
Geoffrey Hinton:
auction.
750
00:29:23,809 --> 00:29:29,450
Geoffrey Hinton:
So at Lake Tahoe, which seemed
like the appropriate place, um,
751
00:29:29,450 --> 00:29:32,609
Geoffrey Hinton:
in a casino, um, a casino hotel.
752
00:29:32,609 --> 00:29:38,369
Geoffrey Hinton:
In twenty twelve, Alex and I set
up a little company for the sole
753
00:29:38,369 --> 00:29:40,809
Geoffrey Hinton:
function of doing an aqua hire.
754
00:29:40,809 --> 00:29:44,609
Geoffrey Hinton:
And there was an auction
between, um, Microsoft and
755
00:29:44,609 --> 00:29:50,450
Geoffrey Hinton:
Google and DeepMind and Baidu.
756
00:29:50,450 --> 00:29:52,809
Geoffrey Hinton:
Um, DeepMind dropped out fairly
early.
757
00:29:52,809 --> 00:29:56,650
Geoffrey Hinton:
Um, and on the ground floor,
758
00:29:56,650 --> 00:29:57,809
Geoffrey Hinton:
they had all these people at
759
00:29:57,809 --> 00:29:59,490
Geoffrey Hinton:
slot machines with cigarettes
760
00:29:59,490 --> 00:30:00,369
Geoffrey Hinton:
hanging out the corner of their
761
00:30:00,369 --> 00:30:01,170
Geoffrey Hinton:
mouth, just pulling these
762
00:30:01,170 --> 00:30:02,009
Geoffrey Hinton:
levers.
763
00:30:02,009 --> 00:30:04,170
Geoffrey Hinton:
And every so often they made
like a thousand dollars and
764
00:30:04,170 --> 00:30:07,589
Geoffrey Hinton:
lights would flash, and we were
upstairs having an auction where
765
00:30:07,589 --> 00:30:09,630
Geoffrey Hinton:
you had to raise by a million.
766
00:30:09,630 --> 00:30:11,789
Geoffrey Hinton:
Um, that was fun.
767
00:30:11,789 --> 00:30:13,670
Geoffrey Hinton:
And the auction went on for
quite a long time.
768
00:30:13,670 --> 00:30:17,750
Geoffrey Hinton:
We were completely amazed when
it got to forty four million.
769
00:30:17,750 --> 00:30:21,710
Geoffrey Hinton:
It was so much money that we
couldn't imagine that any more
770
00:30:21,710 --> 00:30:23,789
Geoffrey Hinton:
money would be useful.
771
00:30:23,789 --> 00:30:25,470
Geoffrey Hinton:
I mean, that seemed like as much
772
00:30:25,470 --> 00:30:26,710
Geoffrey Hinton:
money as anybody could possibly
773
00:30:26,710 --> 00:30:27,470
Geoffrey Hinton:
want.
774
00:30:27,470 --> 00:30:29,630
Geoffrey Hinton:
Um, and so we then became much
775
00:30:29,630 --> 00:30:32,130
Geoffrey Hinton:
more concerned about who we
776
00:30:32,130 --> 00:30:34,630
Geoffrey Hinton:
worked for, and I wouldn't have
777
00:30:34,630 --> 00:30:35,529
Geoffrey Hinton:
been able to get to China
778
00:30:35,529 --> 00:30:36,630
Geoffrey Hinton:
because I couldn't fly at that
779
00:30:36,630 --> 00:30:37,910
Geoffrey Hinton:
time.
780
00:30:37,910 --> 00:30:41,150
Geoffrey Hinton:
And I'd spent the summer of
twenty twelve working with Jeff
781
00:30:41,150 --> 00:30:42,829
Geoffrey Hinton:
Dean at Google.
782
00:30:42,829 --> 00:30:44,430
Geoffrey Hinton:
And I got along really well with
Jeff Dean.
783
00:30:44,430 --> 00:30:46,309
Geoffrey Hinton:
It was a really nice group, and
784
00:30:46,309 --> 00:30:48,630
Geoffrey Hinton:
I figured it was much more
785
00:30:48,630 --> 00:30:50,549
Geoffrey Hinton:
important to work in a really
786
00:30:50,549 --> 00:30:52,509
Geoffrey Hinton:
nice place than to get more
787
00:30:52,509 --> 00:30:53,349
Geoffrey Hinton:
money.
788
00:30:53,349 --> 00:30:56,029
Geoffrey Hinton:
So we actually terminated the
auction.
789
00:30:56,029 --> 00:31:00,750
Geoffrey Hinton:
We told Baidu we got an offer he
couldn't refuse, and the offer
790
00:31:00,750 --> 00:31:03,009
Geoffrey Hinton:
we couldn't refuse was the
chance to work at Google with
791
00:31:03,009 --> 00:31:06,730
Geoffrey Hinton:
Jeff Dean, and that all worked
out very well.
792
00:31:08,329 --> 00:31:14,190
Geoffrey Hinton:
So then I was off to Google and
while we were there a year, um,
793
00:31:14,190 --> 00:31:21,470
Geoffrey Hinton:
along with Shockley and Yoshua
and Bodner in Montreal, um, they
794
00:31:21,470 --> 00:31:26,009
Geoffrey Hinton:
developed, uh, attention
language models with attention,
795
00:31:26,009 --> 00:31:31,970
Geoffrey Hinton:
which was a precursor of
Transformers, and showed that
796
00:31:31,970 --> 00:31:35,930
Geoffrey Hinton:
language models actually work
well for machine translation.
797
00:31:35,930 --> 00:31:37,450
Geoffrey Hinton:
And I think that was the final
798
00:31:37,450 --> 00:31:39,009
Geoffrey Hinton:
nail in the coffin of symbolic
799
00:31:39,009 --> 00:31:40,970
Geoffrey Hinton:
AI, because if anything was
800
00:31:40,970 --> 00:31:42,730
Geoffrey Hinton:
going to be good for symbolic
801
00:31:42,730 --> 00:31:44,849
Geoffrey Hinton:
AI, it was converting symbol
802
00:31:44,849 --> 00:31:46,049
Geoffrey Hinton:
strings in one language into
803
00:31:46,049 --> 00:31:47,089
Geoffrey Hinton:
symbol strings in another
804
00:31:47,089 --> 00:31:48,089
Geoffrey Hinton:
language.
805
00:31:48,089 --> 00:31:50,569
Geoffrey Hinton:
The idea that you might do that
by taking symbol strings and
806
00:31:50,569 --> 00:31:53,569
Geoffrey Hinton:
manipulating them actually
sounded quite plausible.
807
00:31:53,569 --> 00:31:55,809
Geoffrey Hinton:
Um, but that's not the way to do
it.
808
00:31:55,809 --> 00:31:58,009
Geoffrey Hinton:
The way to do it is to
understand what's being said in
809
00:31:58,009 --> 00:32:02,180
Geoffrey Hinton:
one language, by associating big
vectors with words appropriately
810
00:32:02,180 --> 00:32:05,740
Geoffrey Hinton:
vectors, and then convert that
to the other language.
811
00:32:05,740 --> 00:32:08,539
Geoffrey Hinton:
Um, so it was clear by about
812
00:32:08,539 --> 00:32:11,740
Geoffrey Hinton:
twenty fifteen that neural nets
813
00:32:11,740 --> 00:32:12,500
Geoffrey Hinton:
were going to do everything,
814
00:32:12,500 --> 00:32:14,140
Geoffrey Hinton:
including language.
815
00:32:14,140 --> 00:32:16,779
Geoffrey Hinton:
That's the point at which Gary
Marcus published a book chapter
816
00:32:16,779 --> 00:32:18,779
Geoffrey Hinton:
saying neural nets were okay.
817
00:32:18,779 --> 00:32:21,519
Geoffrey Hinton:
Maybe they could do object
recognition, but they'd never do
818
00:32:21,519 --> 00:32:24,420
Geoffrey Hinton:
language because language
involved novel sentences.
819
00:32:24,420 --> 00:32:26,099
Geoffrey Hinton:
They were already doing it.
820
00:32:30,819 --> 00:32:36,099
Tom Mitchell:
Well. So that was twenty
fifteen.
821
00:32:36,099 --> 00:32:37,859
Tom Mitchell:
You were still at Google?
822
00:32:38,180 --> 00:32:39,819
Geoffrey Hinton:
I was at Google. And Ilya then
823
00:32:39,819 --> 00:32:45,259
Geoffrey Hinton:
moved to OpenAI, um, around
824
00:32:45,259 --> 00:32:48,779
Geoffrey Hinton:
twenty fifteen, uh, maybe twenty
825
00:32:48,779 --> 00:32:49,859
Geoffrey Hinton:
fourteen, I can't remember the
826
00:32:49,859 --> 00:32:51,299
Geoffrey Hinton:
year.
827
00:32:51,299 --> 00:32:52,819
Geoffrey Hinton:
And, um.
828
00:32:54,819 --> 00:32:57,539
Geoffrey Hinton:
And then OpenAI did rather well.
829
00:32:57,539 --> 00:32:59,160
Geoffrey Hinton:
Um, Over an hour.
830
00:32:59,160 --> 00:33:01,279
Geoffrey Hinton:
I basically just took stuff that
had been done at Google on
831
00:33:01,279 --> 00:33:07,079
Geoffrey Hinton:
Transformers and put a nicer
interface on it, and realized
832
00:33:07,079 --> 00:33:11,039
Geoffrey Hinton:
which Google hadn't realized
that if you did human
833
00:33:11,039 --> 00:33:14,480
Geoffrey Hinton:
reinforcement learning, you
didn't need that many examples
834
00:33:14,480 --> 00:33:16,319
Geoffrey Hinton:
to make it behave nicer.
835
00:33:16,319 --> 00:33:19,920
Geoffrey Hinton:
Um, you didn't need like one
hundred million examples which
836
00:33:19,920 --> 00:33:22,359
Geoffrey Hinton:
you might have thought you could
do it with.
837
00:33:22,359 --> 00:33:25,480
Geoffrey Hinton:
Like some fraction of a million
examples would already make it
838
00:33:25,480 --> 00:33:27,200
Geoffrey Hinton:
behave a lot better.
839
00:33:27,200 --> 00:33:31,240
Geoffrey Hinton:
So you could actually train it
up to have nicer behavior.
840
00:33:32,279 --> 00:33:34,079
Geoffrey Hinton:
And that was ChatGPT.
841
00:33:34,079 --> 00:33:40,539
Geoffrey Hinton:
Um, Google was then in the
classic situation of not wanting
842
00:33:40,539 --> 00:33:43,680
Geoffrey Hinton:
to interfere with search, which
was its moneymaker.
843
00:33:44,799 --> 00:33:47,200
Geoffrey Hinton:
So it was in this difficult
situation.
844
00:33:47,200 --> 00:33:49,880
Geoffrey Hinton:
Do they do they release chatbots
or not?
845
00:33:49,880 --> 00:33:53,640
Geoffrey Hinton:
But when Microsoft teamed up
with OpenAI, they basically had
846
00:33:53,640 --> 00:33:55,039
Geoffrey Hinton:
to release them.
847
00:33:55,039 --> 00:33:58,099
Geoffrey Hinton:
Um, but they lost a few years.
848
00:33:58,099 --> 00:34:03,019
Geoffrey Hinton:
I think it was partly because
search was working so well, and
849
00:34:03,019 --> 00:34:06,220
Geoffrey Hinton:
it was obvious search would be
better if instead of using
850
00:34:06,220 --> 00:34:10,059
Geoffrey Hinton:
keywords, it used what you
meant, which would mean it had
851
00:34:10,059 --> 00:34:12,219
Geoffrey Hinton:
to understand what you meant.
852
00:34:12,219 --> 00:34:17,019
Geoffrey Hinton:
Um, but they didn't want to
undermine their moneymaker.
853
00:34:18,940 --> 00:34:21,619
Geoffrey Hinton:
No, that's based not on any
inside information.
854
00:34:21,619 --> 00:34:22,940
Geoffrey Hinton:
It just seems obvious.
855
00:34:25,179 --> 00:34:27,300
Tom Mitchell:
Pretty amazing.
856
00:34:27,300 --> 00:34:30,179
Tom Mitchell:
So. So here we are now.
857
00:34:30,179 --> 00:34:35,619
Tom Mitchell:
And you were famously on record,
uh, warning people about some of
858
00:34:35,619 --> 00:34:38,219
Tom Mitchell:
the risks of AI.
859
00:34:38,219 --> 00:34:41,380
Tom Mitchell:
Um, what should what should
860
00:34:41,380 --> 00:34:43,219
Tom Mitchell:
people who are working in this
861
00:34:43,219 --> 00:34:45,659
Tom Mitchell:
area do in response to that
862
00:34:45,659 --> 00:34:46,380
Tom Mitchell:
risk?
863
00:34:47,139 --> 00:34:49,539
Geoffrey Hinton:
Okay, so I didn't actually talk
864
00:34:49,539 --> 00:34:50,780
Geoffrey Hinton:
much about the risks until I
865
00:34:50,780 --> 00:34:51,340
Geoffrey Hinton:
left Google.
866
00:34:51,340 --> 00:34:52,699
Geoffrey Hinton:
I realized in the beginning of
867
00:34:52,699 --> 00:34:54,900
Geoffrey Hinton:
twenty twenty three there was a
868
00:34:54,900 --> 00:34:56,480
Geoffrey Hinton:
huge existential threat I hadn't
869
00:34:56,480 --> 00:35:00,079
Geoffrey Hinton:
fully appreciated, because it's
870
00:35:00,079 --> 00:35:01,320
Geoffrey Hinton:
a better form of intelligence
871
00:35:01,320 --> 00:35:03,960
Geoffrey Hinton:
than us, and it's better because
872
00:35:03,960 --> 00:35:06,519
Geoffrey Hinton:
it can share so different copies
873
00:35:06,519 --> 00:35:08,559
Geoffrey Hinton:
of the same neural net, can look
874
00:35:08,559 --> 00:35:10,719
Geoffrey Hinton:
at different data and share the
875
00:35:10,719 --> 00:35:11,920
Geoffrey Hinton:
gradient, and then update all
876
00:35:11,920 --> 00:35:13,920
Geoffrey Hinton:
their weights in sync and stay
877
00:35:13,920 --> 00:35:16,039
Geoffrey Hinton:
the same so they can keep doing
878
00:35:16,039 --> 00:35:16,880
Geoffrey Hinton:
that.
879
00:35:16,880 --> 00:35:19,559
Geoffrey Hinton:
And when they share the
gradient, they're sharing
880
00:35:19,559 --> 00:35:22,719
Geoffrey Hinton:
information they got from
different data sets.
881
00:35:22,719 --> 00:35:24,719
Geoffrey Hinton:
Um, out of the order of a
882
00:35:24,719 --> 00:35:26,199
Geoffrey Hinton:
trillion bits per episode of
883
00:35:26,199 --> 00:35:26,840
Geoffrey Hinton:
sharing.
884
00:35:26,840 --> 00:35:28,920
Geoffrey Hinton:
If they've got a trillion
weights.
885
00:35:28,920 --> 00:35:31,000
Geoffrey Hinton:
Whereas what we're doing now is
886
00:35:31,000 --> 00:35:32,639
Geoffrey Hinton:
sharing the information and
887
00:35:32,639 --> 00:35:33,639
Geoffrey Hinton:
maybe one hundred bits per
888
00:35:33,639 --> 00:35:34,639
Geoffrey Hinton:
sentence.
889
00:35:34,639 --> 00:35:38,639
Geoffrey Hinton:
Um, so a few bits per second,
maybe if we're lucky, we're
890
00:35:38,639 --> 00:35:40,719
Geoffrey Hinton:
sharing it ten bits per second.
891
00:35:40,719 --> 00:35:45,179
Geoffrey Hinton:
Um, and so you're comparing like
892
00:35:45,179 --> 00:35:46,840
Geoffrey Hinton:
trillions of bits with hundreds
893
00:35:46,840 --> 00:35:47,760
Geoffrey Hinton:
of bits.
894
00:35:47,760 --> 00:35:51,480
Geoffrey Hinton:
There are billions of times
better than us at sharing.
895
00:35:51,480 --> 00:35:55,139
Geoffrey Hinton:
And that's why if I'm running on
different hardware, they can
896
00:35:55,139 --> 00:35:57,420
Geoffrey Hinton:
learn so much more than us.
897
00:35:57,420 --> 00:35:58,659
Geoffrey Hinton:
They can learn from the whole
internet.
898
00:35:58,659 --> 00:36:01,159
Geoffrey Hinton:
It doesn't all have to go
through one piece of hardware,
899
00:36:01,159 --> 00:36:05,099
Geoffrey Hinton:
and it's going to get more
important that effect as we go
900
00:36:05,099 --> 00:36:09,139
Geoffrey Hinton:
to AI agents that operate in the
real world in real time.
901
00:36:09,139 --> 00:36:12,659
Geoffrey Hinton:
Most AI you images, you can just
speed them up and send them
902
00:36:12,659 --> 00:36:14,980
Geoffrey Hinton:
through one network very fast.
903
00:36:14,980 --> 00:36:18,179
Geoffrey Hinton:
Um, because obviously computers
operate, you know, thousands of
904
00:36:18,179 --> 00:36:19,780
Geoffrey Hinton:
times faster than a brain.
905
00:36:19,780 --> 00:36:22,420
Geoffrey Hinton:
But, um, if you're operating in
906
00:36:22,420 --> 00:36:24,340
Geoffrey Hinton:
the real world, you can't get
907
00:36:24,340 --> 00:36:25,940
Geoffrey Hinton:
experience faster.
908
00:36:25,940 --> 00:36:28,659
Geoffrey Hinton:
Um, because the real world has
an actual time scale.
909
00:36:28,659 --> 00:36:29,780
Geoffrey Hinton:
If you're interacting with other
910
00:36:29,780 --> 00:36:30,900
Geoffrey Hinton:
agents who take a little while
911
00:36:30,900 --> 00:36:33,739
Geoffrey Hinton:
to reply, um, then this
912
00:36:33,739 --> 00:36:35,699
Geoffrey Hinton:
advantage that different copies
913
00:36:35,699 --> 00:36:37,199
Geoffrey Hinton:
of the same neural net can share
914
00:36:37,199 --> 00:36:38,059
Geoffrey Hinton:
will be an even bigger
915
00:36:38,059 --> 00:36:39,940
Geoffrey Hinton:
advantage.
916
00:36:39,940 --> 00:36:42,039
Geoffrey Hinton:
So at that point, I decided
917
00:36:42,039 --> 00:36:43,300
Geoffrey Hinton:
there's all these short term
918
00:36:43,300 --> 00:36:45,099
Geoffrey Hinton:
threats, and it wasn't really my
919
00:36:45,099 --> 00:36:46,340
Geoffrey Hinton:
intention to warn about those,
920
00:36:46,340 --> 00:36:47,699
Geoffrey Hinton:
but I got sucked into warning
921
00:36:47,699 --> 00:36:49,059
Geoffrey Hinton:
about those because journalists
922
00:36:49,059 --> 00:36:50,260
Geoffrey Hinton:
always confuse the existential
923
00:36:50,260 --> 00:36:51,019
Geoffrey Hinton:
threat with all the other
924
00:36:51,019 --> 00:36:51,579
Geoffrey Hinton:
threats.
925
00:36:51,579 --> 00:36:53,360
Geoffrey Hinton:
They just muddle all the threats
together.
926
00:36:53,360 --> 00:36:56,539
Geoffrey Hinton:
They move seamlessly from
joblessness to fake videos to
927
00:36:56,539 --> 00:36:59,639
Geoffrey Hinton:
cyber attacks to lethal
autonomous weapons, as if
928
00:36:59,639 --> 00:37:00,960
Geoffrey Hinton:
they're all the same thing.
929
00:37:00,960 --> 00:37:03,139
Geoffrey Hinton:
Um, so I had to sort of clarify
930
00:37:03,139 --> 00:37:05,440
Geoffrey Hinton:
a lot of those threats, but my
931
00:37:05,440 --> 00:37:07,679
Geoffrey Hinton:
main worry was the much longer
932
00:37:07,679 --> 00:37:10,420
Geoffrey Hinton:
term threat, but not long enough
933
00:37:10,420 --> 00:37:12,760
Geoffrey Hinton:
that they will be much smarter
934
00:37:12,760 --> 00:37:13,800
Geoffrey Hinton:
than us.
935
00:37:13,800 --> 00:37:15,360
Geoffrey Hinton:
It's not necessarily the case,
936
00:37:15,360 --> 00:37:17,860
Geoffrey Hinton:
but I think most people, most
937
00:37:17,860 --> 00:37:19,840
Geoffrey Hinton:
neural net experts, believe that
938
00:37:19,840 --> 00:37:21,079
Geoffrey Hinton:
within twenty years we'll have
939
00:37:21,079 --> 00:37:22,639
Geoffrey Hinton:
superintelligent AI.
940
00:37:23,760 --> 00:37:26,519
Geoffrey Hinton:
We vary, you know, Demis thinks
it'll be about ten years.
941
00:37:26,519 --> 00:37:28,239
Geoffrey Hinton:
I think it may be as long as
twenty years.
942
00:37:28,239 --> 00:37:31,360
Geoffrey Hinton:
And it'll very likely be more
than five years.
943
00:37:31,360 --> 00:37:33,639
Geoffrey Hinton:
Um, Dario thinks it'll be three
years.
944
00:37:33,639 --> 00:37:35,960
Geoffrey Hinton:
Um, but then he runs a company.
945
00:37:35,960 --> 00:37:39,199
Geoffrey Hinton:
Um, so any.
946
00:37:39,199 --> 00:37:42,280
Geoffrey Hinton:
Ilya thinks it'll be sooner than
ten years.
947
00:37:42,280 --> 00:37:45,159
Geoffrey Hinton:
Um, we all think it's probably
going to happen.
948
00:37:45,159 --> 00:37:47,519
Geoffrey Hinton:
So the question is, what happens
949
00:37:47,519 --> 00:37:50,010
Geoffrey Hinton:
when AI is a lot smarter than us
950
00:37:50,010 --> 00:37:51,699
Geoffrey Hinton:
And when it's our agents that
951
00:37:51,699 --> 00:37:52,780
Geoffrey Hinton:
are smart enough so they're also
952
00:37:52,780 --> 00:37:55,139
Geoffrey Hinton:
more powerful than us, they can
953
00:37:55,139 --> 00:37:56,400
Geoffrey Hinton:
collaborate with other AI
954
00:37:56,400 --> 00:37:58,860
Geoffrey Hinton:
agents, get stuff done even if
955
00:37:58,860 --> 00:38:02,219
Geoffrey Hinton:
they can't sort of fire guns or
956
00:38:02,219 --> 00:38:03,820
Geoffrey Hinton:
pull switches.
957
00:38:03,820 --> 00:38:05,139
Geoffrey Hinton:
They can persuade people.
958
00:38:05,139 --> 00:38:06,460
Geoffrey Hinton:
And we know AI is already very
959
00:38:06,460 --> 00:38:08,079
Geoffrey Hinton:
good at persuasion and will soon
960
00:38:08,079 --> 00:38:09,219
Geoffrey Hinton:
be much better than people at
961
00:38:09,219 --> 00:38:11,500
Geoffrey Hinton:
persuasion, like in ten years
962
00:38:11,500 --> 00:38:12,699
Geoffrey Hinton:
time.
963
00:38:12,699 --> 00:38:15,179
Geoffrey Hinton:
And so they'll be able to
persuade people to do things
964
00:38:15,179 --> 00:38:18,460
Geoffrey Hinton:
just like Trump persuaded people
to invade the capital.
965
00:38:18,460 --> 00:38:20,739
Geoffrey Hinton:
Um, so they don't actually have
to be able to do anything
966
00:38:20,739 --> 00:38:22,179
Geoffrey Hinton:
themselves except talk.
967
00:38:24,099 --> 00:38:28,800
Geoffrey Hinton:
So most of the tech bros are
968
00:38:28,800 --> 00:38:31,039
Geoffrey Hinton:
thinking they have a model,
969
00:38:31,039 --> 00:38:34,099
Geoffrey Hinton:
which is I'm the CEO, you're the
970
00:38:34,099 --> 00:38:35,059
Geoffrey Hinton:
secretary.
971
00:38:35,059 --> 00:38:36,860
Geoffrey Hinton:
You're much smarter than me.
972
00:38:36,860 --> 00:38:38,480
Geoffrey Hinton:
Um, but I can always fire you,
973
00:38:38,480 --> 00:38:39,820
Geoffrey Hinton:
and you'll make my life really
974
00:38:39,820 --> 00:38:40,420
Geoffrey Hinton:
easy.
975
00:38:40,420 --> 00:38:41,440
Geoffrey Hinton:
Because whatever I want to
976
00:38:41,440 --> 00:38:43,619
Geoffrey Hinton:
happen, I'll sort of be like
977
00:38:43,619 --> 00:38:44,059
Geoffrey Hinton:
Star Trek.
978
00:38:44,059 --> 00:38:47,179
Geoffrey Hinton:
I'll say, make it so and it will
happen.
979
00:38:47,179 --> 00:38:49,840
Geoffrey Hinton:
Um. and I don't really have to
understand it.
980
00:38:49,840 --> 00:38:51,679
Geoffrey Hinton:
I'll still get the credit for it
because I said make it.
981
00:38:51,679 --> 00:38:56,480
Geoffrey Hinton:
So, um, I think that's their
model, and I just don't think
982
00:38:56,480 --> 00:38:57,599
Geoffrey Hinton:
that's going to work.
983
00:38:58,679 --> 00:39:00,679
Geoffrey Hinton:
I think the big problem is how
984
00:39:00,679 --> 00:39:02,039
Geoffrey Hinton:
do we prevent these things ever
985
00:39:02,039 --> 00:39:04,659
Geoffrey Hinton:
wanting to take control or to
986
00:39:04,659 --> 00:39:05,760
Geoffrey Hinton:
take over?
987
00:39:05,760 --> 00:39:08,239
Geoffrey Hinton:
They may have control, but they
988
00:39:08,239 --> 00:39:09,559
Geoffrey Hinton:
may still not want to replace
989
00:39:09,559 --> 00:39:10,800
Geoffrey Hinton:
us.
990
00:39:10,800 --> 00:39:14,239
Geoffrey Hinton:
And so I've fallen back on the
only example I know of a less
991
00:39:14,239 --> 00:39:17,679
Geoffrey Hinton:
intelligent thing controlling a
more intelligent thing.
992
00:39:17,679 --> 00:39:19,840
Geoffrey Hinton:
And that's a baby controlling a
mother.
993
00:39:22,119 --> 00:39:25,079
Geoffrey Hinton:
And evolution has put a huge
amount of work into that.
994
00:39:27,639 --> 00:39:31,159
Geoffrey Hinton:
So evolution has made sure the
mother cannot bear the sound of
995
00:39:31,159 --> 00:39:36,599
Geoffrey Hinton:
the baby crying, and the mother
gets huge rewards, um, for being
996
00:39:36,599 --> 00:39:38,119
Geoffrey Hinton:
nice to the baby.
997
00:39:38,119 --> 00:39:40,079
Geoffrey Hinton:
Um, lots of pleasurable
998
00:39:40,079 --> 00:39:41,519
Geoffrey Hinton:
sensations and just generally
999
00:39:41,519 --> 00:39:42,679
Geoffrey Hinton:
good feelings.
1000
00:39:42,679 --> 00:39:47,739
Geoffrey Hinton:
Um, and We need to do the same
1001
00:39:47,739 --> 00:39:49,019
Geoffrey Hinton:
for these eyes, for being nice
1002
00:39:49,019 --> 00:39:49,300
Geoffrey Hinton:
to us.
1003
00:39:49,300 --> 00:39:51,420
Geoffrey Hinton:
We're still making them.
1004
00:39:51,420 --> 00:39:55,340
Geoffrey Hinton:
And if we could make an AI that
was super intelligent but cared
1005
00:39:55,340 --> 00:39:59,219
Geoffrey Hinton:
more about us than it cared
about about itself or other
1006
00:39:59,219 --> 00:40:03,940
Geoffrey Hinton:
superintelligent AIS, then we
might be okay.
1007
00:40:03,940 --> 00:40:06,940
Geoffrey Hinton:
Um, but we have to accept that
we're going to be the babies,
1008
00:40:06,940 --> 00:40:10,980
Geoffrey Hinton:
and they're going to be the
mothers, and people aren't
1009
00:40:10,980 --> 00:40:12,219
Geoffrey Hinton:
prepared to accept that.
1010
00:40:12,219 --> 00:40:13,539
Geoffrey Hinton:
Trump's not prepared to accept
1011
00:40:13,539 --> 00:40:15,219
Geoffrey Hinton:
that Trump would never accept
1012
00:40:15,219 --> 00:40:15,860
Geoffrey Hinton:
that.
1013
00:40:16,940 --> 00:40:19,300
Geoffrey Hinton:
Um, I think we have a lot more
1014
00:40:19,300 --> 00:40:20,219
Geoffrey Hinton:
hope of the Chinese
1015
00:40:20,219 --> 00:40:22,179
Geoffrey Hinton:
understanding it.
1016
00:40:22,179 --> 00:40:23,380
Geoffrey Hinton:
So I recently went to Shanghai
1017
00:40:23,380 --> 00:40:24,219
Geoffrey Hinton:
and talked to a member of the
1018
00:40:24,219 --> 00:40:24,739
Geoffrey Hinton:
Politburo.
1019
00:40:24,739 --> 00:40:27,619
Geoffrey Hinton:
Me and Eric Schmidt, who aren't
natural allies.
1020
00:40:27,619 --> 00:40:31,500
Geoffrey Hinton:
We're in terms of politics with
a rather different Eric Schmidt,
1021
00:40:31,500 --> 00:40:33,820
Geoffrey Hinton:
for example, thinks Kissinger
was a good guy.
1022
00:40:33,820 --> 00:40:39,340
Geoffrey Hinton:
Um, but we agree on this
existential threat, and the
1023
00:40:39,340 --> 00:40:42,900
Geoffrey Hinton:
Chinese leadership will
understand it much better than
1024
00:40:42,900 --> 00:40:45,639
Geoffrey Hinton:
any of the other leaderships,
because many of them are
1025
00:40:45,639 --> 00:40:48,719
Geoffrey Hinton:
engineers and they actually
understand how this stuff works.
1026
00:40:48,719 --> 00:40:50,320
Geoffrey Hinton:
They understand the argument
1027
00:40:50,320 --> 00:40:51,159
Geoffrey Hinton:
that it's a better form of
1028
00:40:51,159 --> 00:40:52,199
Geoffrey Hinton:
intelligence.
1029
00:40:54,039 --> 00:40:59,440
Geoffrey Hinton:
But I think all the countries
will collaborate on can we make
1030
00:40:59,440 --> 00:41:03,679
Geoffrey Hinton:
it so that it cares more about
us than it does about itself?
1031
00:41:03,679 --> 00:41:05,300
Geoffrey Hinton:
Because there if any country
1032
00:41:05,300 --> 00:41:06,460
Geoffrey Hinton:
figured out how to do that, it'd
1033
00:41:06,460 --> 00:41:07,599
Geoffrey Hinton:
be very happy to tell the other
1034
00:41:07,599 --> 00:41:09,280
Geoffrey Hinton:
countries.
1035
00:41:09,280 --> 00:41:11,960
Geoffrey Hinton:
That's like preventing a global
nuclear war.
1036
00:41:11,960 --> 00:41:15,219
Geoffrey Hinton:
And there the USSR and America
collaborated in the nineteen
1037
00:41:15,219 --> 00:41:16,559
Geoffrey Hinton:
fifties on that.
1038
00:41:16,559 --> 00:41:18,320
Geoffrey Hinton:
Um, the height of the Cold War.
1039
00:41:18,320 --> 00:41:20,280
Geoffrey Hinton:
They still collaborated to
prevent that.
1040
00:41:21,719 --> 00:41:26,159
Geoffrey Hinton:
So what I think we should have
is research institutes in
1041
00:41:26,159 --> 00:41:29,679
Geoffrey Hinton:
different countries that get
access to their own country's
1042
00:41:29,679 --> 00:41:31,880
Geoffrey Hinton:
super smart AI, which they're
not going to give to any other
1043
00:41:31,880 --> 00:41:36,079
Geoffrey Hinton:
country and can do experiments
on how to make it not want to,
1044
00:41:36,079 --> 00:41:39,480
Geoffrey Hinton:
how to make it care more about
people than about itself.
1045
00:41:39,480 --> 00:41:43,429
Geoffrey Hinton:
Um, and share with other
countries how to do that,
1046
00:41:43,429 --> 00:41:46,409
Geoffrey Hinton:
because I believe the techniques
for doing that will be roughly
1047
00:41:46,409 --> 00:41:49,489
Geoffrey Hinton:
orthogonal to the techniques for
making it smarter.
1048
00:41:49,489 --> 00:41:50,449
Geoffrey Hinton:
They're not going to share the
1049
00:41:50,449 --> 00:41:51,210
Geoffrey Hinton:
techniques for making it
1050
00:41:51,210 --> 00:41:51,889
Geoffrey Hinton:
smarter, because they're all
1051
00:41:51,889 --> 00:41:52,869
Geoffrey Hinton:
doing cyber attacks on each
1052
00:41:52,869 --> 00:41:54,289
Geoffrey Hinton:
other, and they all know the
1053
00:41:54,289 --> 00:41:55,210
Geoffrey Hinton:
best.
1054
00:41:55,210 --> 00:41:56,489
Geoffrey Hinton:
You want a better AI to do
1055
00:41:56,489 --> 00:41:58,289
Geoffrey Hinton:
better cyber attacks and better
1056
00:41:58,289 --> 00:41:59,610
Geoffrey Hinton:
fake videos and better
1057
00:41:59,610 --> 00:42:01,210
Geoffrey Hinton:
autonomous weapons.
1058
00:42:01,210 --> 00:42:02,489
Geoffrey Hinton:
They're never going to share
that stuff.
1059
00:42:02,489 --> 00:42:04,369
Geoffrey Hinton:
They're anti-aligned.
1060
00:42:04,369 --> 00:42:06,969
Geoffrey Hinton:
But on not having AI replace us,
1061
00:42:06,969 --> 00:42:08,610
Geoffrey Hinton:
they're aligned so they will
1062
00:42:08,610 --> 00:42:09,489
Geoffrey Hinton:
collaborate.
1063
00:42:10,650 --> 00:42:12,530
Geoffrey Hinton:
Now, one of the things you ought
to mention, I ought to mention
1064
00:42:12,530 --> 00:42:14,730
Geoffrey Hinton:
Russ Salakhutdinov, um.
1065
00:42:17,210 --> 00:42:18,969
Geoffrey Hinton:
He was one of my best students.
1066
00:42:18,969 --> 00:42:23,530
Geoffrey Hinton:
Um, he came to Toronto, did his
PhD at Toronto.
1067
00:42:23,530 --> 00:42:27,530
Geoffrey Hinton:
Um, he did a postdoc with Josh
Tenenbaum, and then he wanted to
1068
00:42:27,530 --> 00:42:32,570
Geoffrey Hinton:
come back to Toronto, and he had
a faculty offer from Harvard.
1069
00:42:32,570 --> 00:42:35,530
Geoffrey Hinton:
And I really tried to get the
Department of Computer Science,
1070
00:42:35,530 --> 00:42:39,849
Geoffrey Hinton:
which had an open position in
machine learning, to give a job
1071
00:42:39,849 --> 00:42:42,309
Geoffrey Hinton:
to Russ and they refused.
1072
00:42:43,429 --> 00:42:47,230
Geoffrey Hinton:
Basically, this was about twenty
eleven or twelve.
1073
00:42:47,230 --> 00:42:49,090
Geoffrey Hinton:
No, this was two thousand and
1074
00:42:49,090 --> 00:42:50,349
Geoffrey Hinton:
probably twenty twelve or
1075
00:42:50,349 --> 00:42:51,030
Geoffrey Hinton:
thirteen.
1076
00:42:52,070 --> 00:42:55,670
Geoffrey Hinton:
My department was one of the
last departments to accept that
1077
00:42:55,670 --> 00:42:57,989
Geoffrey Hinton:
neural networks really worked.
1078
00:42:57,989 --> 00:42:59,630
Geoffrey Hinton:
They had a big AI group, and the
1079
00:42:59,630 --> 00:43:00,869
Geoffrey Hinton:
big AI group said you had got
1080
00:43:00,869 --> 00:43:01,869
Geoffrey Hinton:
several people in neural
1081
00:43:01,869 --> 00:43:02,469
Geoffrey Hinton:
networks already.
1082
00:43:02,469 --> 00:43:03,949
Geoffrey Hinton:
That's your quota.
1083
00:43:03,949 --> 00:43:07,070
Geoffrey Hinton:
We're short on people in
knowledge representation, and we
1084
00:43:07,070 --> 00:43:09,230
Geoffrey Hinton:
need as many people in
computational linguistics as we
1085
00:43:09,230 --> 00:43:10,750
Geoffrey Hinton:
do in neural networks.
1086
00:43:10,750 --> 00:43:14,510
Geoffrey Hinton:
Um, and they refused to give us
a job.
1087
00:43:14,510 --> 00:43:15,670
Geoffrey Hinton:
So we eventually got a job in
1088
00:43:15,670 --> 00:43:17,590
Geoffrey Hinton:
statistics so that it could be
1089
00:43:17,590 --> 00:43:18,510
Geoffrey Hinton:
in Toronto.
1090
00:43:19,829 --> 00:43:21,510
Geoffrey Hinton:
And we were trying to negotiate
1091
00:43:21,510 --> 00:43:22,750
Geoffrey Hinton:
it for him to move to computer
1092
00:43:22,750 --> 00:43:23,869
Geoffrey Hinton:
science.
1093
00:43:23,869 --> 00:43:28,829
Geoffrey Hinton:
And then CMU swooped in, and I
think they offered him tenure at
1094
00:43:28,829 --> 00:43:30,909
Geoffrey Hinton:
CMU, and that was that.
1095
00:43:31,230 --> 00:43:37,949
Tom Mitchell:
Well, you have a whole cadre of
former students who are, um,
1096
00:43:37,949 --> 00:43:42,909
Tom Mitchell:
really leading the charge,
leading the way, and in a lot of
1097
00:43:42,909 --> 00:43:46,489
Tom Mitchell:
areas of neural nets, it's
pretty amazing if you.
1098
00:43:46,329 --> 00:43:47,090
Geoffrey Hinton:
Well, it was luck.
1099
00:43:47,090 --> 00:43:48,010
Geoffrey Hinton:
It was luck basically.
1100
00:43:48,010 --> 00:43:50,730
Geoffrey Hinton:
There were so few people who
believed in neural nets.
1101
00:43:50,730 --> 00:43:51,289
Geoffrey Hinton:
Who was Yann?
1102
00:43:51,289 --> 00:43:54,489
Geoffrey Hinton:
There was Yoshua, there was me,
the Schmidhuber.
1103
00:43:54,489 --> 00:43:55,449
Geoffrey Hinton:
Uh, there were a few other
1104
00:43:55,449 --> 00:43:57,570
Geoffrey Hinton:
people, but MIT didn't have
1105
00:43:57,570 --> 00:43:57,969
Geoffrey Hinton:
anybody.
1106
00:43:57,969 --> 00:43:59,650
Geoffrey Hinton:
Stanford didn't have anybody.
1107
00:43:59,650 --> 00:44:01,809
Geoffrey Hinton:
Um, Berkeley didn't have
anybody.
1108
00:44:01,809 --> 00:44:04,010
Geoffrey Hinton:
Um, Mike Jordan made sure of
that.
1109
00:44:04,010 --> 00:44:11,250
Geoffrey Hinton:
And, um, so the few of us who
believed in it got the really
1110
00:44:11,250 --> 00:44:15,050
Geoffrey Hinton:
good students who believed in
it, and that was great.
1111
00:44:16,409 --> 00:44:17,170
Tom Mitchell:
It worked.
1112
00:44:17,969 --> 00:44:22,090
Geoffrey Hinton:
People like Russ and E and
George Stahl and other people.
1113
00:44:22,090 --> 00:44:22,530
Geoffrey Hinton:
It was.
1114
00:44:22,530 --> 00:44:23,130
Geoffrey Hinton:
Yeah.
1115
00:44:23,849 --> 00:44:26,849
Tom Mitchell:
So if you could, uh, one final
question.
1116
00:44:26,849 --> 00:44:35,170
Tom Mitchell:
If you could give advice to new
PhD students now entering this
1117
00:44:35,170 --> 00:44:37,710
Tom Mitchell:
area, what would you say?
1118
00:44:39,590 --> 00:44:41,670
Geoffrey Hinton:
Sometimes I'd say become a
plumber.
1119
00:44:41,670 --> 00:44:43,070
Geoffrey Hinton:
You're too late.
1120
00:44:43,070 --> 00:44:49,110
Geoffrey Hinton:
Um. But actually, I say, if
you're a CMU and you're doing
1121
00:44:49,110 --> 00:44:52,949
Geoffrey Hinton:
this, you may be in the small
fraction of people who survive
1122
00:44:52,949 --> 00:44:56,269
Geoffrey Hinton:
in ER and don't get replaced,
because for quite a while
1123
00:44:56,269 --> 00:45:00,710
Geoffrey Hinton:
there's going to be creative
people making our work better.
1124
00:45:00,710 --> 00:45:03,949
Geoffrey Hinton:
And you've got a good chance of
being one of those people if
1125
00:45:03,949 --> 00:45:04,989
Geoffrey Hinton:
you're at CMU.
1126
00:45:07,309 --> 00:45:07,869
Tom Mitchell:
All right.
1127
00:45:07,869 --> 00:45:09,550
Tom Mitchell:
Well, we'll take that.
1128
00:45:09,550 --> 00:45:13,789
Tom Mitchell:
Uh, Jeff, thank you so much for
spending the time sharing that.
1129
00:45:13,789 --> 00:45:18,349
Tom Mitchell:
Um, it's it's always great to
catch up and, um.
1130
00:45:18,349 --> 00:45:19,070
Tom Mitchell:
Thank you.
1131
00:45:19,710 --> 00:45:21,469
Geoffrey Hinton:
Okay. Well, thank you for
inviting me.
1132
00:45:24,349 --> 00:45:25,429
Speaker 3:
Tom Mitchell is the founders
1133
00:45:25,429 --> 00:45:26,789
Speaker 3:
university professor at Carnegie
1134
00:45:26,789 --> 00:45:27,989
Speaker 3:
Mellon University.
1135
00:45:27,989 --> 00:45:28,630
Speaker 3:
Machine learning.
1136
00:45:28,630 --> 00:45:29,349
Speaker 3:
How did we get here?
1137
00:45:29,349 --> 00:45:31,829
Speaker 3:
Is produced by the Stanford
Digital Economy Lab.
1138
00:45:31,829 --> 00:45:32,989
Speaker 3:
If you enjoyed this episode,
1139
00:45:32,989 --> 00:45:34,090
Speaker 3:
subscribe wherever you listen to
1140
00:45:34,090 --> 00:45:35,110
Speaker 3:
podcasts.
00:00:00,680 --> 00:00:02,560
Tom Mitchell:
Welcome to machine learning.
2
00:00:02,560 --> 00:00:03,919
Tom Mitchell:
How did we get here?
3
00:00:03,919 --> 00:00:05,799
Tom Mitchell:
I'm Tom Mitchell.
4
00:00:05,799 --> 00:00:08,339
Tom Mitchell:
Today's episode is an interview
5
00:00:08,339 --> 00:00:10,320
Tom Mitchell:
with Geoff Hinton, one of the
6
00:00:10,320 --> 00:00:13,519
Tom Mitchell:
pioneers in the field of neural
7
00:00:13,519 --> 00:00:14,919
Tom Mitchell:
network learning.
8
00:00:14,919 --> 00:00:18,660
Tom Mitchell:
Geoff started out early, as
you'll hear, in the seventies,
9
00:00:18,660 --> 00:00:22,199
Tom Mitchell:
nineteen seventies, and has
continued working in neural
10
00:00:22,199 --> 00:00:24,519
Tom Mitchell:
networks ever since.
11
00:00:24,519 --> 00:00:26,000
Tom Mitchell:
During the period of the
12
00:00:26,000 --> 00:00:28,280
Tom Mitchell:
nineteen nineties and early two
13
00:00:28,280 --> 00:00:30,480
Tom Mitchell:
thousand, when neural networks
14
00:00:30,480 --> 00:00:33,039
Tom Mitchell:
were really in disfavor in the
15
00:00:33,039 --> 00:00:35,119
Tom Mitchell:
field of machine learning, Geoff
16
00:00:35,119 --> 00:00:38,460
Tom Mitchell:
nevertheless persisted, and he
17
00:00:38,460 --> 00:00:41,600
Tom Mitchell:
co-led the triumphant return of
18
00:00:41,600 --> 00:00:43,159
Tom Mitchell:
neural networks in the form of
19
00:00:43,159 --> 00:00:46,500
Tom Mitchell:
deep networks in the twenty ten
20
00:00:46,500 --> 00:00:48,600
Tom Mitchell:
ish period.
21
00:00:48,600 --> 00:00:53,840
Tom Mitchell:
In twenty eighteen, Geoff, along
with Joshua Bengio and Yann
22
00:00:53,840 --> 00:00:58,600
Tom Mitchell:
LeCun, received the Turing Award
in Computer Science.
23
00:00:58,600 --> 00:01:00,899
Tom Mitchell:
That's the highest award given
24
00:01:00,899 --> 00:01:03,259
Tom Mitchell:
in the field of computer science
25
00:01:03,259 --> 00:01:06,359
Tom Mitchell:
to researchers in twenty twenty
26
00:01:06,359 --> 00:01:07,180
Tom Mitchell:
four.
27
00:01:07,180 --> 00:01:09,719
Tom Mitchell:
Jeff, along with John Hopcroft,
28
00:01:09,719 --> 00:01:12,439
Tom Mitchell:
were awarded the twenty twenty
29
00:01:12,439 --> 00:01:15,859
Tom Mitchell:
four Nobel Prize in Physics for
30
00:01:15,859 --> 00:01:18,540
Tom Mitchell:
their work on artificial neural
31
00:01:18,540 --> 00:01:20,019
Tom Mitchell:
networks.
32
00:01:20,019 --> 00:01:22,019
Tom Mitchell:
I hope you enjoy the episode.
33
00:01:37,420 --> 00:01:41,140
Tom Mitchell:
I'm pleased to have with me
today Geoff Hinton, one of the
34
00:01:41,140 --> 00:01:43,659
Tom Mitchell:
pioneers of machine learning.
35
00:01:43,659 --> 00:01:45,540
Tom Mitchell:
Geoff, great to see you again.
36
00:01:45,700 --> 00:01:47,019
Geoffrey Hinton:
Thanks for inviting me.
37
00:01:47,379 --> 00:01:52,579
Tom Mitchell:
What I'd like to do today is get
two things.
38
00:01:52,579 --> 00:01:54,659
Tom Mitchell:
Two types of things from you.
39
00:01:54,659 --> 00:01:59,400
Tom Mitchell:
One is your own personal history
and how you got into this field
40
00:01:59,400 --> 00:02:02,159
Tom Mitchell:
and what happened after you did.
41
00:02:02,159 --> 00:02:06,640
Tom Mitchell:
And the second is kind of your
perspective on the whole field
42
00:02:06,640 --> 00:02:11,280
Tom Mitchell:
of machine learning, AI, and how
things are turning out.
43
00:02:11,120 --> 00:02:13,599
Geoffrey Hinton:
So when I was in high school, I
had a very smart friend who was
44
00:02:13,599 --> 00:02:18,080
Geoffrey Hinton:
a very good mathematician and
read widely, unlike me.
45
00:02:18,080 --> 00:02:20,520
Geoffrey Hinton:
And he came into school one day
46
00:02:20,520 --> 00:02:22,039
Geoffrey Hinton:
and talked about how memories
47
00:02:22,039 --> 00:02:23,400
Geoffrey Hinton:
might be distributed over the
48
00:02:23,400 --> 00:02:24,800
Geoffrey Hinton:
brain rather than localized in a
49
00:02:24,800 --> 00:02:26,840
Geoffrey Hinton:
place like a hologram, because
50
00:02:26,840 --> 00:02:27,759
Geoffrey Hinton:
this would have been nineteen
51
00:02:27,759 --> 00:02:29,240
Geoffrey Hinton:
sixty six and holograms had just
52
00:02:29,240 --> 00:02:30,199
Geoffrey Hinton:
come out.
53
00:02:30,199 --> 00:02:32,680
Geoffrey Hinton:
And that got me interested in
54
00:02:32,680 --> 00:02:34,000
Geoffrey Hinton:
how our memories represented in
55
00:02:34,000 --> 00:02:35,400
Geoffrey Hinton:
the brain.
56
00:02:35,400 --> 00:02:37,960
Geoffrey Hinton:
And I've been interested in that
ever since.
57
00:02:39,639 --> 00:02:43,680
Tom Mitchell:
Now, when I met you, we were
both at Carnegie Mellon.
58
00:02:43,680 --> 00:02:46,500
Tom Mitchell:
It was nineteen eighty six when,
59
00:02:46,500 --> 00:02:48,240
Tom Mitchell:
uh, we really got to do some
60
00:02:48,240 --> 00:02:49,919
Tom Mitchell:
work together or teach a course
61
00:02:49,919 --> 00:02:50,759
Tom Mitchell:
together.
62
00:02:50,759 --> 00:02:51,400
Tom Mitchell:
What?
63
00:02:51,400 --> 00:02:53,039
Tom Mitchell:
How did you get from nineteen
64
00:02:53,039 --> 00:02:55,120
Tom Mitchell:
sixty six up till Eighteen
65
00:02:55,120 --> 00:02:55,939
Tom Mitchell:
eighty six.
66
00:02:55,939 --> 00:02:58,219
Tom Mitchell:
What was the path?
67
00:02:58,780 --> 00:02:59,780
Geoffrey Hinton:
Slightly
68
00:02:59,780 --> 00:03:03,159
Geoffrey Hinton:
rocky. So I went to
69
00:03:03,159 --> 00:03:07,340
Geoffrey Hinton:
university. I studied physics,
chemistry and physiology, and in
70
00:03:07,340 --> 00:03:11,460
Geoffrey Hinton:
physiology. In the last term,
they're going to teach us, um,
how the
71
00:03:11,460 --> 00:03:12,419
Geoffrey Hinton:
central nervous system
72
00:03:12,419 --> 00:03:13,780
Geoffrey Hinton:
worked. And I was very
73
00:03:13,780 --> 00:03:17,900
Geoffrey Hinton:
excited. And they taught us how
action potentials are conducted
74
00:03:17,900 --> 00:03:20,740
Geoffrey Hinton:
along an axon, which wasn't what
I meant by how it
75
00:03:20,740 --> 00:03:23,300
Geoffrey Hinton:
worked. And so I switched to
76
00:03:23,300 --> 00:03:26,219
Geoffrey Hinton:
philosophy. That was even less
77
00:03:26,219 --> 00:03:28,060
Geoffrey Hinton:
useful. And then I switched
78
00:03:28,060 --> 00:03:29,819
Geoffrey Hinton:
to psychology, which was
79
00:03:29,819 --> 00:03:30,300
Geoffrey Hinton:
completely
80
00:03:30,300 --> 00:03:32,219
Geoffrey Hinton:
hopeless. Um, and then I became
a
81
00:03:32,219 --> 00:03:36,020
Geoffrey Hinton:
carpenter. And after I'd been a
82
00:03:36,020 --> 00:03:38,520
Geoffrey Hinton:
carpenter for about nine months,
I met
83
00:03:38,520 --> 00:03:38,840
Geoffrey Hinton:
a
84
00:03:38,840 --> 00:03:42,020
Geoffrey Hinton:
carpenter. And he was so much
better than me, I decided it'd
be easier
85
00:03:42,020 --> 00:03:42,379
Geoffrey Hinton:
to be an
86
00:03:42,379 --> 00:03:46,180
Geoffrey Hinton:
academic. Um, so I went to
graduate
87
00:03:46,180 --> 00:03:48,259
Geoffrey Hinton:
school in Edinburgh, um,
88
00:03:48,259 --> 00:03:49,520
Geoffrey Hinton:
with Longuet-higgins, who
89
00:03:49,520 --> 00:03:50,979
Geoffrey Hinton:
had published interesting stuff
90
00:03:50,979 --> 00:03:53,759
Geoffrey Hinton:
on, um, using neural nets for
91
00:03:53,759 --> 00:03:53,919
Geoffrey Hinton:
a
92
00:03:53,919 --> 00:03:59,360
Geoffrey Hinton:
memory. Unfortunately, around
the time I arrived, Winograd's
thesis
93
00:03:59,360 --> 00:04:02,759
Geoffrey Hinton:
came out and he switched his
allegiance to symbolic AI
94
00:04:02,759 --> 00:04:03,800
Geoffrey Hinton:
and gave up on neural
95
00:04:03,800 --> 00:04:07,280
Geoffrey Hinton:
nets. And so I spent five years
as
96
00:04:07,280 --> 00:04:08,960
Geoffrey Hinton:
his graduate student with
97
00:04:08,960 --> 00:04:10,159
Geoffrey Hinton:
him, trying to persuade me to
give
98
00:04:10,159 --> 00:04:10,639
Geoffrey Hinton:
up neural
99
00:04:10,639 --> 00:04:12,280
Geoffrey Hinton:
nets. And he never
100
00:04:12,280 --> 00:04:16,439
Geoffrey Hinton:
succeeded. Um, in the end, he
was very helpful to
101
00:04:16,439 --> 00:04:18,980
Geoffrey Hinton:
me. But for a long time, there
was a lot of argument about how
102
00:04:18,980 --> 00:04:21,279
Geoffrey Hinton:
I should really be doing
symbolic AI, and all this neural
103
00:04:21,279 --> 00:04:22,079
Geoffrey Hinton:
net stuff was complete
104
00:04:22,079 --> 00:04:24,319
Geoffrey Hinton:
nonsense. And everybody else in
105
00:04:24,319 --> 00:04:26,019
Geoffrey Hinton:
Edinburgh believed that neural
nets
106
00:04:26,019 --> 00:04:26,199
Geoffrey Hinton:
were
107
00:04:26,199 --> 00:04:29,600
Geoffrey Hinton:
nonsense. Um, we actually a
couple of
108
00:04:29,600 --> 00:04:32,480
Geoffrey Hinton:
exceptions. There was a post-doc
called David Willshaw who'd
109
00:04:32,480 --> 00:04:35,199
Geoffrey Hinton:
done associative memory, and he
basically done something
110
00:04:35,199 --> 00:04:36,000
Geoffrey Hinton:
quite like Hopfield
111
00:04:36,000 --> 00:04:37,459
Geoffrey Hinton:
nets. But a long time before
112
00:04:37,459 --> 00:04:41,360
Geoffrey Hinton:
Hopfield and Aaron Sloman was a
113
00:04:41,360 --> 00:04:42,680
Geoffrey Hinton:
visitor for a while, and he was
114
00:04:42,680 --> 00:04:42,879
Geoffrey Hinton:
more
115
00:04:42,879 --> 00:04:47,360
Geoffrey Hinton:
sympathetic. Um, but basically
they all knew it was
116
00:04:47,360 --> 00:04:50,639
Geoffrey Hinton:
rubbish. And they would explain
to me
117
00:04:50,639 --> 00:04:52,470
Geoffrey Hinton:
how neural nets can't even
118
00:04:52,470 --> 00:04:52,610
Geoffrey Hinton:
do
119
00:04:52,610 --> 00:04:56,089
Geoffrey Hinton:
recursion. So because
everything,
120
00:04:56,089 --> 00:04:58,189
Geoffrey Hinton:
everybody believed in recursion,
then,
121
00:04:58,189 --> 00:05:00,930
Geoffrey Hinton:
um, I actually figured out how
to
122
00:05:00,930 --> 00:05:02,009
Geoffrey Hinton:
do true recursion in a
123
00:05:02,009 --> 00:05:04,889
Geoffrey Hinton:
neural network and implemented
it on
124
00:05:04,889 --> 00:05:05,370
Geoffrey Hinton:
a machine
125
00:05:05,370 --> 00:05:07,910
Geoffrey Hinton:
with. I think by then it had one
hundred and ninety two
126
00:05:07,910 --> 00:05:11,050
Geoffrey Hinton:
kilobytes of memory, and it was
only shared by forty
127
00:05:11,050 --> 00:05:14,410
Geoffrey Hinton:
people. Um, but it had a huge
disk that had two
128
00:05:14,410 --> 00:05:15,970
Geoffrey Hinton:
megabytes. So you never ran out
of your
129
00:05:15,970 --> 00:05:18,250
Geoffrey Hinton:
memory? Um, because you used
virtual
130
00:05:18,250 --> 00:05:20,449
Geoffrey Hinton:
memory. And I actually
implemented
131
00:05:20,449 --> 00:05:23,389
Geoffrey Hinton:
a little neural net that did
132
00:05:23,389 --> 00:05:23,569
Geoffrey Hinton:
true
133
00:05:23,569 --> 00:05:29,569
Geoffrey Hinton:
recursion. That is, in the
recursive call, it used the same
neurons and
134
00:05:29,569 --> 00:05:32,769
Geoffrey Hinton:
the same connection strings for
the recursive call as it did for
135
00:05:32,769 --> 00:05:33,290
Geoffrey Hinton:
the high level
136
00:05:33,290 --> 00:05:38,470
Geoffrey Hinton:
call. Now, if you do that, of
course it had to offload all
137
00:05:38,470 --> 00:05:41,889
Geoffrey Hinton:
the parameters of the high level
call into some short term
138
00:05:41,889 --> 00:05:43,569
Geoffrey Hinton:
memory onto a
139
00:05:43,569 --> 00:05:45,170
Geoffrey Hinton:
stack. Eventually, and I figured
140
00:05:45,170 --> 00:05:46,410
Geoffrey Hinton:
out how to implement a stack
141
00:05:46,410 --> 00:05:48,050
Geoffrey Hinton:
with associative memory in a
142
00:05:48,050 --> 00:05:48,410
Geoffrey Hinton:
neural
143
00:05:48,410 --> 00:05:52,350
Geoffrey Hinton:
net. Um, so I had this little
neural net running that was
doing
144
00:05:52,350 --> 00:05:56,189
Geoffrey Hinton:
full recursion in neural nets,
and that was the first talk I
145
00:05:56,189 --> 00:05:58,029
Geoffrey Hinton:
gave. And people were very
146
00:05:58,029 --> 00:06:01,670
Geoffrey Hinton:
puzzled. They said, why would
you want to do recursion in a
neural
147
00:06:01,670 --> 00:06:05,490
Geoffrey Hinton:
net? I mean, it's so easy to do
in pop two, which was our our
148
00:06:05,490 --> 00:06:11,910
Geoffrey Hinton:
sort of, uh, unfortunate bastard
child of Pascal and
149
00:06:11,910 --> 00:06:15,589
Geoffrey Hinton:
Lisp. Um, although I don't think
Pascal existed
150
00:06:15,589 --> 00:06:17,009
Geoffrey Hinton:
then. Um,
151
00:06:17,009 --> 00:06:18,990
Geoffrey Hinton:
so.
152
00:06:18,990 --> 00:06:22,209
Geoffrey Hinton:
Yeah. So I keep meaning to go
back to
153
00:06:22,629 --> 00:06:24,629
Tom Mitchell:
I was going to ask, is there a
154
00:06:24,629 --> 00:06:26,509
Tom Mitchell:
future to recursion for neural
155
00:06:26,509 --> 00:06:27,189
Tom Mitchell:
nets?
156
00:06:27,829 --> 00:06:28,670
Geoffrey Hinton:
Oh, yes.
157
00:06:28,670 --> 00:06:31,189
Geoffrey Hinton:
I mean, to do true recursion,
158
00:06:31,189 --> 00:06:32,389
Geoffrey Hinton:
you have to use the same neurons
159
00:06:32,389 --> 00:06:33,490
Geoffrey Hinton:
and weights for the recursive
160
00:06:33,490 --> 00:06:34,069
Geoffrey Hinton:
call.
161
00:06:34,069 --> 00:06:36,389
Geoffrey Hinton:
That means you have to have a
stack, something like a stack,
162
00:06:36,389 --> 00:06:38,949
Geoffrey Hinton:
to store the parameters of the
high level call.
163
00:06:38,949 --> 00:06:40,990
Geoffrey Hinton:
That all works if you have fast
weights.
164
00:06:40,990 --> 00:06:42,670
Geoffrey Hinton:
So that was the first thing I
did with fast weights in
165
00:06:42,670 --> 00:06:44,470
Geoffrey Hinton:
nineteen seventy three.
166
00:06:44,470 --> 00:06:47,029
Geoffrey Hinton:
I should say fast weights were
invented by Schmidhuber in
167
00:06:47,029 --> 00:06:48,269
Geoffrey Hinton:
nineteen ninety, something.
168
00:06:50,449 --> 00:06:51,370
Tom Mitchell:
Fair enough.
169
00:06:51,370 --> 00:06:56,569
Tom Mitchell:
Okay, so then, um, you moved on
from Edinburgh.
170
00:06:56,569 --> 00:06:58,470
Tom Mitchell:
Did you come directly to
171
00:06:58,470 --> 00:07:00,269
Tom Mitchell:
Carnegie Mellon from there, or
172
00:07:00,269 --> 00:07:01,009
Tom Mitchell:
how did.
173
00:07:00,850 --> 00:07:05,170
Geoffrey Hinton:
Oh, no. No. Um, I dropped out
174
00:07:05,170 --> 00:07:07,569
Geoffrey Hinton:
again after I finished my
175
00:07:07,569 --> 00:07:08,610
Geoffrey Hinton:
thesis.
176
00:07:08,610 --> 00:07:10,250
Geoffrey Hinton:
I dropped out and became a
177
00:07:10,250 --> 00:07:11,529
Geoffrey Hinton:
teacher in a free school in
178
00:07:11,529 --> 00:07:12,569
Geoffrey Hinton:
London.
179
00:07:12,569 --> 00:07:15,529
Geoffrey Hinton:
Um, it was voluntary, I was
unpaid.
180
00:07:15,529 --> 00:07:19,410
Geoffrey Hinton:
They were rough, emotionally
disturbed inner city kids.
181
00:07:19,410 --> 00:07:20,529
Geoffrey Hinton:
And after a few months of that,
182
00:07:20,529 --> 00:07:22,009
Geoffrey Hinton:
I again decided academia might
183
00:07:22,009 --> 00:07:23,129
Geoffrey Hinton:
be easier.
184
00:07:23,129 --> 00:07:28,410
Geoffrey Hinton:
Um, so I went back to a post-doc
with Aaron Sloman in Sussex.
185
00:07:28,410 --> 00:07:33,610
Geoffrey Hinton:
Higgins had moved from Edinburgh
to Sussex and um, as I was
186
00:07:33,610 --> 00:07:36,810
Geoffrey Hinton:
finishing my PhD, I got a
post-doc with Aaron Sloman.
187
00:07:36,810 --> 00:07:40,850
Geoffrey Hinton:
Um, and there were no proper
faculty jobs in Britain.
188
00:07:40,850 --> 00:07:42,889
Geoffrey Hinton:
Then there was one job in the
189
00:07:42,889 --> 00:07:43,970
Geoffrey Hinton:
whole of Britain which Alan
190
00:07:43,970 --> 00:07:49,310
Geoffrey Hinton:
Bundy got, um, and so I applied
191
00:07:49,310 --> 00:07:52,889
Geoffrey Hinton:
for jobs in the States, and I
192
00:07:52,889 --> 00:07:56,329
Geoffrey Hinton:
got a job as a postdoc in UCSD,
193
00:07:56,329 --> 00:07:57,790
Geoffrey Hinton:
um, with Don Norman and Dave
194
00:07:57,790 --> 00:07:59,149
Geoffrey Hinton:
Rumelhart.
195
00:07:59,149 --> 00:08:02,230
Geoffrey Hinton:
And I really got along very well
with Dave Rumelhart, and that
196
00:08:02,230 --> 00:08:03,550
Geoffrey Hinton:
made a huge difference.
197
00:08:03,550 --> 00:08:08,029
Geoffrey Hinton:
So I moved from a country where
it was a sort of small country,
198
00:08:08,029 --> 00:08:13,430
Geoffrey Hinton:
Britain, and there was only room
for one ideology, and the
199
00:08:13,430 --> 00:08:17,589
Geoffrey Hinton:
ideology was symbolic AI and
neural nets was just rubbish.
200
00:08:18,709 --> 00:08:22,189
Geoffrey Hinton:
And I moved to the states where,
um, on the West coast, on the
201
00:08:22,189 --> 00:08:24,930
Geoffrey Hinton:
East Coast, it was symbolic AI
but on the West Coast they were
202
00:08:24,930 --> 00:08:26,230
Geoffrey Hinton:
kind of more open.
203
00:08:26,230 --> 00:08:27,310
Geoffrey Hinton:
And in particular, Don Norman
204
00:08:27,310 --> 00:08:28,529
Geoffrey Hinton:
and Dave Rumelhart thought
205
00:08:28,529 --> 00:08:29,550
Geoffrey Hinton:
neural nets were worth
206
00:08:29,550 --> 00:08:30,790
Geoffrey Hinton:
considering.
207
00:08:30,790 --> 00:08:33,909
Geoffrey Hinton:
Um, so it was a huge liberation
208
00:08:33,909 --> 00:08:36,309
Geoffrey Hinton:
to be in a place where neural
209
00:08:36,309 --> 00:08:37,570
Geoffrey Hinton:
nets were regarded as not
210
00:08:37,570 --> 00:08:38,870
Geoffrey Hinton:
obvious nonsense.
211
00:08:40,350 --> 00:08:41,110
Geoffrey Hinton:
And.
212
00:08:43,149 --> 00:08:44,850
Geoffrey Hinton:
Um, I went there.
213
00:08:44,850 --> 00:08:48,769
Geoffrey Hinton:
I, I got to meet Terry Sinofsky,
who I invited to a conference.
214
00:08:48,769 --> 00:08:51,490
Geoffrey Hinton:
And we'd been sort of lifelong
friends and collaborators.
215
00:08:51,490 --> 00:08:55,409
Geoffrey Hinton:
Um, I got to meet Francis Crick
later on, who was there.
216
00:08:55,409 --> 00:08:56,610
Geoffrey Hinton:
So I was there for a couple of
years.
217
00:08:56,610 --> 00:08:57,529
Geoffrey Hinton:
And then I got a job in
218
00:08:57,529 --> 00:08:59,250
Geoffrey Hinton:
Cambridge in the Applied
219
00:08:59,250 --> 00:09:02,110
Geoffrey Hinton:
Psychology Research Unit, um,
220
00:09:02,110 --> 00:09:03,129
Geoffrey Hinton:
where I was meant to do applied
221
00:09:03,129 --> 00:09:04,769
Geoffrey Hinton:
psychology.
222
00:09:04,769 --> 00:09:08,370
Geoffrey Hinton:
And I was strongly reminded of
William James's comment about
223
00:09:08,370 --> 00:09:12,409
Geoffrey Hinton:
applied psychology, which is to
do applied psychology, you have
224
00:09:12,409 --> 00:09:14,210
Geoffrey Hinton:
to have something to apply.
225
00:09:14,210 --> 00:09:19,009
Geoffrey Hinton:
Um. But I actually did some
interesting stuff.
226
00:09:19,009 --> 00:09:21,049
Geoffrey Hinton:
It was just around the time some
227
00:09:21,049 --> 00:09:23,470
Geoffrey Hinton:
workstations were coming out,
228
00:09:23,470 --> 00:09:26,549
Geoffrey Hinton:
and they had a contract with the
229
00:09:26,549 --> 00:09:29,370
Geoffrey Hinton:
British Telephone Company to
230
00:09:29,370 --> 00:09:31,330
Geoffrey Hinton:
help with network management and
231
00:09:31,330 --> 00:09:32,330
Geoffrey Hinton:
network management.
232
00:09:32,330 --> 00:09:34,490
Geoffrey Hinton:
Then was all done by hand.
233
00:09:35,490 --> 00:09:38,190
Geoffrey Hinton:
And you had information about,
234
00:09:38,190 --> 00:09:39,850
Geoffrey Hinton:
um, the loads on various
235
00:09:39,850 --> 00:09:41,929
Geoffrey Hinton:
switching centers, and the
236
00:09:41,929 --> 00:09:44,230
Geoffrey Hinton:
information was on a huge wall
237
00:09:44,230 --> 00:09:46,990
Geoffrey Hinton:
that was twenty feet high and
238
00:09:46,990 --> 00:09:48,470
Geoffrey Hinton:
worked like you sometimes see at
239
00:09:48,470 --> 00:09:49,669
Geoffrey Hinton:
train stations.
240
00:09:49,669 --> 00:09:52,549
Geoffrey Hinton:
It was little flaps with white
letters on that come down.
241
00:09:52,549 --> 00:09:54,990
Geoffrey Hinton:
They sort of rotate around until
you get the right flap.
242
00:09:54,990 --> 00:09:58,730
Geoffrey Hinton:
And so you could see all these
numbers, um, that said how busy
243
00:09:58,730 --> 00:10:00,909
Geoffrey Hinton:
each switching station was.
244
00:10:00,909 --> 00:10:04,350
Geoffrey Hinton:
Um, and I figured a sun
245
00:10:04,350 --> 00:10:05,590
Geoffrey Hinton:
workstation could do that and
246
00:10:05,590 --> 00:10:08,309
Geoffrey Hinton:
would be a lot cheaper, and the
247
00:10:08,309 --> 00:10:09,590
Geoffrey Hinton:
resolution wasn't that good
248
00:10:09,590 --> 00:10:10,389
Geoffrey Hinton:
then.
249
00:10:10,389 --> 00:10:13,409
Geoffrey Hinton:
So I had to figure out if you
could display the states of all
250
00:10:13,409 --> 00:10:16,110
Geoffrey Hinton:
the switching stations in
Britain on the screen of a sun
251
00:10:16,110 --> 00:10:20,309
Geoffrey Hinton:
workstation, and you couldn't
type the names, but if you had
252
00:10:20,309 --> 00:10:24,750
Geoffrey Hinton:
two letters for each name, you
could get two letters there.
253
00:10:24,750 --> 00:10:28,509
Geoffrey Hinton:
And I worked on a display with
254
00:10:28,509 --> 00:10:30,389
Geoffrey Hinton:
two letter names, and there were
255
00:10:30,389 --> 00:10:31,470
Geoffrey Hinton:
a large number of switching
256
00:10:31,470 --> 00:10:32,470
Geoffrey Hinton:
stations.
257
00:10:32,470 --> 00:10:33,710
Geoffrey Hinton:
There were hundreds.
258
00:10:33,710 --> 00:10:34,950
Geoffrey Hinton:
And the question was could could
259
00:10:34,950 --> 00:10:36,429
Geoffrey Hinton:
an operator remember which they
260
00:10:36,429 --> 00:10:37,190
Geoffrey Hinton:
were?
261
00:10:37,190 --> 00:10:38,490
Geoffrey Hinton:
So I actually taught myself to
262
00:10:38,490 --> 00:10:39,710
Geoffrey Hinton:
remember all those two letter
263
00:10:39,710 --> 00:10:41,210
Geoffrey Hinton:
names.
264
00:10:41,210 --> 00:10:45,169
Geoffrey Hinton:
Um, they were in very small type
to fit on, and I actually got a
265
00:10:45,169 --> 00:10:47,370
Geoffrey Hinton:
serious migraine from looking at
it too long.
266
00:10:47,370 --> 00:10:52,809
Geoffrey Hinton:
Um, that was my interaction with
human factors, and then I.
267
00:10:53,690 --> 00:10:58,169
Tom Mitchell:
It does remind me of that was
around the time that Unix was
268
00:10:58,169 --> 00:11:02,690
Tom Mitchell:
getting invented, and all the
commands had no vowels in them.
269
00:11:02,690 --> 00:11:04,289
Tom Mitchell:
So there was a theme there?
270
00:11:04,610 --> 00:11:08,730
Geoffrey Hinton:
Yes. Um, so I wrote a report on
271
00:11:08,730 --> 00:11:09,769
Geoffrey Hinton:
it and they said it was a very
272
00:11:09,769 --> 00:11:10,169
Geoffrey Hinton:
nice report.
273
00:11:10,169 --> 00:11:11,450
Geoffrey Hinton:
Thank you very much.
274
00:11:11,450 --> 00:11:12,169
Geoffrey Hinton:
And they weren't going to
275
00:11:12,169 --> 00:11:13,970
Geoffrey Hinton:
implement it, even though it
276
00:11:13,970 --> 00:11:14,649
Geoffrey Hinton:
would have been much more
277
00:11:14,649 --> 00:11:15,610
Geoffrey Hinton:
efficient and much easier to
278
00:11:15,610 --> 00:11:16,330
Geoffrey Hinton:
update.
279
00:11:16,330 --> 00:11:18,250
Geoffrey Hinton:
And I said, why not?
280
00:11:18,250 --> 00:11:21,490
Geoffrey Hinton:
And they confidentially
explained to me that, well, when
281
00:11:21,490 --> 00:11:26,090
Geoffrey Hinton:
people come and visit the
network control center, um, or
282
00:11:26,090 --> 00:11:28,090
Geoffrey Hinton:
when actually when they visit
the headquarters of British
283
00:11:28,090 --> 00:11:31,370
Geoffrey Hinton:
Telecom, um, they have to have
something to show them, like the
284
00:11:31,370 --> 00:11:34,210
Geoffrey Hinton:
politicians have to see
something, and they would always
285
00:11:34,210 --> 00:11:37,529
Geoffrey Hinton:
show them this huge wall that
displayed the state of all the
286
00:11:37,529 --> 00:11:40,629
Geoffrey Hinton:
networks, of all the switching
stations, and they were very
287
00:11:40,629 --> 00:11:41,750
Geoffrey Hinton:
impressed by that.
288
00:11:41,750 --> 00:11:43,029
Geoffrey Hinton:
And if they got rid of the huge
289
00:11:43,029 --> 00:11:45,230
Geoffrey Hinton:
wall and had just some
290
00:11:45,230 --> 00:11:46,789
Geoffrey Hinton:
workstations, they were very
291
00:11:46,789 --> 00:11:48,870
Geoffrey Hinton:
worried that network management
292
00:11:48,870 --> 00:11:49,830
Geoffrey Hinton:
would get less funds from
293
00:11:49,830 --> 00:11:50,909
Geoffrey Hinton:
British Telecom.
294
00:11:50,909 --> 00:11:53,710
Geoffrey Hinton:
So they were going to keep their
huge wall.
295
00:11:53,710 --> 00:11:56,070
Geoffrey Hinton:
I learned a lot then about
applied research.
296
00:11:57,629 --> 00:11:59,070
Geoffrey Hinton:
It's not about whether it works,
297
00:11:59,070 --> 00:12:00,350
Geoffrey Hinton:
it's whether about the company
298
00:12:00,350 --> 00:12:01,269
Geoffrey Hinton:
likes it.
299
00:12:03,429 --> 00:12:04,990
Tom Mitchell:
Fair enough, fair enough.
300
00:12:05,389 --> 00:12:11,870
Geoffrey Hinton:
Then after that, um, I went back
to sea to San Diego for six
301
00:12:11,870 --> 00:12:15,950
Geoffrey Hinton:
months, and that's when we
worked on the PDP books with
302
00:12:15,950 --> 00:12:18,350
Geoffrey Hinton:
Dave Rumelhart and McClelland.
303
00:12:18,350 --> 00:12:20,750
Geoffrey Hinton:
Um, I was one of the authors
304
00:12:20,750 --> 00:12:22,350
Geoffrey Hinton:
until almost when they were
305
00:12:22,350 --> 00:12:22,750
Geoffrey Hinton:
published.
306
00:12:22,750 --> 00:12:23,909
Geoffrey Hinton:
And the last minute I dropped
307
00:12:23,909 --> 00:12:25,309
Geoffrey Hinton:
out because at that point I
308
00:12:25,309 --> 00:12:26,629
Geoffrey Hinton:
decided Boltzmann machines were
309
00:12:26,629 --> 00:12:27,629
Geoffrey Hinton:
the future.
310
00:12:27,629 --> 00:12:30,370
Geoffrey Hinton:
Boltzmann machines was just a
much better idea than back prop,
311
00:12:30,370 --> 00:12:32,110
Geoffrey Hinton:
and back prop was a silly idea.
312
00:12:32,110 --> 00:12:34,070
Geoffrey Hinton:
Boltzmann machines were a much
better idea.
313
00:12:34,070 --> 00:12:36,470
Geoffrey Hinton:
Um, and there was no point being
an author of a book where the
314
00:12:36,470 --> 00:12:39,850
Geoffrey Hinton:
main thing was, um, back prop.
315
00:12:39,850 --> 00:12:42,210
Geoffrey Hinton:
Uh, that was a mistake.
316
00:12:42,210 --> 00:12:49,610
Geoffrey Hinton:
Um. But in nineteen eighty four,
I figured out.
317
00:12:49,610 --> 00:12:53,730
Geoffrey Hinton:
Yeah, in nineteen eighty two,
then I applied to CMU, and
318
00:12:53,730 --> 00:12:57,350
Geoffrey Hinton:
because it was a private
university, um, you could just
319
00:12:57,350 --> 00:12:59,690
Geoffrey Hinton:
they didn't have to sort of
advertise very widely.
320
00:12:59,690 --> 00:13:05,889
Geoffrey Hinton:
Um, and Scott Fahlman was sort
of my, um, host.
321
00:13:05,889 --> 00:13:07,809
Geoffrey Hinton:
He sort of interacted with him
322
00:13:07,809 --> 00:13:09,610
Geoffrey Hinton:
at many workshops, and we got
323
00:13:09,610 --> 00:13:11,330
Geoffrey Hinton:
along well, and he pushed hard
324
00:13:11,330 --> 00:13:12,629
Geoffrey Hinton:
to get for me to get them to go
325
00:13:12,629 --> 00:13:13,450
Geoffrey Hinton:
there.
326
00:13:13,450 --> 00:13:15,049
Geoffrey Hinton:
And I had a very funny
interview.
327
00:13:15,049 --> 00:13:16,610
Geoffrey Hinton:
So I went there.
328
00:13:16,610 --> 00:13:18,490
Geoffrey Hinton:
On the first day there, I gave a
329
00:13:18,490 --> 00:13:20,289
Geoffrey Hinton:
talk into computer science, and
330
00:13:20,289 --> 00:13:21,570
Geoffrey Hinton:
then Scott Farmer took me out
331
00:13:21,570 --> 00:13:24,649
Geoffrey Hinton:
for lunch at a place it might
332
00:13:24,649 --> 00:13:25,769
Geoffrey Hinton:
have been called the oh, I can't
333
00:13:25,769 --> 00:13:27,769
Geoffrey Hinton:
remember what it was called, but
334
00:13:27,769 --> 00:13:29,250
Geoffrey Hinton:
it had a motto, which is if you
335
00:13:29,250 --> 00:13:30,610
Geoffrey Hinton:
don't get sick, you got a bad
336
00:13:30,610 --> 00:13:31,450
Geoffrey Hinton:
one.
337
00:13:31,450 --> 00:13:35,049
Geoffrey Hinton:
Um, and I got terribly sick.
338
00:13:35,049 --> 00:13:36,870
Geoffrey Hinton:
And the next day I had acute
diarrhea.
339
00:13:36,870 --> 00:13:37,710
Geoffrey Hinton:
I couldn't eat anything.
340
00:13:37,710 --> 00:13:41,029
Geoffrey Hinton:
I was living on coffee and
Coca-Cola.
341
00:13:41,029 --> 00:13:48,230
Geoffrey Hinton:
Um, I gave a talk in psychology,
um, about mental imagery and my
342
00:13:48,230 --> 00:13:49,990
Geoffrey Hinton:
theory of mental imagery.
343
00:13:49,990 --> 00:13:54,350
Geoffrey Hinton:
And then there was someone at
the end who, um, asked a
344
00:13:54,350 --> 00:13:55,990
Geoffrey Hinton:
question which I didn't
understand to begin with.
345
00:13:55,990 --> 00:13:59,389
Geoffrey Hinton:
And then I realized the question
he was asking was, did I believe
346
00:13:59,389 --> 00:14:03,029
Geoffrey Hinton:
the theory of someone called
Marcel just about how you
347
00:14:03,029 --> 00:14:05,870
Geoffrey Hinton:
weren't really rotating an image
in your mind, you were just
348
00:14:05,870 --> 00:14:08,269
Geoffrey Hinton:
looking backwards and forwards
between two things.
349
00:14:08,269 --> 00:14:11,470
Geoffrey Hinton:
Um, and in my reply I said, oh,
350
00:14:11,470 --> 00:14:12,990
Geoffrey Hinton:
I see you mean that silly theory
351
00:14:12,990 --> 00:14:13,629
Geoffrey Hinton:
by Marcel.
352
00:14:13,629 --> 00:14:13,990
Geoffrey Hinton:
Marcel?
353
00:14:13,990 --> 00:14:16,830
Geoffrey Hinton:
Just where I didn't realize it
was Marcel.
354
00:14:16,830 --> 00:14:18,509
Geoffrey Hinton:
Just asking the question.
355
00:14:18,509 --> 00:14:22,269
Geoffrey Hinton:
Um, and after that, I got a
356
00:14:22,269 --> 00:14:23,409
Geoffrey Hinton:
request to go and see Nico
357
00:14:23,409 --> 00:14:24,990
Geoffrey Hinton:
Habermann.
358
00:14:24,990 --> 00:14:26,149
Geoffrey Hinton:
Now, Nico and I were always
359
00:14:26,149 --> 00:14:27,350
Geoffrey Hinton:
great friends, even though we
360
00:14:27,350 --> 00:14:28,750
Geoffrey Hinton:
were politically extremely
361
00:14:28,750 --> 00:14:29,990
Geoffrey Hinton:
different.
362
00:14:29,990 --> 00:14:33,674
Geoffrey Hinton:
I was a sort of leftie, nineteen
sixties radical with long hair
363
00:14:33,674 --> 00:14:35,399
Geoffrey Hinton:
and rather disheveled.
364
00:14:35,399 --> 00:14:39,539
Geoffrey Hinton:
Niko was a European gentleman
who was very nicely dressed,
365
00:14:39,539 --> 00:14:42,399
Geoffrey Hinton:
worked with the Defense
Department, set up an institute
366
00:14:42,399 --> 00:14:44,840
Geoffrey Hinton:
I wasn't allowed to go to
because I was a foreigner.
367
00:14:44,840 --> 00:14:48,080
Geoffrey Hinton:
Um, but we got along very well,
and I think it was because of
368
00:14:48,080 --> 00:14:49,480
Geoffrey Hinton:
our initial interview.
369
00:14:50,320 --> 00:14:54,720
Tom Mitchell:
So Niko was the department head
in computer science?
370
00:14:54,559 --> 00:14:56,639
Geoffrey Hinton:
Yes. And so in the initial
371
00:14:56,639 --> 00:14:58,639
Geoffrey Hinton:
interview, he said, so we've
372
00:14:58,639 --> 00:14:59,840
Geoffrey Hinton:
decided to offer you the
373
00:14:59,840 --> 00:15:00,759
Geoffrey Hinton:
position.
374
00:15:00,759 --> 00:15:03,639
Geoffrey Hinton:
And I said, oh, oh, there's
something you should know.
375
00:15:03,639 --> 00:15:05,240
Geoffrey Hinton:
And he said, oh what's that?
376
00:15:05,240 --> 00:15:06,379
Geoffrey Hinton:
And I said, well, I don't
377
00:15:06,379 --> 00:15:07,440
Geoffrey Hinton:
actually know any computer
378
00:15:07,440 --> 00:15:08,759
Geoffrey Hinton:
science.
379
00:15:08,759 --> 00:15:10,759
Geoffrey Hinton:
And he said, it's okay here.
380
00:15:10,759 --> 00:15:11,519
Geoffrey Hinton:
It's okay.
381
00:15:11,519 --> 00:15:13,559
Geoffrey Hinton:
We have people here who do.
382
00:15:13,559 --> 00:15:15,799
Geoffrey Hinton:
So I said, okay.
383
00:15:15,799 --> 00:15:17,039
Geoffrey Hinton:
So I said, okay.
384
00:15:17,039 --> 00:15:18,799
Geoffrey Hinton:
In that case I accept.
385
00:15:18,799 --> 00:15:20,679
Geoffrey Hinton:
And Niko said, don't you think
386
00:15:20,679 --> 00:15:22,080
Geoffrey Hinton:
perhaps we should talk about the
387
00:15:22,080 --> 00:15:23,559
Geoffrey Hinton:
salary?
388
00:15:23,559 --> 00:15:25,360
Geoffrey Hinton:
And I said, oh no, I'm not
interested in salary.
389
00:15:25,360 --> 00:15:26,360
Geoffrey Hinton:
You can pay me whatever you
like.
390
00:15:26,360 --> 00:15:28,440
Geoffrey Hinton:
I'm not doing it for the money.
391
00:15:28,440 --> 00:15:30,120
Geoffrey Hinton:
And he said, well, how does
392
00:15:30,120 --> 00:15:31,399
Geoffrey Hinton:
twenty six thousand sound to
393
00:15:31,399 --> 00:15:31,679
Geoffrey Hinton:
you?
394
00:15:31,679 --> 00:15:34,279
Geoffrey Hinton:
So if that sounds fine, um, I
395
00:15:34,279 --> 00:15:35,919
Geoffrey Hinton:
discovered I was being paid ten
396
00:15:35,919 --> 00:15:37,419
Geoffrey Hinton:
thousand than the next lowest
397
00:15:37,419 --> 00:15:38,659
Geoffrey Hinton:
paid professor.
398
00:15:38,659 --> 00:15:40,139
Geoffrey Hinton:
Ten thousand less.
399
00:15:40,139 --> 00:15:44,820
Geoffrey Hinton:
Um, but every year I got a big
pay rise, and me and Niko got
400
00:15:44,820 --> 00:15:46,820
Geoffrey Hinton:
along very well after that,
because he knew I wasn't doing
401
00:15:46,820 --> 00:15:47,620
Geoffrey Hinton:
it for the money.
402
00:15:49,980 --> 00:15:50,500
Geoffrey Hinton:
Change.
403
00:15:50,500 --> 00:15:52,340
Geoffrey Hinton:
Things have changed so much.
404
00:15:55,220 --> 00:15:56,940
Tom Mitchell:
That's. That's fantastic.
405
00:15:56,940 --> 00:15:59,480
Tom Mitchell:
Okay, so now we're up to the mid
406
00:15:59,480 --> 00:16:02,820
Tom Mitchell:
eighties when really neural nets
407
00:16:02,820 --> 00:16:04,259
Tom Mitchell:
are reborn.
408
00:16:04,259 --> 00:16:05,940
Tom Mitchell:
Is that the right word?
409
00:16:05,940 --> 00:16:06,539
Tom Mitchell:
How would you.
410
00:16:06,340 --> 00:16:08,899
Geoffrey Hinton:
Yes. We back we back
propagation.
411
00:16:08,899 --> 00:16:10,100
Geoffrey Hinton:
I mean, we didn't invent it.
412
00:16:10,100 --> 00:16:11,659
Geoffrey Hinton:
We invented by several different
413
00:16:11,659 --> 00:16:14,679
Geoffrey Hinton:
groups, but we showed that it
414
00:16:14,679 --> 00:16:15,980
Geoffrey Hinton:
really worked to learn
415
00:16:15,980 --> 00:16:17,259
Geoffrey Hinton:
representations.
416
00:16:17,259 --> 00:16:19,820
Geoffrey Hinton:
And as you know, sort of one of
the big problems in AI is how do
417
00:16:19,820 --> 00:16:22,100
Geoffrey Hinton:
you learn new representations?
418
00:16:22,100 --> 00:16:24,580
Geoffrey Hinton:
How do you avoid having to put
them all in by hand?
419
00:16:24,580 --> 00:16:27,940
Geoffrey Hinton:
Um, and my particular example,
420
00:16:27,940 --> 00:16:29,299
Geoffrey Hinton:
which was the family trees
421
00:16:29,299 --> 00:16:30,639
Geoffrey Hinton:
example, where you take all the
422
00:16:30,639 --> 00:16:31,840
Geoffrey Hinton:
information in some family
423
00:16:31,840 --> 00:16:33,879
Geoffrey Hinton:
trees, you convert it into
424
00:16:33,879 --> 00:16:36,200
Geoffrey Hinton:
triples of symbols like John has
425
00:16:36,200 --> 00:16:37,639
Geoffrey Hinton:
Father Mary.
426
00:16:37,639 --> 00:16:39,840
Geoffrey Hinton:
Um, and then you train a neural
427
00:16:39,840 --> 00:16:41,519
Geoffrey Hinton:
net to predict the last term in
428
00:16:41,519 --> 00:16:42,639
Geoffrey Hinton:
a triple given the first two
429
00:16:42,639 --> 00:16:43,639
Geoffrey Hinton:
terms.
430
00:16:43,639 --> 00:16:45,600
Geoffrey Hinton:
So it's just like the big
language models.
431
00:16:45,600 --> 00:16:48,200
Geoffrey Hinton:
You're predicting the next word
given the context.
432
00:16:48,200 --> 00:16:50,120
Geoffrey Hinton:
It's just much simpler.
433
00:16:50,120 --> 00:16:53,100
Geoffrey Hinton:
I had one hundred and twelve
434
00:16:53,100 --> 00:16:55,000
Geoffrey Hinton:
total examples, of which one
435
00:16:55,000 --> 00:16:56,440
Geoffrey Hinton:
hundred and four were training
436
00:16:56,440 --> 00:16:57,720
Geoffrey Hinton:
examples and eight were test
437
00:16:57,720 --> 00:16:59,759
Geoffrey Hinton:
examples, which is a bit less
438
00:16:59,759 --> 00:17:01,080
Geoffrey Hinton:
than the trillion examples they
439
00:17:01,080 --> 00:17:02,279
Geoffrey Hinton:
have nowadays.
440
00:17:02,279 --> 00:17:04,319
Geoffrey Hinton:
Um, but it was the same idea.
441
00:17:04,319 --> 00:17:08,200
Geoffrey Hinton:
You convert a symbol into a
feature vector.
442
00:17:08,200 --> 00:17:11,200
Geoffrey Hinton:
You then have the feature
vectors of the context interact
443
00:17:11,200 --> 00:17:14,000
Geoffrey Hinton:
um, via a hidden layer.
444
00:17:14,000 --> 00:17:15,640
Geoffrey Hinton:
They then predict the features
445
00:17:15,640 --> 00:17:17,839
Geoffrey Hinton:
of the next symbol, and from
446
00:17:17,839 --> 00:17:18,920
Geoffrey Hinton:
those features you guess what
447
00:17:18,920 --> 00:17:20,440
Geoffrey Hinton:
the next symbol should be, and
448
00:17:20,440 --> 00:17:21,400
Geoffrey Hinton:
you try and maximize the
449
00:17:21,400 --> 00:17:22,640
Geoffrey Hinton:
probability of predicting the
450
00:17:22,640 --> 00:17:23,839
Geoffrey Hinton:
next symbol.
451
00:17:23,839 --> 00:17:28,039
Geoffrey Hinton:
And you then backpropagate
through the feature interactions
452
00:17:28,039 --> 00:17:30,980
Geoffrey Hinton:
and through the process that
converts a symbol into features.
453
00:17:30,980 --> 00:17:34,460
Geoffrey Hinton:
And that way you learn, um,
feature vectors to represent the
454
00:17:34,460 --> 00:17:38,460
Geoffrey Hinton:
symbols and how these vectors
should interact to predict the
455
00:17:38,460 --> 00:17:40,059
Geoffrey Hinton:
features of the next symbol.
456
00:17:40,059 --> 00:17:43,380
Geoffrey Hinton:
And that's what these big
language models do, except it's
457
00:17:43,380 --> 00:17:44,339
Geoffrey Hinton:
a bit more complicated.
458
00:17:44,339 --> 00:17:46,819
Geoffrey Hinton:
The feature interactions are
much more complicated.
459
00:17:46,819 --> 00:17:47,859
Geoffrey Hinton:
They have many more layers of
460
00:17:47,859 --> 00:17:48,940
Geoffrey Hinton:
interaction, so they can
461
00:17:48,940 --> 00:17:51,140
Geoffrey Hinton:
disambiguate ambiguous symbols
462
00:17:51,140 --> 00:17:52,880
Geoffrey Hinton:
and get refine the shade of
463
00:17:52,880 --> 00:17:53,819
Geoffrey Hinton:
meaning of things where the
464
00:17:53,819 --> 00:17:54,740
Geoffrey Hinton:
meaning depends a lot on the
465
00:17:54,740 --> 00:17:55,859
Geoffrey Hinton:
context.
466
00:17:55,859 --> 00:18:01,660
Geoffrey Hinton:
Um, but it's basically an
extremely simple version of the
467
00:18:01,660 --> 00:18:02,819
Geoffrey Hinton:
current large language models.
468
00:18:02,819 --> 00:18:03,900
Geoffrey Hinton:
I called it a tiny language
469
00:18:03,900 --> 00:18:06,140
Geoffrey Hinton:
model, and that convinced the
470
00:18:06,140 --> 00:18:08,140
Geoffrey Hinton:
editors of nature that we really
471
00:18:08,140 --> 00:18:08,980
Geoffrey Hinton:
could learn interesting
472
00:18:08,980 --> 00:18:11,380
Geoffrey Hinton:
representations, because the
473
00:18:11,380 --> 00:18:12,500
Geoffrey Hinton:
vectors I learned for the
474
00:18:12,500 --> 00:18:14,099
Geoffrey Hinton:
symbols, which were people and
475
00:18:14,099 --> 00:18:15,740
Geoffrey Hinton:
relationships, they had six
476
00:18:15,740 --> 00:18:17,019
Geoffrey Hinton:
components.
477
00:18:17,019 --> 00:18:21,140
Geoffrey Hinton:
And if you used weight decay,
you could interpret what all
478
00:18:21,140 --> 00:18:22,220
Geoffrey Hinton:
those components were.
479
00:18:22,220 --> 00:18:24,579
Geoffrey Hinton:
And they were sensible semantic
features.
480
00:18:24,579 --> 00:18:27,400
Geoffrey Hinton:
They were the nationality of the
person and the generation of the
481
00:18:27,400 --> 00:18:30,960
Geoffrey Hinton:
person, and which branch of the
family tree they were in.
482
00:18:30,960 --> 00:18:33,839
Geoffrey Hinton:
And so it would learn things
483
00:18:33,839 --> 00:18:36,980
Geoffrey Hinton:
like the relationship uncle
484
00:18:36,980 --> 00:18:39,240
Geoffrey Hinton:
requires the output person to be
485
00:18:39,240 --> 00:18:40,640
Geoffrey Hinton:
one generation older than the
486
00:18:40,640 --> 00:18:41,839
Geoffrey Hinton:
input person.
487
00:18:42,920 --> 00:18:45,640
Geoffrey Hinton:
And so it would have generations
for people.
488
00:18:45,640 --> 00:18:46,799
Geoffrey Hinton:
And if the input person was a
489
00:18:46,799 --> 00:18:48,920
Geoffrey Hinton:
generation two, it would predict
490
00:18:48,920 --> 00:18:50,279
Geoffrey Hinton:
that the output person would be
491
00:18:50,279 --> 00:18:51,759
Geoffrey Hinton:
generation one.
492
00:18:51,759 --> 00:18:54,160
Geoffrey Hinton:
Um, so it was actually learning
493
00:18:54,160 --> 00:18:55,640
Geoffrey Hinton:
a whole bunch of little rules
494
00:18:55,640 --> 00:18:57,839
Geoffrey Hinton:
just probabilistically.
495
00:18:57,839 --> 00:18:59,920
Geoffrey Hinton:
And the people interested in
rule based induction got
496
00:18:59,920 --> 00:19:02,319
Geoffrey Hinton:
interested in it because they
said, oh, we can do that too.
497
00:19:02,319 --> 00:19:02,799
Geoffrey Hinton:
And it's true.
498
00:19:02,799 --> 00:19:04,319
Geoffrey Hinton:
They could do that too, with
499
00:19:04,319 --> 00:19:04,960
Geoffrey Hinton:
rules that weren't
500
00:19:04,960 --> 00:19:06,319
Geoffrey Hinton:
probabilistic.
501
00:19:06,319 --> 00:19:08,079
Geoffrey Hinton:
The point about neural nets is
502
00:19:08,079 --> 00:19:09,440
Geoffrey Hinton:
they can mimic something that
503
00:19:09,440 --> 00:19:11,440
Geoffrey Hinton:
learns discrete rules, but they
504
00:19:11,440 --> 00:19:11,799
Geoffrey Hinton:
can.
505
00:19:11,799 --> 00:19:14,640
Geoffrey Hinton:
They're also perfectly happy if
the rules are just usually true.
506
00:19:14,640 --> 00:19:17,359
Geoffrey Hinton:
And they use the preponderance
of the evidence then, which is
507
00:19:17,359 --> 00:19:20,960
Geoffrey Hinton:
much harder to do in, um, logic.
508
00:19:20,960 --> 00:19:27,059
Geoffrey Hinton:
And so that it was that example
which, curiously, was a little
509
00:19:27,059 --> 00:19:30,460
Geoffrey Hinton:
language model, um, that
convinced the editors of nature
510
00:19:30,460 --> 00:19:31,779
Geoffrey Hinton:
to publish a paper.
511
00:19:31,779 --> 00:19:33,619
Geoffrey Hinton:
I know because I talked to them
later.
512
00:19:33,619 --> 00:19:35,700
Geoffrey Hinton:
The referees, I talked to the
referee, one of the referees
513
00:19:35,700 --> 00:19:38,700
Geoffrey Hinton:
later, and he said, yeah, it was
that example that did it.
514
00:19:38,700 --> 00:19:40,819
Geoffrey Hinton:
And then we were all very
excited.
515
00:19:40,819 --> 00:19:44,140
Geoffrey Hinton:
We thought, we can solve
everything.
516
00:19:44,140 --> 00:19:45,140
Geoffrey Hinton:
You just have to give it a lot
517
00:19:45,140 --> 00:19:46,460
Geoffrey Hinton:
of training data and run
518
00:19:46,460 --> 00:19:48,180
Geoffrey Hinton:
backprop, and it'll learn all
519
00:19:48,180 --> 00:19:49,940
Geoffrey Hinton:
the representations you need and
520
00:19:49,940 --> 00:19:50,859
Geoffrey Hinton:
it'll learn to do parallel
521
00:19:50,859 --> 00:19:51,900
Geoffrey Hinton:
computation.
522
00:19:51,900 --> 00:19:53,779
Geoffrey Hinton:
Because at that time people were
very interested in parallel
523
00:19:53,779 --> 00:19:56,940
Geoffrey Hinton:
computation, but it was quite
hard to program.
524
00:19:56,940 --> 00:19:59,740
Geoffrey Hinton:
And the idea was, well, this
will have all these neurons
525
00:19:59,740 --> 00:20:02,420
Geoffrey Hinton:
inside and they'll all be
operating in parallel and it'll
526
00:20:02,420 --> 00:20:05,700
Geoffrey Hinton:
figure out how to use them so
there aren't any problems.
527
00:20:05,700 --> 00:20:07,900
Geoffrey Hinton:
At that point, people were very
interested in races and things
528
00:20:07,900 --> 00:20:10,140
Geoffrey Hinton:
like that, and you didn't have
to worry about any of that.
529
00:20:10,140 --> 00:20:11,299
Geoffrey Hinton:
It was all synchronous and you
530
00:20:11,299 --> 00:20:12,660
Geoffrey Hinton:
just they just learned what to
531
00:20:12,660 --> 00:20:13,259
Geoffrey Hinton:
do.
532
00:20:13,259 --> 00:20:18,460
Geoffrey Hinton:
Um, so we thought we'd solved
everything and little did we
533
00:20:18,460 --> 00:20:19,539
Geoffrey Hinton:
know we had.
534
00:20:19,539 --> 00:20:22,099
Geoffrey Hinton:
It's just we needed more data
and more compute.
535
00:20:24,440 --> 00:20:26,440
Tom Mitchell:
So then there's the long period
536
00:20:26,440 --> 00:20:27,880
Tom Mitchell:
of waiting for more data and
537
00:20:27,880 --> 00:20:28,960
Tom Mitchell:
more compute.
538
00:20:29,559 --> 00:20:34,359
Geoffrey Hinton:
And yeah, not realizing that
that was the main problem.
539
00:20:34,359 --> 00:20:36,440
Geoffrey Hinton:
Obviously, with other little
problems, there were more
540
00:20:36,440 --> 00:20:39,859
Geoffrey Hinton:
sensible kinds of neurons to use
than more sensible ways to
541
00:20:39,859 --> 00:20:41,440
Geoffrey Hinton:
regularize it and all that.
542
00:20:41,440 --> 00:20:43,000
Geoffrey Hinton:
Um, and things like transformers
543
00:20:43,000 --> 00:20:44,039
Geoffrey Hinton:
had to be invented to make it
544
00:20:44,039 --> 00:20:45,079
Geoffrey Hinton:
really efficient.
545
00:20:45,079 --> 00:20:50,359
Geoffrey Hinton:
Um, but basically backprop was
the way to do it.
546
00:20:50,359 --> 00:20:52,680
Geoffrey Hinton:
And you couldn't convince
547
00:20:52,680 --> 00:20:53,880
Geoffrey Hinton:
anybody when computers were
548
00:20:53,880 --> 00:20:54,559
Geoffrey Hinton:
slow.
549
00:20:54,559 --> 00:20:56,519
Geoffrey Hinton:
It will work for little
problems.
550
00:20:56,519 --> 00:20:57,759
Geoffrey Hinton:
It will work for slightly bigger
551
00:20:57,759 --> 00:20:59,480
Geoffrey Hinton:
problems, like a few years
552
00:20:59,480 --> 00:20:59,960
Geoffrey Hinton:
later.
553
00:20:59,960 --> 00:21:03,680
Geoffrey Hinton:
Yang got it working for for
mNIST, recognizing digits.
554
00:21:03,680 --> 00:21:05,220
Geoffrey Hinton:
But all the vision people said,
555
00:21:05,220 --> 00:21:06,240
Geoffrey Hinton:
you know, that's not real
556
00:21:06,240 --> 00:21:07,279
Geoffrey Hinton:
vision.
557
00:21:07,279 --> 00:21:10,680
Geoffrey Hinton:
Um, you're never going to do it
with real images that are high
558
00:21:10,680 --> 00:21:12,279
Geoffrey Hinton:
resolution on the web.
559
00:21:13,400 --> 00:21:16,500
Geoffrey Hinton:
And so it wasn't until about
twenty twelve that they had to
560
00:21:16,500 --> 00:21:18,000
Geoffrey Hinton:
eat their words.
561
00:21:19,680 --> 00:21:20,240
Tom Mitchell:
That's right.
562
00:21:20,240 --> 00:21:21,700
Tom Mitchell:
That was the year when.
563
00:21:21,700 --> 00:21:22,980
Tom Mitchell:
Well, you tell this story.
564
00:21:22,980 --> 00:21:24,900
Tom Mitchell:
You were the first person.
565
00:21:25,740 --> 00:21:29,059
Geoffrey Hinton:
Uh, well, I was the advisor of
the first two people.
566
00:21:29,059 --> 00:21:31,099
Geoffrey Hinton:
Now, it's not quite fair,
because Jan had already
567
00:21:31,099 --> 00:21:34,460
Geoffrey Hinton:
basically shown that they worked
for real images.
568
00:21:34,460 --> 00:21:36,700
Geoffrey Hinton:
Um, and Jan realized when Feifei
569
00:21:36,700 --> 00:21:37,660
Geoffrey Hinton:
came up with the ImageNet
570
00:21:37,660 --> 00:21:38,619
Geoffrey Hinton:
dataset.
571
00:21:38,619 --> 00:21:39,819
Geoffrey Hinton:
Jan realized they could win that
572
00:21:39,819 --> 00:21:41,619
Geoffrey Hinton:
competition, and he tried to get
573
00:21:41,619 --> 00:21:43,099
Geoffrey Hinton:
graduate students and postdocs
574
00:21:43,099 --> 00:21:44,579
Geoffrey Hinton:
in his lab to do it, and they
575
00:21:44,579 --> 00:21:45,819
Geoffrey Hinton:
all declined.
576
00:21:47,420 --> 00:21:54,059
Geoffrey Hinton:
Um, and Ilya, Ilya Sutskever
realized that, um, backprop
577
00:21:54,059 --> 00:21:56,140
Geoffrey Hinton:
would just kill ImageNet.
578
00:21:56,140 --> 00:22:00,380
Geoffrey Hinton:
Um, and he wanted Alex to work
on it.
579
00:22:00,380 --> 00:22:02,259
Geoffrey Hinton:
And I didn't really want to work
on it.
580
00:22:02,259 --> 00:22:03,500
Geoffrey Hinton:
Um, Alex had already been
581
00:22:03,500 --> 00:22:04,819
Geoffrey Hinton:
working on small images and
582
00:22:04,819 --> 00:22:06,079
Geoffrey Hinton:
recognizing small images in
583
00:22:06,079 --> 00:22:07,019
Geoffrey Hinton:
c410.
584
00:22:07,019 --> 00:22:09,660
Geoffrey Hinton:
Um, and you pre-processed
585
00:22:09,660 --> 00:22:10,740
Geoffrey Hinton:
everything for Alex to make it
586
00:22:10,740 --> 00:22:11,660
Geoffrey Hinton:
easy.
587
00:22:11,660 --> 00:22:14,119
Geoffrey Hinton:
And I bought Alex two Nvidia
588
00:22:14,119 --> 00:22:16,460
Geoffrey Hinton:
GPUs to have in his bedroom at
589
00:22:16,460 --> 00:22:17,259
Geoffrey Hinton:
home.
590
00:22:17,259 --> 00:22:22,359
Geoffrey Hinton:
Um, And Alex, then get on with
get on with it.
591
00:22:22,359 --> 00:22:24,160
Geoffrey Hinton:
And he was an absolutely wizard
programmer.
592
00:22:24,160 --> 00:22:27,000
Geoffrey Hinton:
He wrote amazing code on
593
00:22:27,000 --> 00:22:29,000
Geoffrey Hinton:
multiple GPUs to do convolution
594
00:22:29,000 --> 00:22:29,839
Geoffrey Hinton:
really efficiently.
595
00:22:29,839 --> 00:22:32,359
Geoffrey Hinton:
Much better code than anybody
else had ever written.
596
00:22:32,359 --> 00:22:40,279
Geoffrey Hinton:
Um, I believe and so it's a
combination of Ilya realizing we
597
00:22:40,279 --> 00:22:43,640
Geoffrey Hinton:
really had to do this, and Ilya
was involved in the design of
598
00:22:43,640 --> 00:22:48,220
Geoffrey Hinton:
the net and so on, but Alex's
programming skills and then I
599
00:22:48,220 --> 00:22:51,920
Geoffrey Hinton:
added a few ideas, like use
rectified linear units instead
600
00:22:51,920 --> 00:22:56,480
Geoffrey Hinton:
of sigmoid units and use little
patches of the images.
601
00:22:56,480 --> 00:22:57,440
Geoffrey Hinton:
I mean big patches of the
602
00:22:57,440 --> 00:22:59,079
Geoffrey Hinton:
images, so you can translate
603
00:22:59,079 --> 00:23:00,559
Geoffrey Hinton:
things around a bit to get some
604
00:23:00,559 --> 00:23:02,480
Geoffrey Hinton:
translation invariance, as well
605
00:23:02,480 --> 00:23:05,799
Geoffrey Hinton:
as using convolution, um, and
606
00:23:05,799 --> 00:23:06,640
Geoffrey Hinton:
use dropout.
607
00:23:06,640 --> 00:23:08,920
Geoffrey Hinton:
So that was one of the first
applications of dropout.
608
00:23:08,920 --> 00:23:11,119
Geoffrey Hinton:
And that helped about one
percent.
609
00:23:11,119 --> 00:23:13,200
Geoffrey Hinton:
It really helped.
610
00:23:13,200 --> 00:23:15,880
Geoffrey Hinton:
And then we beat the best vision
systems.
611
00:23:15,880 --> 00:23:19,069
Geoffrey Hinton:
The best vision systems were
sort of plateauing at twenty
612
00:23:19,069 --> 00:23:21,049
Geoffrey Hinton:
five percent errors.
613
00:23:21,049 --> 00:23:23,410
Geoffrey Hinton:
That's errors for getting the
right answer in the top in your
614
00:23:23,410 --> 00:23:24,769
Geoffrey Hinton:
top five bets.
615
00:23:24,769 --> 00:23:30,089
Geoffrey Hinton:
Um, and we got like fifteen
percent, fifteen or sixteen
616
00:23:30,089 --> 00:23:31,650
Geoffrey Hinton:
depending on how you count it.
617
00:23:31,650 --> 00:23:33,809
Geoffrey Hinton:
So we got almost half the error
rate.
618
00:23:35,049 --> 00:23:37,369
Geoffrey Hinton:
And what happened then was what
619
00:23:37,369 --> 00:23:38,529
Geoffrey Hinton:
ought to happen in science but
620
00:23:38,529 --> 00:23:40,569
Geoffrey Hinton:
seldom does.
621
00:23:40,569 --> 00:23:45,109
Geoffrey Hinton:
So our most vigorous opponents,
like Jitendra Malik and
622
00:23:45,109 --> 00:23:48,609
Geoffrey Hinton:
Zisserman Andrew Zisserman,
looked at these results and
623
00:23:48,609 --> 00:23:50,970
Geoffrey Hinton:
said, okay, you were right.
624
00:23:50,970 --> 00:23:53,049
Geoffrey Hinton:
That never happens in science.
625
00:23:53,049 --> 00:23:56,009
Geoffrey Hinton:
And slightly irritating the
Andrew Zisserman then switched
626
00:23:56,009 --> 00:23:57,650
Geoffrey Hinton:
to doing this.
627
00:23:57,650 --> 00:24:00,849
Geoffrey Hinton:
He had some very good postdocs
or students working with him.
628
00:24:00,849 --> 00:24:07,089
Geoffrey Hinton:
Simonyan um, and um, after about
a year, they were making better
629
00:24:07,089 --> 00:24:08,890
Geoffrey Hinton:
networks than us.
630
00:24:08,890 --> 00:24:11,049
Geoffrey Hinton:
But that was really the.
631
00:24:13,009 --> 00:24:14,289
Geoffrey Hinton:
As far as the general public was
632
00:24:14,289 --> 00:24:15,710
Geoffrey Hinton:
concerned, that was the start of
633
00:24:15,710 --> 00:24:17,549
Geoffrey Hinton:
this big swing towards deep
634
00:24:17,549 --> 00:24:19,390
Geoffrey Hinton:
learning in twenty twelve when
635
00:24:19,390 --> 00:24:20,589
Geoffrey Hinton:
we really nailed computer
636
00:24:20,589 --> 00:24:21,029
Geoffrey Hinton:
vision.
637
00:24:21,029 --> 00:24:22,309
Geoffrey Hinton:
But it actually happened before
that.
638
00:24:22,309 --> 00:24:25,509
Geoffrey Hinton:
It happened in two thousand and
nine when we showed how you
639
00:24:25,509 --> 00:24:28,509
Geoffrey Hinton:
could do speech recognition, or
rather the acoustic modeling
640
00:24:28,509 --> 00:24:29,950
Geoffrey Hinton:
part of speech recognition.
641
00:24:29,950 --> 00:24:31,109
Geoffrey Hinton:
We showed how you could do that
642
00:24:31,109 --> 00:24:32,269
Geoffrey Hinton:
a bit better than the best
643
00:24:32,269 --> 00:24:33,750
Geoffrey Hinton:
technology.
644
00:24:33,750 --> 00:24:36,829
Geoffrey Hinton:
And that influenced all the big
speech groups, the big speech
645
00:24:36,829 --> 00:24:42,829
Geoffrey Hinton:
groups that IBM and Microsoft,
um, and somewhere else.
646
00:24:42,829 --> 00:24:43,150
Geoffrey Hinton:
Google.
647
00:24:43,150 --> 00:24:43,750
Geoffrey Hinton:
Yes.
648
00:24:43,750 --> 00:24:48,190
Geoffrey Hinton:
Um, all switched to doing neural
nets for acoustic modeling.
649
00:24:48,190 --> 00:24:51,069
Geoffrey Hinton:
And so by twenty ten, it was
650
00:24:51,069 --> 00:24:52,230
Geoffrey Hinton:
clear that neural nets were the
651
00:24:52,230 --> 00:24:53,150
Geoffrey Hinton:
right way to do acoustic
652
00:24:53,150 --> 00:24:54,230
Geoffrey Hinton:
modeling.
653
00:24:54,230 --> 00:24:56,230
Geoffrey Hinton:
And we had lots of people
onside.
654
00:24:56,230 --> 00:24:58,109
Geoffrey Hinton:
Um, and but in twenty twelve, it
655
00:24:58,109 --> 00:24:58,930
Geoffrey Hinton:
actually came out for the
656
00:24:58,930 --> 00:25:00,210
Geoffrey Hinton:
Android, and suddenly the
657
00:25:00,210 --> 00:25:02,109
Geoffrey Hinton:
Android caught up with Siri in
658
00:25:02,109 --> 00:25:04,069
Geoffrey Hinton:
speech recognition.
659
00:25:04,069 --> 00:25:07,430
Geoffrey Hinton:
So really we demonstrated it for
speech before that, but that
660
00:25:07,430 --> 00:25:09,390
Geoffrey Hinton:
didn't make a big impact.
661
00:25:09,390 --> 00:25:11,710
Geoffrey Hinton:
The reason it worked for speech
was they had a big data set.
662
00:25:11,710 --> 00:25:13,849
Geoffrey Hinton:
They had millions of examples.
663
00:25:13,849 --> 00:25:14,730
Geoffrey Hinton:
They were one of the areas.
664
00:25:14,730 --> 00:25:16,069
Geoffrey Hinton:
Unlike vision, they had big data
665
00:25:16,069 --> 00:25:17,369
Geoffrey Hinton:
sets because of the DARPA speech
666
00:25:17,369 --> 00:25:18,450
Geoffrey Hinton:
project.
667
00:25:18,450 --> 00:25:21,930
Geoffrey Hinton:
Um, because they really wanted
to be able to benchmark systems.
668
00:25:21,930 --> 00:25:25,089
Geoffrey Hinton:
Um, also, speech is easier than
vision.
669
00:25:25,089 --> 00:25:27,690
Geoffrey Hinton:
Speech is just vision with
either one or two pixels.
670
00:25:27,690 --> 00:25:29,809
Geoffrey Hinton:
It's just they change rather
fast.
671
00:25:29,809 --> 00:25:32,930
Geoffrey Hinton:
Um, and.
672
00:25:35,130 --> 00:25:39,250
Geoffrey Hinton:
So we demonstrated for speech
when we did it for vision.
673
00:25:39,250 --> 00:25:42,410
Geoffrey Hinton:
The big companies already knew
it worked for speech and they
674
00:25:42,410 --> 00:25:44,089
Geoffrey Hinton:
saw it work for vision.
675
00:25:44,089 --> 00:25:46,890
Geoffrey Hinton:
And so they realized it was sort
of universal.
676
00:25:46,890 --> 00:25:51,450
Geoffrey Hinton:
Um, it wasn't just a specific
trick for a specific domain.
677
00:25:51,450 --> 00:25:54,250
Geoffrey Hinton:
It will work for perception in
general.
678
00:25:54,250 --> 00:25:55,210
Geoffrey Hinton:
They didn't realize at that
679
00:25:55,210 --> 00:25:55,930
Geoffrey Hinton:
point it would work for
680
00:25:55,930 --> 00:25:56,410
Geoffrey Hinton:
language.
681
00:25:56,410 --> 00:25:59,529
Geoffrey Hinton:
And nor really did we, even
though our very first impressive
682
00:25:59,529 --> 00:26:01,130
Geoffrey Hinton:
example was for language.
683
00:26:01,130 --> 00:26:01,849
Geoffrey Hinton:
Um.
684
00:26:04,329 --> 00:26:04,690
Geoffrey Hinton:
Yeah.
685
00:26:04,690 --> 00:26:10,849
Geoffrey Hinton:
So in twenty twelve, there was
this big swing to neural
686
00:26:10,849 --> 00:26:17,750
Geoffrey Hinton:
networks And that's when Jensen
at Nvidia realized he finally
687
00:26:17,750 --> 00:26:22,190
Geoffrey Hinton:
realized those Nvidia boards
weren't just for gaming.
688
00:26:22,190 --> 00:26:25,349
Geoffrey Hinton:
They were supercomputers for
doing machine learning.
689
00:26:25,349 --> 00:26:29,750
Geoffrey Hinton:
Now, I actually gave a talk at
NIPS in two thousand and nine
690
00:26:29,750 --> 00:26:33,430
Geoffrey Hinton:
when I told a thousand people
this was about speech, I told a
691
00:26:33,430 --> 00:26:36,589
Geoffrey Hinton:
thousand people, if you want to
do machine learning now, you
692
00:26:36,589 --> 00:26:39,150
Geoffrey Hinton:
have to buy Nvidia GPUs.
693
00:26:39,150 --> 00:26:42,269
Geoffrey Hinton:
Nvidia GPUs will make your
program go about thirty times as
694
00:26:42,269 --> 00:26:47,789
Geoffrey Hinton:
fast because they're relatively
easy to utilize parallelism.
695
00:26:47,789 --> 00:26:49,990
Geoffrey Hinton:
They're just right for neural
nets.
696
00:26:49,990 --> 00:26:52,670
Geoffrey Hinton:
It was Rick Zelinsky who was a
student of mine at CMU, who told
697
00:26:52,670 --> 00:26:57,549
Geoffrey Hinton:
me that in about two thousand
and six, um, and it was true.
698
00:26:58,990 --> 00:27:03,210
Geoffrey Hinton:
And, um, I sent mail to Nvidia
699
00:27:03,210 --> 00:27:04,230
Geoffrey Hinton:
saying, how about giving me a
700
00:27:04,230 --> 00:27:04,589
Geoffrey Hinton:
free one?
701
00:27:04,589 --> 00:27:06,690
Geoffrey Hinton:
Because I told a thousand
machine learning researchers to
702
00:27:06,690 --> 00:27:08,430
Geoffrey Hinton:
buy your boards.
703
00:27:08,430 --> 00:27:10,450
Geoffrey Hinton:
And they declined.
704
00:27:10,450 --> 00:27:15,890
Geoffrey Hinton:
Um. Years later, Jensen came to
Toronto and gave a talk and
705
00:27:15,890 --> 00:27:19,490
Geoffrey Hinton:
mentioned how Toronto, you know,
was the place where they
706
00:27:19,490 --> 00:27:24,250
Geoffrey Hinton:
convinced him that Nvidia GPUs
were good for AI.
707
00:27:24,250 --> 00:27:26,589
Geoffrey Hinton:
Um, and that it all happened in
708
00:27:26,589 --> 00:27:28,250
Geoffrey Hinton:
twenty twelve, and I couldn't
709
00:27:28,250 --> 00:27:28,609
Geoffrey Hinton:
resist it.
710
00:27:28,609 --> 00:27:29,089
Geoffrey Hinton:
At the end.
711
00:27:29,089 --> 00:27:30,130
Geoffrey Hinton:
I said, well, I told you in two
712
00:27:30,130 --> 00:27:31,730
Geoffrey Hinton:
thousand and nine that you
713
00:27:31,730 --> 00:27:32,609
Geoffrey Hinton:
ignored me.
714
00:27:33,650 --> 00:27:38,250
Geoffrey Hinton:
And what he should have said
was, well, you're very silly.
715
00:27:38,250 --> 00:27:40,650
Geoffrey Hinton:
You should have bought stock in
two thousand and nine.
716
00:27:42,569 --> 00:27:44,569
Geoffrey Hinton:
If I'd done that, I'd be a
billionaire.
717
00:27:44,569 --> 00:27:49,289
Geoffrey Hinton:
Um, but, um, instead, he gave
me.
718
00:27:49,289 --> 00:27:52,170
Geoffrey Hinton:
He opened his briefcase and gave
me their very special, very
719
00:27:52,170 --> 00:27:55,730
Geoffrey Hinton:
latest GPU, of which they'd only
made a few that had twice as
720
00:27:55,730 --> 00:27:58,450
Geoffrey Hinton:
much memory as any other GPU.
721
00:27:58,450 --> 00:28:00,410
Geoffrey Hinton:
So that was a nice move by
Jensen.
722
00:28:00,809 --> 00:28:03,650
Tom Mitchell:
That's a great story too.
723
00:28:03,650 --> 00:28:09,390
Tom Mitchell:
So then in the twenty tens,
things really just kind of rapid
724
00:28:09,390 --> 00:28:11,710
Tom Mitchell:
fire started taking off.
725
00:28:11,710 --> 00:28:12,990
Tom Mitchell:
Take us through that.
726
00:28:14,309 --> 00:28:16,549
Geoffrey Hinton:
So speech worked.
727
00:28:16,549 --> 00:28:20,950
Geoffrey Hinton:
Um, I we got a good
collaboration between the
728
00:28:20,950 --> 00:28:24,549
Geoffrey Hinton:
research groups at IBM and
Google and, um.
729
00:28:29,589 --> 00:28:31,990
Geoffrey Hinton:
Toronto and Microsoft.
730
00:28:31,990 --> 00:28:33,069
Geoffrey Hinton:
Yeah.
731
00:28:33,069 --> 00:28:35,470
Geoffrey Hinton:
Um, we actually published a
joint paper, which is sort of
732
00:28:35,470 --> 00:28:39,750
Geoffrey Hinton:
quite rare in this stuff about
the sort of new view of how to
733
00:28:39,750 --> 00:28:41,950
Geoffrey Hinton:
do acoustic modeling.
734
00:28:41,950 --> 00:28:51,470
Geoffrey Hinton:
Um. And then we did vision, and
then, um, I started getting lots
735
00:28:51,470 --> 00:28:54,670
Geoffrey Hinton:
of requests from big companies
who wanted to.
736
00:28:54,670 --> 00:29:01,029
Geoffrey Hinton:
By me or by me and Alex and Ilya
or fund our company or get us to
737
00:29:01,029 --> 00:29:02,509
Geoffrey Hinton:
come work for them.
738
00:29:02,509 --> 00:29:06,450
Geoffrey Hinton:
Um, and I realized this stuff
was probably valuable.
739
00:29:06,450 --> 00:29:08,930
Geoffrey Hinton:
We had no idea how much it was
worth.
740
00:29:08,930 --> 00:29:11,529
Geoffrey Hinton:
Um, so Craig Butler, who was the
741
00:29:11,529 --> 00:29:12,450
Geoffrey Hinton:
chair of the Department of
742
00:29:12,450 --> 00:29:14,170
Geoffrey Hinton:
Computer Science, was an expert
743
00:29:14,170 --> 00:29:15,289
Geoffrey Hinton:
on auctions.
744
00:29:15,289 --> 00:29:17,250
Geoffrey Hinton:
And he said, you know, you
745
00:29:17,250 --> 00:29:18,970
Geoffrey Hinton:
should actually, since you have
746
00:29:18,970 --> 00:29:19,890
Geoffrey Hinton:
no idea what it's worth, but
747
00:29:19,890 --> 00:29:20,970
Geoffrey Hinton:
there's people, many people
748
00:29:20,970 --> 00:29:22,009
Geoffrey Hinton:
interested, you should set up an
749
00:29:22,009 --> 00:29:22,650
Geoffrey Hinton:
auction.
750
00:29:23,809 --> 00:29:29,450
Geoffrey Hinton:
So at Lake Tahoe, which seemed
like the appropriate place, um,
751
00:29:29,450 --> 00:29:32,609
Geoffrey Hinton:
in a casino, um, a casino hotel.
752
00:29:32,609 --> 00:29:38,369
Geoffrey Hinton:
In twenty twelve, Alex and I set
up a little company for the sole
753
00:29:38,369 --> 00:29:40,809
Geoffrey Hinton:
function of doing an aqua hire.
754
00:29:40,809 --> 00:29:44,609
Geoffrey Hinton:
And there was an auction
between, um, Microsoft and
755
00:29:44,609 --> 00:29:50,450
Geoffrey Hinton:
Google and DeepMind and Baidu.
756
00:29:50,450 --> 00:29:52,809
Geoffrey Hinton:
Um, DeepMind dropped out fairly
early.
757
00:29:52,809 --> 00:29:56,650
Geoffrey Hinton:
Um, and on the ground floor,
758
00:29:56,650 --> 00:29:57,809
Geoffrey Hinton:
they had all these people at
759
00:29:57,809 --> 00:29:59,490
Geoffrey Hinton:
slot machines with cigarettes
760
00:29:59,490 --> 00:30:00,369
Geoffrey Hinton:
hanging out the corner of their
761
00:30:00,369 --> 00:30:01,170
Geoffrey Hinton:
mouth, just pulling these
762
00:30:01,170 --> 00:30:02,009
Geoffrey Hinton:
levers.
763
00:30:02,009 --> 00:30:04,170
Geoffrey Hinton:
And every so often they made
like a thousand dollars and
764
00:30:04,170 --> 00:30:07,589
Geoffrey Hinton:
lights would flash, and we were
upstairs having an auction where
765
00:30:07,589 --> 00:30:09,630
Geoffrey Hinton:
you had to raise by a million.
766
00:30:09,630 --> 00:30:11,789
Geoffrey Hinton:
Um, that was fun.
767
00:30:11,789 --> 00:30:13,670
Geoffrey Hinton:
And the auction went on for
quite a long time.
768
00:30:13,670 --> 00:30:17,750
Geoffrey Hinton:
We were completely amazed when
it got to forty four million.
769
00:30:17,750 --> 00:30:21,710
Geoffrey Hinton:
It was so much money that we
couldn't imagine that any more
770
00:30:21,710 --> 00:30:23,789
Geoffrey Hinton:
money would be useful.
771
00:30:23,789 --> 00:30:25,470
Geoffrey Hinton:
I mean, that seemed like as much
772
00:30:25,470 --> 00:30:26,710
Geoffrey Hinton:
money as anybody could possibly
773
00:30:26,710 --> 00:30:27,470
Geoffrey Hinton:
want.
774
00:30:27,470 --> 00:30:29,630
Geoffrey Hinton:
Um, and so we then became much
775
00:30:29,630 --> 00:30:32,130
Geoffrey Hinton:
more concerned about who we
776
00:30:32,130 --> 00:30:34,630
Geoffrey Hinton:
worked for, and I wouldn't have
777
00:30:34,630 --> 00:30:35,529
Geoffrey Hinton:
been able to get to China
778
00:30:35,529 --> 00:30:36,630
Geoffrey Hinton:
because I couldn't fly at that
779
00:30:36,630 --> 00:30:37,910
Geoffrey Hinton:
time.
780
00:30:37,910 --> 00:30:41,150
Geoffrey Hinton:
And I'd spent the summer of
twenty twelve working with Jeff
781
00:30:41,150 --> 00:30:42,829
Geoffrey Hinton:
Dean at Google.
782
00:30:42,829 --> 00:30:44,430
Geoffrey Hinton:
And I got along really well with
Jeff Dean.
783
00:30:44,430 --> 00:30:46,309
Geoffrey Hinton:
It was a really nice group, and
784
00:30:46,309 --> 00:30:48,630
Geoffrey Hinton:
I figured it was much more
785
00:30:48,630 --> 00:30:50,549
Geoffrey Hinton:
important to work in a really
786
00:30:50,549 --> 00:30:52,509
Geoffrey Hinton:
nice place than to get more
787
00:30:52,509 --> 00:30:53,349
Geoffrey Hinton:
money.
788
00:30:53,349 --> 00:30:56,029
Geoffrey Hinton:
So we actually terminated the
auction.
789
00:30:56,029 --> 00:31:00,750
Geoffrey Hinton:
We told Baidu we got an offer he
couldn't refuse, and the offer
790
00:31:00,750 --> 00:31:03,009
Geoffrey Hinton:
we couldn't refuse was the
chance to work at Google with
791
00:31:03,009 --> 00:31:06,730
Geoffrey Hinton:
Jeff Dean, and that all worked
out very well.
792
00:31:08,329 --> 00:31:14,190
Geoffrey Hinton:
So then I was off to Google and
while we were there a year, um,
793
00:31:14,190 --> 00:31:21,470
Geoffrey Hinton:
along with Shockley and Yoshua
and Bodner in Montreal, um, they
794
00:31:21,470 --> 00:31:26,009
Geoffrey Hinton:
developed, uh, attention
language models with attention,
795
00:31:26,009 --> 00:31:31,970
Geoffrey Hinton:
which was a precursor of
Transformers, and showed that
796
00:31:31,970 --> 00:31:35,930
Geoffrey Hinton:
language models actually work
well for machine translation.
797
00:31:35,930 --> 00:31:37,450
Geoffrey Hinton:
And I think that was the final
798
00:31:37,450 --> 00:31:39,009
Geoffrey Hinton:
nail in the coffin of symbolic
799
00:31:39,009 --> 00:31:40,970
Geoffrey Hinton:
AI, because if anything was
800
00:31:40,970 --> 00:31:42,730
Geoffrey Hinton:
going to be good for symbolic
801
00:31:42,730 --> 00:31:44,849
Geoffrey Hinton:
AI, it was converting symbol
802
00:31:44,849 --> 00:31:46,049
Geoffrey Hinton:
strings in one language into
803
00:31:46,049 --> 00:31:47,089
Geoffrey Hinton:
symbol strings in another
804
00:31:47,089 --> 00:31:48,089
Geoffrey Hinton:
language.
805
00:31:48,089 --> 00:31:50,569
Geoffrey Hinton:
The idea that you might do that
by taking symbol strings and
806
00:31:50,569 --> 00:31:53,569
Geoffrey Hinton:
manipulating them actually
sounded quite plausible.
807
00:31:53,569 --> 00:31:55,809
Geoffrey Hinton:
Um, but that's not the way to do
it.
808
00:31:55,809 --> 00:31:58,009
Geoffrey Hinton:
The way to do it is to
understand what's being said in
809
00:31:58,009 --> 00:32:02,180
Geoffrey Hinton:
one language, by associating big
vectors with words appropriately
810
00:32:02,180 --> 00:32:05,740
Geoffrey Hinton:
vectors, and then convert that
to the other language.
811
00:32:05,740 --> 00:32:08,539
Geoffrey Hinton:
Um, so it was clear by about
812
00:32:08,539 --> 00:32:11,740
Geoffrey Hinton:
twenty fifteen that neural nets
813
00:32:11,740 --> 00:32:12,500
Geoffrey Hinton:
were going to do everything,
814
00:32:12,500 --> 00:32:14,140
Geoffrey Hinton:
including language.
815
00:32:14,140 --> 00:32:16,779
Geoffrey Hinton:
That's the point at which Gary
Marcus published a book chapter
816
00:32:16,779 --> 00:32:18,779
Geoffrey Hinton:
saying neural nets were okay.
817
00:32:18,779 --> 00:32:21,519
Geoffrey Hinton:
Maybe they could do object
recognition, but they'd never do
818
00:32:21,519 --> 00:32:24,420
Geoffrey Hinton:
language because language
involved novel sentences.
819
00:32:24,420 --> 00:32:26,099
Geoffrey Hinton:
They were already doing it.
820
00:32:30,819 --> 00:32:36,099
Tom Mitchell:
Well. So that was twenty
fifteen.
821
00:32:36,099 --> 00:32:37,859
Tom Mitchell:
You were still at Google?
822
00:32:38,180 --> 00:32:39,819
Geoffrey Hinton:
I was at Google. And Ilya then
823
00:32:39,819 --> 00:32:45,259
Geoffrey Hinton:
moved to OpenAI, um, around
824
00:32:45,259 --> 00:32:48,779
Geoffrey Hinton:
twenty fifteen, uh, maybe twenty
825
00:32:48,779 --> 00:32:49,859
Geoffrey Hinton:
fourteen, I can't remember the
826
00:32:49,859 --> 00:32:51,299
Geoffrey Hinton:
year.
827
00:32:51,299 --> 00:32:52,819
Geoffrey Hinton:
And, um.
828
00:32:54,819 --> 00:32:57,539
Geoffrey Hinton:
And then OpenAI did rather well.
829
00:32:57,539 --> 00:32:59,160
Geoffrey Hinton:
Um, Over an hour.
830
00:32:59,160 --> 00:33:01,279
Geoffrey Hinton:
I basically just took stuff that
had been done at Google on
831
00:33:01,279 --> 00:33:07,079
Geoffrey Hinton:
Transformers and put a nicer
interface on it, and realized
832
00:33:07,079 --> 00:33:11,039
Geoffrey Hinton:
which Google hadn't realized
that if you did human
833
00:33:11,039 --> 00:33:14,480
Geoffrey Hinton:
reinforcement learning, you
didn't need that many examples
834
00:33:14,480 --> 00:33:16,319
Geoffrey Hinton:
to make it behave nicer.
835
00:33:16,319 --> 00:33:19,920
Geoffrey Hinton:
Um, you didn't need like one
hundred million examples which
836
00:33:19,920 --> 00:33:22,359
Geoffrey Hinton:
you might have thought you could
do it with.
837
00:33:22,359 --> 00:33:25,480
Geoffrey Hinton:
Like some fraction of a million
examples would already make it
838
00:33:25,480 --> 00:33:27,200
Geoffrey Hinton:
behave a lot better.
839
00:33:27,200 --> 00:33:31,240
Geoffrey Hinton:
So you could actually train it
up to have nicer behavior.
840
00:33:32,279 --> 00:33:34,079
Geoffrey Hinton:
And that was ChatGPT.
841
00:33:34,079 --> 00:33:40,539
Geoffrey Hinton:
Um, Google was then in the
classic situation of not wanting
842
00:33:40,539 --> 00:33:43,680
Geoffrey Hinton:
to interfere with search, which
was its moneymaker.
843
00:33:44,799 --> 00:33:47,200
Geoffrey Hinton:
So it was in this difficult
situation.
844
00:33:47,200 --> 00:33:49,880
Geoffrey Hinton:
Do they do they release chatbots
or not?
845
00:33:49,880 --> 00:33:53,640
Geoffrey Hinton:
But when Microsoft teamed up
with OpenAI, they basically had
846
00:33:53,640 --> 00:33:55,039
Geoffrey Hinton:
to release them.
847
00:33:55,039 --> 00:33:58,099
Geoffrey Hinton:
Um, but they lost a few years.
848
00:33:58,099 --> 00:34:03,019
Geoffrey Hinton:
I think it was partly because
search was working so well, and
849
00:34:03,019 --> 00:34:06,220
Geoffrey Hinton:
it was obvious search would be
better if instead of using
850
00:34:06,220 --> 00:34:10,059
Geoffrey Hinton:
keywords, it used what you
meant, which would mean it had
851
00:34:10,059 --> 00:34:12,219
Geoffrey Hinton:
to understand what you meant.
852
00:34:12,219 --> 00:34:17,019
Geoffrey Hinton:
Um, but they didn't want to
undermine their moneymaker.
853
00:34:18,940 --> 00:34:21,619
Geoffrey Hinton:
No, that's based not on any
inside information.
854
00:34:21,619 --> 00:34:22,940
Geoffrey Hinton:
It just seems obvious.
855
00:34:25,179 --> 00:34:27,300
Tom Mitchell:
Pretty amazing.
856
00:34:27,300 --> 00:34:30,179
Tom Mitchell:
So. So here we are now.
857
00:34:30,179 --> 00:34:35,619
Tom Mitchell:
And you were famously on record,
uh, warning people about some of
858
00:34:35,619 --> 00:34:38,219
Tom Mitchell:
the risks of AI.
859
00:34:38,219 --> 00:34:41,380
Tom Mitchell:
Um, what should what should
860
00:34:41,380 --> 00:34:43,219
Tom Mitchell:
people who are working in this
861
00:34:43,219 --> 00:34:45,659
Tom Mitchell:
area do in response to that
862
00:34:45,659 --> 00:34:46,380
Tom Mitchell:
risk?
863
00:34:47,139 --> 00:34:49,539
Geoffrey Hinton:
Okay, so I didn't actually talk
864
00:34:49,539 --> 00:34:50,780
Geoffrey Hinton:
much about the risks until I
865
00:34:50,780 --> 00:34:51,340
Geoffrey Hinton:
left Google.
866
00:34:51,340 --> 00:34:52,699
Geoffrey Hinton:
I realized in the beginning of
867
00:34:52,699 --> 00:34:54,900
Geoffrey Hinton:
twenty twenty three there was a
868
00:34:54,900 --> 00:34:56,480
Geoffrey Hinton:
huge existential threat I hadn't
869
00:34:56,480 --> 00:35:00,079
Geoffrey Hinton:
fully appreciated, because it's
870
00:35:00,079 --> 00:35:01,320
Geoffrey Hinton:
a better form of intelligence
871
00:35:01,320 --> 00:35:03,960
Geoffrey Hinton:
than us, and it's better because
872
00:35:03,960 --> 00:35:06,519
Geoffrey Hinton:
it can share so different copies
873
00:35:06,519 --> 00:35:08,559
Geoffrey Hinton:
of the same neural net, can look
874
00:35:08,559 --> 00:35:10,719
Geoffrey Hinton:
at different data and share the
875
00:35:10,719 --> 00:35:11,920
Geoffrey Hinton:
gradient, and then update all
876
00:35:11,920 --> 00:35:13,920
Geoffrey Hinton:
their weights in sync and stay
877
00:35:13,920 --> 00:35:16,039
Geoffrey Hinton:
the same so they can keep doing
878
00:35:16,039 --> 00:35:16,880
Geoffrey Hinton:
that.
879
00:35:16,880 --> 00:35:19,559
Geoffrey Hinton:
And when they share the
gradient, they're sharing
880
00:35:19,559 --> 00:35:22,719
Geoffrey Hinton:
information they got from
different data sets.
881
00:35:22,719 --> 00:35:24,719
Geoffrey Hinton:
Um, out of the order of a
882
00:35:24,719 --> 00:35:26,199
Geoffrey Hinton:
trillion bits per episode of
883
00:35:26,199 --> 00:35:26,840
Geoffrey Hinton:
sharing.
884
00:35:26,840 --> 00:35:28,920
Geoffrey Hinton:
If they've got a trillion
weights.
885
00:35:28,920 --> 00:35:31,000
Geoffrey Hinton:
Whereas what we're doing now is
886
00:35:31,000 --> 00:35:32,639
Geoffrey Hinton:
sharing the information and
887
00:35:32,639 --> 00:35:33,639
Geoffrey Hinton:
maybe one hundred bits per
888
00:35:33,639 --> 00:35:34,639
Geoffrey Hinton:
sentence.
889
00:35:34,639 --> 00:35:38,639
Geoffrey Hinton:
Um, so a few bits per second,
maybe if we're lucky, we're
890
00:35:38,639 --> 00:35:40,719
Geoffrey Hinton:
sharing it ten bits per second.
891
00:35:40,719 --> 00:35:45,179
Geoffrey Hinton:
Um, and so you're comparing like
892
00:35:45,179 --> 00:35:46,840
Geoffrey Hinton:
trillions of bits with hundreds
893
00:35:46,840 --> 00:35:47,760
Geoffrey Hinton:
of bits.
894
00:35:47,760 --> 00:35:51,480
Geoffrey Hinton:
There are billions of times
better than us at sharing.
895
00:35:51,480 --> 00:35:55,139
Geoffrey Hinton:
And that's why if I'm running on
different hardware, they can
896
00:35:55,139 --> 00:35:57,420
Geoffrey Hinton:
learn so much more than us.
897
00:35:57,420 --> 00:35:58,659
Geoffrey Hinton:
They can learn from the whole
internet.
898
00:35:58,659 --> 00:36:01,159
Geoffrey Hinton:
It doesn't all have to go
through one piece of hardware,
899
00:36:01,159 --> 00:36:05,099
Geoffrey Hinton:
and it's going to get more
important that effect as we go
900
00:36:05,099 --> 00:36:09,139
Geoffrey Hinton:
to AI agents that operate in the
real world in real time.
901
00:36:09,139 --> 00:36:12,659
Geoffrey Hinton:
Most AI you images, you can just
speed them up and send them
902
00:36:12,659 --> 00:36:14,980
Geoffrey Hinton:
through one network very fast.
903
00:36:14,980 --> 00:36:18,179
Geoffrey Hinton:
Um, because obviously computers
operate, you know, thousands of
904
00:36:18,179 --> 00:36:19,780
Geoffrey Hinton:
times faster than a brain.
905
00:36:19,780 --> 00:36:22,420
Geoffrey Hinton:
But, um, if you're operating in
906
00:36:22,420 --> 00:36:24,340
Geoffrey Hinton:
the real world, you can't get
907
00:36:24,340 --> 00:36:25,940
Geoffrey Hinton:
experience faster.
908
00:36:25,940 --> 00:36:28,659
Geoffrey Hinton:
Um, because the real world has
an actual time scale.
909
00:36:28,659 --> 00:36:29,780
Geoffrey Hinton:
If you're interacting with other
910
00:36:29,780 --> 00:36:30,900
Geoffrey Hinton:
agents who take a little while
911
00:36:30,900 --> 00:36:33,739
Geoffrey Hinton:
to reply, um, then this
912
00:36:33,739 --> 00:36:35,699
Geoffrey Hinton:
advantage that different copies
913
00:36:35,699 --> 00:36:37,199
Geoffrey Hinton:
of the same neural net can share
914
00:36:37,199 --> 00:36:38,059
Geoffrey Hinton:
will be an even bigger
915
00:36:38,059 --> 00:36:39,940
Geoffrey Hinton:
advantage.
916
00:36:39,940 --> 00:36:42,039
Geoffrey Hinton:
So at that point, I decided
917
00:36:42,039 --> 00:36:43,300
Geoffrey Hinton:
there's all these short term
918
00:36:43,300 --> 00:36:45,099
Geoffrey Hinton:
threats, and it wasn't really my
919
00:36:45,099 --> 00:36:46,340
Geoffrey Hinton:
intention to warn about those,
920
00:36:46,340 --> 00:36:47,699
Geoffrey Hinton:
but I got sucked into warning
921
00:36:47,699 --> 00:36:49,059
Geoffrey Hinton:
about those because journalists
922
00:36:49,059 --> 00:36:50,260
Geoffrey Hinton:
always confuse the existential
923
00:36:50,260 --> 00:36:51,019
Geoffrey Hinton:
threat with all the other
924
00:36:51,019 --> 00:36:51,579
Geoffrey Hinton:
threats.
925
00:36:51,579 --> 00:36:53,360
Geoffrey Hinton:
They just muddle all the threats
together.
926
00:36:53,360 --> 00:36:56,539
Geoffrey Hinton:
They move seamlessly from
joblessness to fake videos to
927
00:36:56,539 --> 00:36:59,639
Geoffrey Hinton:
cyber attacks to lethal
autonomous weapons, as if
928
00:36:59,639 --> 00:37:00,960
Geoffrey Hinton:
they're all the same thing.
929
00:37:00,960 --> 00:37:03,139
Geoffrey Hinton:
Um, so I had to sort of clarify
930
00:37:03,139 --> 00:37:05,440
Geoffrey Hinton:
a lot of those threats, but my
931
00:37:05,440 --> 00:37:07,679
Geoffrey Hinton:
main worry was the much longer
932
00:37:07,679 --> 00:37:10,420
Geoffrey Hinton:
term threat, but not long enough
933
00:37:10,420 --> 00:37:12,760
Geoffrey Hinton:
that they will be much smarter
934
00:37:12,760 --> 00:37:13,800
Geoffrey Hinton:
than us.
935
00:37:13,800 --> 00:37:15,360
Geoffrey Hinton:
It's not necessarily the case,
936
00:37:15,360 --> 00:37:17,860
Geoffrey Hinton:
but I think most people, most
937
00:37:17,860 --> 00:37:19,840
Geoffrey Hinton:
neural net experts, believe that
938
00:37:19,840 --> 00:37:21,079
Geoffrey Hinton:
within twenty years we'll have
939
00:37:21,079 --> 00:37:22,639
Geoffrey Hinton:
superintelligent AI.
940
00:37:23,760 --> 00:37:26,519
Geoffrey Hinton:
We vary, you know, Demis thinks
it'll be about ten years.
941
00:37:26,519 --> 00:37:28,239
Geoffrey Hinton:
I think it may be as long as
twenty years.
942
00:37:28,239 --> 00:37:31,360
Geoffrey Hinton:
And it'll very likely be more
than five years.
943
00:37:31,360 --> 00:37:33,639
Geoffrey Hinton:
Um, Dario thinks it'll be three
years.
944
00:37:33,639 --> 00:37:35,960
Geoffrey Hinton:
Um, but then he runs a company.
945
00:37:35,960 --> 00:37:39,199
Geoffrey Hinton:
Um, so any.
946
00:37:39,199 --> 00:37:42,280
Geoffrey Hinton:
Ilya thinks it'll be sooner than
ten years.
947
00:37:42,280 --> 00:37:45,159
Geoffrey Hinton:
Um, we all think it's probably
going to happen.
948
00:37:45,159 --> 00:37:47,519
Geoffrey Hinton:
So the question is, what happens
949
00:37:47,519 --> 00:37:50,010
Geoffrey Hinton:
when AI is a lot smarter than us
950
00:37:50,010 --> 00:37:51,699
Geoffrey Hinton:
And when it's our agents that
951
00:37:51,699 --> 00:37:52,780
Geoffrey Hinton:
are smart enough so they're also
952
00:37:52,780 --> 00:37:55,139
Geoffrey Hinton:
more powerful than us, they can
953
00:37:55,139 --> 00:37:56,400
Geoffrey Hinton:
collaborate with other AI
954
00:37:56,400 --> 00:37:58,860
Geoffrey Hinton:
agents, get stuff done even if
955
00:37:58,860 --> 00:38:02,219
Geoffrey Hinton:
they can't sort of fire guns or
956
00:38:02,219 --> 00:38:03,820
Geoffrey Hinton:
pull switches.
957
00:38:03,820 --> 00:38:05,139
Geoffrey Hinton:
They can persuade people.
958
00:38:05,139 --> 00:38:06,460
Geoffrey Hinton:
And we know AI is already very
959
00:38:06,460 --> 00:38:08,079
Geoffrey Hinton:
good at persuasion and will soon
960
00:38:08,079 --> 00:38:09,219
Geoffrey Hinton:
be much better than people at
961
00:38:09,219 --> 00:38:11,500
Geoffrey Hinton:
persuasion, like in ten years
962
00:38:11,500 --> 00:38:12,699
Geoffrey Hinton:
time.
963
00:38:12,699 --> 00:38:15,179
Geoffrey Hinton:
And so they'll be able to
persuade people to do things
964
00:38:15,179 --> 00:38:18,460
Geoffrey Hinton:
just like Trump persuaded people
to invade the capital.
965
00:38:18,460 --> 00:38:20,739
Geoffrey Hinton:
Um, so they don't actually have
to be able to do anything
966
00:38:20,739 --> 00:38:22,179
Geoffrey Hinton:
themselves except talk.
967
00:38:24,099 --> 00:38:28,800
Geoffrey Hinton:
So most of the tech bros are
968
00:38:28,800 --> 00:38:31,039
Geoffrey Hinton:
thinking they have a model,
969
00:38:31,039 --> 00:38:34,099
Geoffrey Hinton:
which is I'm the CEO, you're the
970
00:38:34,099 --> 00:38:35,059
Geoffrey Hinton:
secretary.
971
00:38:35,059 --> 00:38:36,860
Geoffrey Hinton:
You're much smarter than me.
972
00:38:36,860 --> 00:38:38,480
Geoffrey Hinton:
Um, but I can always fire you,
973
00:38:38,480 --> 00:38:39,820
Geoffrey Hinton:
and you'll make my life really
974
00:38:39,820 --> 00:38:40,420
Geoffrey Hinton:
easy.
975
00:38:40,420 --> 00:38:41,440
Geoffrey Hinton:
Because whatever I want to
976
00:38:41,440 --> 00:38:43,619
Geoffrey Hinton:
happen, I'll sort of be like
977
00:38:43,619 --> 00:38:44,059
Geoffrey Hinton:
Star Trek.
978
00:38:44,059 --> 00:38:47,179
Geoffrey Hinton:
I'll say, make it so and it will
happen.
979
00:38:47,179 --> 00:38:49,840
Geoffrey Hinton:
Um. and I don't really have to
understand it.
980
00:38:49,840 --> 00:38:51,679
Geoffrey Hinton:
I'll still get the credit for it
because I said make it.
981
00:38:51,679 --> 00:38:56,480
Geoffrey Hinton:
So, um, I think that's their
model, and I just don't think
982
00:38:56,480 --> 00:38:57,599
Geoffrey Hinton:
that's going to work.
983
00:38:58,679 --> 00:39:00,679
Geoffrey Hinton:
I think the big problem is how
984
00:39:00,679 --> 00:39:02,039
Geoffrey Hinton:
do we prevent these things ever
985
00:39:02,039 --> 00:39:04,659
Geoffrey Hinton:
wanting to take control or to
986
00:39:04,659 --> 00:39:05,760
Geoffrey Hinton:
take over?
987
00:39:05,760 --> 00:39:08,239
Geoffrey Hinton:
They may have control, but they
988
00:39:08,239 --> 00:39:09,559
Geoffrey Hinton:
may still not want to replace
989
00:39:09,559 --> 00:39:10,800
Geoffrey Hinton:
us.
990
00:39:10,800 --> 00:39:14,239
Geoffrey Hinton:
And so I've fallen back on the
only example I know of a less
991
00:39:14,239 --> 00:39:17,679
Geoffrey Hinton:
intelligent thing controlling a
more intelligent thing.
992
00:39:17,679 --> 00:39:19,840
Geoffrey Hinton:
And that's a baby controlling a
mother.
993
00:39:22,119 --> 00:39:25,079
Geoffrey Hinton:
And evolution has put a huge
amount of work into that.
994
00:39:27,639 --> 00:39:31,159
Geoffrey Hinton:
So evolution has made sure the
mother cannot bear the sound of
995
00:39:31,159 --> 00:39:36,599
Geoffrey Hinton:
the baby crying, and the mother
gets huge rewards, um, for being
996
00:39:36,599 --> 00:39:38,119
Geoffrey Hinton:
nice to the baby.
997
00:39:38,119 --> 00:39:40,079
Geoffrey Hinton:
Um, lots of pleasurable
998
00:39:40,079 --> 00:39:41,519
Geoffrey Hinton:
sensations and just generally
999
00:39:41,519 --> 00:39:42,679
Geoffrey Hinton:
good feelings.
1000
00:39:42,679 --> 00:39:47,739
Geoffrey Hinton:
Um, and We need to do the same
1001
00:39:47,739 --> 00:39:49,019
Geoffrey Hinton:
for these eyes, for being nice
1002
00:39:49,019 --> 00:39:49,300
Geoffrey Hinton:
to us.
1003
00:39:49,300 --> 00:39:51,420
Geoffrey Hinton:
We're still making them.
1004
00:39:51,420 --> 00:39:55,340
Geoffrey Hinton:
And if we could make an AI that
was super intelligent but cared
1005
00:39:55,340 --> 00:39:59,219
Geoffrey Hinton:
more about us than it cared
about about itself or other
1006
00:39:59,219 --> 00:40:03,940
Geoffrey Hinton:
superintelligent AIS, then we
might be okay.
1007
00:40:03,940 --> 00:40:06,940
Geoffrey Hinton:
Um, but we have to accept that
we're going to be the babies,
1008
00:40:06,940 --> 00:40:10,980
Geoffrey Hinton:
and they're going to be the
mothers, and people aren't
1009
00:40:10,980 --> 00:40:12,219
Geoffrey Hinton:
prepared to accept that.
1010
00:40:12,219 --> 00:40:13,539
Geoffrey Hinton:
Trump's not prepared to accept
1011
00:40:13,539 --> 00:40:15,219
Geoffrey Hinton:
that Trump would never accept
1012
00:40:15,219 --> 00:40:15,860
Geoffrey Hinton:
that.
1013
00:40:16,940 --> 00:40:19,300
Geoffrey Hinton:
Um, I think we have a lot more
1014
00:40:19,300 --> 00:40:20,219
Geoffrey Hinton:
hope of the Chinese
1015
00:40:20,219 --> 00:40:22,179
Geoffrey Hinton:
understanding it.
1016
00:40:22,179 --> 00:40:23,380
Geoffrey Hinton:
So I recently went to Shanghai
1017
00:40:23,380 --> 00:40:24,219
Geoffrey Hinton:
and talked to a member of the
1018
00:40:24,219 --> 00:40:24,739
Geoffrey Hinton:
Politburo.
1019
00:40:24,739 --> 00:40:27,619
Geoffrey Hinton:
Me and Eric Schmidt, who aren't
natural allies.
1020
00:40:27,619 --> 00:40:31,500
Geoffrey Hinton:
We're in terms of politics with
a rather different Eric Schmidt,
1021
00:40:31,500 --> 00:40:33,820
Geoffrey Hinton:
for example, thinks Kissinger
was a good guy.
1022
00:40:33,820 --> 00:40:39,340
Geoffrey Hinton:
Um, but we agree on this
existential threat, and the
1023
00:40:39,340 --> 00:40:42,900
Geoffrey Hinton:
Chinese leadership will
understand it much better than
1024
00:40:42,900 --> 00:40:45,639
Geoffrey Hinton:
any of the other leaderships,
because many of them are
1025
00:40:45,639 --> 00:40:48,719
Geoffrey Hinton:
engineers and they actually
understand how this stuff works.
1026
00:40:48,719 --> 00:40:50,320
Geoffrey Hinton:
They understand the argument
1027
00:40:50,320 --> 00:40:51,159
Geoffrey Hinton:
that it's a better form of
1028
00:40:51,159 --> 00:40:52,199
Geoffrey Hinton:
intelligence.
1029
00:40:54,039 --> 00:40:59,440
Geoffrey Hinton:
But I think all the countries
will collaborate on can we make
1030
00:40:59,440 --> 00:41:03,679
Geoffrey Hinton:
it so that it cares more about
us than it does about itself?
1031
00:41:03,679 --> 00:41:05,300
Geoffrey Hinton:
Because there if any country
1032
00:41:05,300 --> 00:41:06,460
Geoffrey Hinton:
figured out how to do that, it'd
1033
00:41:06,460 --> 00:41:07,599
Geoffrey Hinton:
be very happy to tell the other
1034
00:41:07,599 --> 00:41:09,280
Geoffrey Hinton:
countries.
1035
00:41:09,280 --> 00:41:11,960
Geoffrey Hinton:
That's like preventing a global
nuclear war.
1036
00:41:11,960 --> 00:41:15,219
Geoffrey Hinton:
And there the USSR and America
collaborated in the nineteen
1037
00:41:15,219 --> 00:41:16,559
Geoffrey Hinton:
fifties on that.
1038
00:41:16,559 --> 00:41:18,320
Geoffrey Hinton:
Um, the height of the Cold War.
1039
00:41:18,320 --> 00:41:20,280
Geoffrey Hinton:
They still collaborated to
prevent that.
1040
00:41:21,719 --> 00:41:26,159
Geoffrey Hinton:
So what I think we should have
is research institutes in
1041
00:41:26,159 --> 00:41:29,679
Geoffrey Hinton:
different countries that get
access to their own country's
1042
00:41:29,679 --> 00:41:31,880
Geoffrey Hinton:
super smart AI, which they're
not going to give to any other
1043
00:41:31,880 --> 00:41:36,079
Geoffrey Hinton:
country and can do experiments
on how to make it not want to,
1044
00:41:36,079 --> 00:41:39,480
Geoffrey Hinton:
how to make it care more about
people than about itself.
1045
00:41:39,480 --> 00:41:43,429
Geoffrey Hinton:
Um, and share with other
countries how to do that,
1046
00:41:43,429 --> 00:41:46,409
Geoffrey Hinton:
because I believe the techniques
for doing that will be roughly
1047
00:41:46,409 --> 00:41:49,489
Geoffrey Hinton:
orthogonal to the techniques for
making it smarter.
1048
00:41:49,489 --> 00:41:50,449
Geoffrey Hinton:
They're not going to share the
1049
00:41:50,449 --> 00:41:51,210
Geoffrey Hinton:
techniques for making it
1050
00:41:51,210 --> 00:41:51,889
Geoffrey Hinton:
smarter, because they're all
1051
00:41:51,889 --> 00:41:52,869
Geoffrey Hinton:
doing cyber attacks on each
1052
00:41:52,869 --> 00:41:54,289
Geoffrey Hinton:
other, and they all know the
1053
00:41:54,289 --> 00:41:55,210
Geoffrey Hinton:
best.
1054
00:41:55,210 --> 00:41:56,489
Geoffrey Hinton:
You want a better AI to do
1055
00:41:56,489 --> 00:41:58,289
Geoffrey Hinton:
better cyber attacks and better
1056
00:41:58,289 --> 00:41:59,610
Geoffrey Hinton:
fake videos and better
1057
00:41:59,610 --> 00:42:01,210
Geoffrey Hinton:
autonomous weapons.
1058
00:42:01,210 --> 00:42:02,489
Geoffrey Hinton:
They're never going to share
that stuff.
1059
00:42:02,489 --> 00:42:04,369
Geoffrey Hinton:
They're anti-aligned.
1060
00:42:04,369 --> 00:42:06,969
Geoffrey Hinton:
But on not having AI replace us,
1061
00:42:06,969 --> 00:42:08,610
Geoffrey Hinton:
they're aligned so they will
1062
00:42:08,610 --> 00:42:09,489
Geoffrey Hinton:
collaborate.
1063
00:42:10,650 --> 00:42:12,530
Geoffrey Hinton:
Now, one of the things you ought
to mention, I ought to mention
1064
00:42:12,530 --> 00:42:14,730
Geoffrey Hinton:
Russ Salakhutdinov, um.
1065
00:42:17,210 --> 00:42:18,969
Geoffrey Hinton:
He was one of my best students.
1066
00:42:18,969 --> 00:42:23,530
Geoffrey Hinton:
Um, he came to Toronto, did his
PhD at Toronto.
1067
00:42:23,530 --> 00:42:27,530
Geoffrey Hinton:
Um, he did a postdoc with Josh
Tenenbaum, and then he wanted to
1068
00:42:27,530 --> 00:42:32,570
Geoffrey Hinton:
come back to Toronto, and he had
a faculty offer from Harvard.
1069
00:42:32,570 --> 00:42:35,530
Geoffrey Hinton:
And I really tried to get the
Department of Computer Science,
1070
00:42:35,530 --> 00:42:39,849
Geoffrey Hinton:
which had an open position in
machine learning, to give a job
1071
00:42:39,849 --> 00:42:42,309
Geoffrey Hinton:
to Russ and they refused.
1072
00:42:43,429 --> 00:42:47,230
Geoffrey Hinton:
Basically, this was about twenty
eleven or twelve.
1073
00:42:47,230 --> 00:42:49,090
Geoffrey Hinton:
No, this was two thousand and
1074
00:42:49,090 --> 00:42:50,349
Geoffrey Hinton:
probably twenty twelve or
1075
00:42:50,349 --> 00:42:51,030
Geoffrey Hinton:
thirteen.
1076
00:42:52,070 --> 00:42:55,670
Geoffrey Hinton:
My department was one of the
last departments to accept that
1077
00:42:55,670 --> 00:42:57,989
Geoffrey Hinton:
neural networks really worked.
1078
00:42:57,989 --> 00:42:59,630
Geoffrey Hinton:
They had a big AI group, and the
1079
00:42:59,630 --> 00:43:00,869
Geoffrey Hinton:
big AI group said you had got
1080
00:43:00,869 --> 00:43:01,869
Geoffrey Hinton:
several people in neural
1081
00:43:01,869 --> 00:43:02,469
Geoffrey Hinton:
networks already.
1082
00:43:02,469 --> 00:43:03,949
Geoffrey Hinton:
That's your quota.
1083
00:43:03,949 --> 00:43:07,070
Geoffrey Hinton:
We're short on people in
knowledge representation, and we
1084
00:43:07,070 --> 00:43:09,230
Geoffrey Hinton:
need as many people in
computational linguistics as we
1085
00:43:09,230 --> 00:43:10,750
Geoffrey Hinton:
do in neural networks.
1086
00:43:10,750 --> 00:43:14,510
Geoffrey Hinton:
Um, and they refused to give us
a job.
1087
00:43:14,510 --> 00:43:15,670
Geoffrey Hinton:
So we eventually got a job in
1088
00:43:15,670 --> 00:43:17,590
Geoffrey Hinton:
statistics so that it could be
1089
00:43:17,590 --> 00:43:18,510
Geoffrey Hinton:
in Toronto.
1090
00:43:19,829 --> 00:43:21,510
Geoffrey Hinton:
And we were trying to negotiate
1091
00:43:21,510 --> 00:43:22,750
Geoffrey Hinton:
it for him to move to computer
1092
00:43:22,750 --> 00:43:23,869
Geoffrey Hinton:
science.
1093
00:43:23,869 --> 00:43:28,829
Geoffrey Hinton:
And then CMU swooped in, and I
think they offered him tenure at
1094
00:43:28,829 --> 00:43:30,909
Geoffrey Hinton:
CMU, and that was that.
1095
00:43:31,230 --> 00:43:37,949
Tom Mitchell:
Well, you have a whole cadre of
former students who are, um,
1096
00:43:37,949 --> 00:43:42,909
Tom Mitchell:
really leading the charge,
leading the way, and in a lot of
1097
00:43:42,909 --> 00:43:46,489
Tom Mitchell:
areas of neural nets, it's
pretty amazing if you.
1098
00:43:46,329 --> 00:43:47,090
Geoffrey Hinton:
Well, it was luck.
1099
00:43:47,090 --> 00:43:48,010
Geoffrey Hinton:
It was luck basically.
1100
00:43:48,010 --> 00:43:50,730
Geoffrey Hinton:
There were so few people who
believed in neural nets.
1101
00:43:50,730 --> 00:43:51,289
Geoffrey Hinton:
Who was Yann?
1102
00:43:51,289 --> 00:43:54,489
Geoffrey Hinton:
There was Yoshua, there was me,
the Schmidhuber.
1103
00:43:54,489 --> 00:43:55,449
Geoffrey Hinton:
Uh, there were a few other
1104
00:43:55,449 --> 00:43:57,570
Geoffrey Hinton:
people, but MIT didn't have
1105
00:43:57,570 --> 00:43:57,969
Geoffrey Hinton:
anybody.
1106
00:43:57,969 --> 00:43:59,650
Geoffrey Hinton:
Stanford didn't have anybody.
1107
00:43:59,650 --> 00:44:01,809
Geoffrey Hinton:
Um, Berkeley didn't have
anybody.
1108
00:44:01,809 --> 00:44:04,010
Geoffrey Hinton:
Um, Mike Jordan made sure of
that.
1109
00:44:04,010 --> 00:44:11,250
Geoffrey Hinton:
And, um, so the few of us who
believed in it got the really
1110
00:44:11,250 --> 00:44:15,050
Geoffrey Hinton:
good students who believed in
it, and that was great.
1111
00:44:16,409 --> 00:44:17,170
Tom Mitchell:
It worked.
1112
00:44:17,969 --> 00:44:22,090
Geoffrey Hinton:
People like Russ and E and
George Stahl and other people.
1113
00:44:22,090 --> 00:44:22,530
Geoffrey Hinton:
It was.
1114
00:44:22,530 --> 00:44:23,130
Geoffrey Hinton:
Yeah.
1115
00:44:23,849 --> 00:44:26,849
Tom Mitchell:
So if you could, uh, one final
question.
1116
00:44:26,849 --> 00:44:35,170
Tom Mitchell:
If you could give advice to new
PhD students now entering this
1117
00:44:35,170 --> 00:44:37,710
Tom Mitchell:
area, what would you say?
1118
00:44:39,590 --> 00:44:41,670
Geoffrey Hinton:
Sometimes I'd say become a
plumber.
1119
00:44:41,670 --> 00:44:43,070
Geoffrey Hinton:
You're too late.
1120
00:44:43,070 --> 00:44:49,110
Geoffrey Hinton:
Um. But actually, I say, if
you're a CMU and you're doing
1121
00:44:49,110 --> 00:44:52,949
Geoffrey Hinton:
this, you may be in the small
fraction of people who survive
1122
00:44:52,949 --> 00:44:56,269
Geoffrey Hinton:
in ER and don't get replaced,
because for quite a while
1123
00:44:56,269 --> 00:45:00,710
Geoffrey Hinton:
there's going to be creative
people making our work better.
1124
00:45:00,710 --> 00:45:03,949
Geoffrey Hinton:
And you've got a good chance of
being one of those people if
1125
00:45:03,949 --> 00:45:04,989
Geoffrey Hinton:
you're at CMU.
1126
00:45:07,309 --> 00:45:07,869
Tom Mitchell:
All right.
1127
00:45:07,869 --> 00:45:09,550
Tom Mitchell:
Well, we'll take that.
1128
00:45:09,550 --> 00:45:13,789
Tom Mitchell:
Uh, Jeff, thank you so much for
spending the time sharing that.
1129
00:45:13,789 --> 00:45:18,349
Tom Mitchell:
Um, it's it's always great to
catch up and, um.
1130
00:45:18,349 --> 00:45:19,070
Tom Mitchell:
Thank you.
1131
00:45:19,710 --> 00:45:21,469
Geoffrey Hinton:
Okay. Well, thank you for
inviting me.
1132
00:45:24,349 --> 00:45:25,429
Speaker 3:
Tom Mitchell is the founders
1133
00:45:25,429 --> 00:45:26,789
Speaker 3:
university professor at Carnegie
1134
00:45:26,789 --> 00:45:27,989
Speaker 3:
Mellon University.
1135
00:45:27,989 --> 00:45:28,630
Speaker 3:
Machine learning.
1136
00:45:28,630 --> 00:45:29,349
Speaker 3:
How did we get here?
1137
00:45:29,349 --> 00:45:31,829
Speaker 3:
Is produced by the Stanford
Digital Economy Lab.
1138
00:45:31,829 --> 00:45:32,989
Speaker 3:
If you enjoyed this episode,
1139
00:45:32,989 --> 00:45:34,090
Speaker 3:
subscribe wherever you listen to
1140
00:45:34,090 --> 00:45:35,110
Speaker 3:
podcasts.
