WEBVTT
1
00:00:10.310 --> 00:00:13.949
Welcome to the BIGGA standard experience.
I'm your guest host for today, getting
2
00:00:13.990 --> 00:00:18.910
in Reuben. I'm the founder and
CEO of pandio. On our show today
3
00:00:18.949 --> 00:00:23.780
we have Sam Anthony, the CTEO
and Co founder of perceptive automats. Up
4
00:00:23.780 --> 00:00:27.460
Today. We are speaking about the
missing piece for autonomous driving. Welcome to
5
00:00:27.500 --> 00:00:30.100
the show. Thanks for having me. Great I'd love to hear a little
6
00:00:30.100 --> 00:00:34.100
bit more about your background, if
you wouldn't mind starting their sure. So
7
00:00:34.659 --> 00:00:40.530
perceptive ATOMEDA came out of a bunch
of research projects that my cofounders and myself
8
00:00:40.729 --> 00:00:46.130
did when I was a Grad student
actually at Harvard and Vision Sciences Laboratory Harbor.
9
00:00:46.329 --> 00:00:49.770
My background, it's kind of a
stint as a computer hacker, which
10
00:00:49.770 --> 00:00:52.520
we can talk about if you really
want to. But after that my background
11
00:00:52.560 --> 00:00:57.280
is in before Grad School was in
Web Development and and web applications, and
12
00:00:57.399 --> 00:01:03.679
when I got to Harvard, what
I was working on was doing careful psychological
13
00:01:03.799 --> 00:01:10.349
experiments, so careful human behavioral experiments
at Web scale, and this is something
14
00:01:10.390 --> 00:01:12.109
you know. People Talk About Mechanical
Turk and that's kind of a way you
15
00:01:12.310 --> 00:01:19.299
can do this, but what we
pioneered was really taking seriously being doing good,
16
00:01:19.900 --> 00:01:26.180
careful vision science experiments, and what
my cofounders gave Cox and Walter Shire
17
00:01:26.299 --> 00:01:33.049
and I realized was that what I
was doing with these experiments was happening in
18
00:01:33.129 --> 00:01:36.170
a scale where we could start thinking
about using it for machine learning, and
19
00:01:36.969 --> 00:01:41.209
that led us to kind of a
follow up realization that the way everyone else
20
00:01:41.290 --> 00:01:44.569
did machine learning was really missing at
trick, because the way everyone else does
21
00:01:44.689 --> 00:01:51.280
machine learning they think of the labeling
as just this kind of like noisy process
22
00:01:51.319 --> 00:01:56.760
that happens in advance where you're trying
to get your training samples, your things
23
00:01:56.799 --> 00:02:00.280
that you're going to feed your machine
learning model, tagged with like a fact
24
00:02:00.319 --> 00:02:04.950
about the world, and we'll really
realize that that that that process of labeling
25
00:02:05.430 --> 00:02:09.990
was really, in many ways,
exactly the same as running a psychological discernment,
26
00:02:10.069 --> 00:02:14.229
except in the psychological experiment you can
learn much, much more about how
27
00:02:14.270 --> 00:02:19.659
people look at the world. So
we kind of combined these insights from computer
28
00:02:19.740 --> 00:02:23.099
vision and from humanive behavioral science and
ended up with a with a with an
29
00:02:23.099 --> 00:02:30.849
approach for the methodology where that methodology
is capable of building machine learning models that
30
00:02:31.050 --> 00:02:36.729
will answer questions and kind of the
specific category and specific categories things that people
31
00:02:36.770 --> 00:02:40.969
are effortlessly good at and computers are
not good at, and so that's a
32
00:02:42.009 --> 00:02:45.680
pretty useful a pretty interesting set of
problems. And so that's kind of where
33
00:02:45.680 --> 00:02:49.479
we started and after quite a bit
of iteration we got to the problem that
34
00:02:49.599 --> 00:02:53.560
for solving with perceptive Automata, which
is understanding humans in the world from the
35
00:02:53.680 --> 00:02:59.669
perspective of autonomous vehicles, in the
autos robots. In Graduate School. Was
36
00:02:59.710 --> 00:03:04.389
that your first for into the Book
Behavioral Science or had you been studying that
37
00:03:04.550 --> 00:03:07.750
earlier? I studied it an Undergrad. So I had a super weird,
38
00:03:07.909 --> 00:03:13.780
like totally not scalable career path.
I directly out of High School for went
39
00:03:13.939 --> 00:03:16.580
college because that was the the firstcom
boom and I was like, I'll just
40
00:03:16.620 --> 00:03:20.539
go make a lot of money doing
you know, they'll pay me how much
41
00:03:20.580 --> 00:03:23.819
to make a web page, and
I so I did that and I kind
42
00:03:23.819 --> 00:03:27.849
of moved through tech and had kind
of a initial career in tech for about
43
00:03:27.930 --> 00:03:31.289
ten years and I left that because
it felt like the work that I was
44
00:03:31.330 --> 00:03:37.289
doing building web applications was getting commoditized
and went back to college and had the
45
00:03:37.370 --> 00:03:44.199
chance to sort of pick between the
University of California campuses and the way it
46
00:03:44.280 --> 00:03:47.080
work was transferring into the school.
So I had to pick a major to
47
00:03:47.159 --> 00:03:52.439
transfer in and and at every UC
campus except for one, I said I
48
00:03:52.439 --> 00:03:55.389
would major in math. And then, for whatever reason, UCSD I was
49
00:03:55.469 --> 00:03:59.469
like, Oh, there, I
heard their cognitive science program is pretty good.
50
00:03:59.509 --> 00:04:01.229
I'll put down cognitive science on this
one. And I got in not
51
00:04:01.349 --> 00:04:04.870
everywhere, but almost everywhere, and
realize I absolutely did not want to be
52
00:04:04.870 --> 00:04:08.990
a math major, and so I
ended up in the cognitive science program at
53
00:04:09.270 --> 00:04:12.780
UCSD and everything kind of everything kind
of float from their nice. Yeah,
54
00:04:12.780 --> 00:04:15.379
UCS. These actually down the street
from us. I'm in the Carmel Vallege.
55
00:04:15.459 --> 00:04:17.699
Oh, excellent, excellent, excellent. It's I'll tell you what.
56
00:04:17.779 --> 00:04:23.139
I having moved back to Boston,
the I have regrets about the the weather
57
00:04:23.180 --> 00:04:26.329
difference. Well, I do the
reverse, you know. I went to
58
00:04:26.850 --> 00:04:30.449
the College at Brandise University. Of
My brother was a ucks so I spent
59
00:04:30.529 --> 00:04:33.089
some time in summer go as well, you know. And but I grew
60
00:04:33.129 --> 00:04:36.689
up in Arizona. So I went
from Arizona to Boston and I knew,
61
00:04:36.730 --> 00:04:40.399
Oh boyd want to be an Arizona
that. I knew I didn't want to
62
00:04:40.399 --> 00:04:45.439
be a Boston. Yeah, so
I'd love to get in a little more
63
00:04:45.480 --> 00:04:47.240
about what you're working on now.
So can you tell us a little bit
64
00:04:47.279 --> 00:04:51.839
about the black box metaphor and how
this relates to what you're doing? Absolutely,
65
00:04:51.990 --> 00:04:57.310
so, what we're our system does, is our system allows and a
66
00:04:57.389 --> 00:05:01.589
ton of US vehicle to know what's
in the heads of pedestrians. And that's
67
00:05:01.629 --> 00:05:04.269
it like a weird thing to say, right, so, because what's in
68
00:05:04.389 --> 00:05:06.180
my head? Well, it's in
my head, you know. You know,
69
00:05:06.220 --> 00:05:09.579
there's not a telepathy right, how
are you going to figure that out?
70
00:05:09.620 --> 00:05:12.540
But we, as humans, were
very, very social and or very
71
00:05:12.620 --> 00:05:16.579
good at social reasoning. So when
we look at another human we've very quickly
72
00:05:16.740 --> 00:05:20.730
come up with a whole set of
inferences about what's in their head, what
73
00:05:20.850 --> 00:05:26.529
they're thinking, what they want and
whether or not we're right in some absolute
74
00:05:26.649 --> 00:05:30.449
sense, we're right enough to act
on it and for it to for it
75
00:05:30.529 --> 00:05:33.410
to lead to such behavior. Right. By contrast, when you train a
76
00:05:33.529 --> 00:05:39.240
traditional machine learning model, what that
machine learning model is doing is it's trying
77
00:05:39.319 --> 00:05:45.199
to extract a mapping between a set
of pixels and a set of numbers.
78
00:05:45.600 --> 00:05:48.839
Where that mapping comes from, you
don't care. It's irrelevant. What you're
79
00:05:48.839 --> 00:05:51.990
really trying to do is you're trying
to you have some mapping, and again
80
00:05:53.069 --> 00:05:55.629
this goes back to the training process
and your training data. You have a
81
00:05:55.709 --> 00:05:58.470
whole bunch of images and they mapped
a whole bunch of numbers. And let's
82
00:05:58.470 --> 00:06:02.550
say that that mapping is created by
black box, because for all intents and
83
00:06:02.589 --> 00:06:05.980
purposes, that those people think of
it. And then you learn what mapping
84
00:06:06.019 --> 00:06:10.259
goes with what. Then you see
it, see a pedestrian in the world
85
00:06:10.379 --> 00:06:14.459
and you're like, okay, well, which black box corresponds to those pixels
86
00:06:14.459 --> 00:06:15.899
that have seeing in the world?
So one way to think about this is
87
00:06:15.939 --> 00:06:18.569
you take that black box, you
put it on top of the pedestrian and
88
00:06:18.689 --> 00:06:23.410
that black box spits out these numbers, and those numbers are what you can
89
00:06:23.490 --> 00:06:26.889
know. I think probably it's already
a parent that there's a pretty big difference
90
00:06:26.930 --> 00:06:32.279
between between those approaches. And fundamentally, when you're training a machine learning model,
91
00:06:32.319 --> 00:06:39.240
if you are agnostic or if you
don't care about where the mapping between
92
00:06:39.720 --> 00:06:43.639
the images that you have and the
labels comes from, then you're kind of
93
00:06:43.720 --> 00:06:46.040
locked in, like you can have
a black box that's fits out more numbers
94
00:06:46.120 --> 00:06:48.829
you can have a black box.
It's you know, tells you this is
95
00:06:48.910 --> 00:06:53.750
what the person's skeletal polls looks like, this person is carrying a phone,
96
00:06:53.990 --> 00:06:58.629
this person all kinds of external things
about the person that are more or less
97
00:06:58.670 --> 00:07:02.300
helpful. But what it's an impossible
to do actually the traditional supervise learning,
98
00:07:02.339 --> 00:07:08.699
with traditional machine learning, is to
make a useful inference about what's in that
99
00:07:08.860 --> 00:07:13.019
person's head. So when we say
the perceptive a commony removes the black box.
100
00:07:13.540 --> 00:07:15.529
That's what we need. By training
machine learning models the way we do,
101
00:07:16.490 --> 00:07:20.370
we can answer questions that are the
kind of questions a person would answer,
102
00:07:20.410 --> 00:07:25.930
that are questions about what's in that
person's head. And so when you
103
00:07:25.970 --> 00:07:28.889
do that, all of a sudden
you don't have a bunch of black boxes
104
00:07:29.089 --> 00:07:30.639
sitting out numbers, but you can
say, okay, wait, these are
105
00:07:30.800 --> 00:07:35.439
these are people, these are people
with intentions and goals and awareness, and
106
00:07:35.920 --> 00:07:40.319
act around them based on that as
wis. So, at the end of
107
00:07:40.360 --> 00:07:43.920
the day, are you really just
looking at centually people and their actions and
108
00:07:44.040 --> 00:07:47.629
then figuring out those actions are leading
up to certain other actions? Or,
109
00:07:47.870 --> 00:07:50.790
because you talk about being in the
person's head, like how are you connecting
110
00:07:50.829 --> 00:07:54.949
it to actually what's in their head
versus you know their actions, because I
111
00:07:54.949 --> 00:07:58.470
guess that's the ultimate output that you
want, is a better understanding of what
112
00:07:58.819 --> 00:08:01.699
what their actions may be. You
know, based on what's in their head.
113
00:08:01.699 --> 00:08:03.180
Right, yes and no. And
I think to turn that around,
114
00:08:03.500 --> 00:08:07.060
if you think about when you're driving
and you see someone standing by us at
115
00:08:07.139 --> 00:08:09.860
the side of the road and you
say to Yourself, Oh, they're waiting
116
00:08:09.939 --> 00:08:13.410
to see if they can cross in
front of me. How do you know
117
00:08:13.529 --> 00:08:16.889
that? And so the answer is
that we as people, yeah, we're
118
00:08:16.970 --> 00:08:22.050
using visual features that the of the
pedestrian. We're using things about how somebody
119
00:08:22.170 --> 00:08:24.810
looks. Right, we're not talking
to them, we're not we're not just
120
00:08:26.050 --> 00:08:31.199
making a guess, but we are
so good at using those visual features that
121
00:08:31.319 --> 00:08:35.200
it happens essentially unconsciously and what really
pops into our head is, well,
122
00:08:35.279 --> 00:08:39.759
this is what's in that person said. So when we train models, what
123
00:08:39.919 --> 00:08:43.509
we want to do is we want
to have models that first of all pay
124
00:08:43.509 --> 00:08:48.429
attention to all of those visual features
that we, the humans, use but
125
00:08:48.509 --> 00:08:52.149
are unaware of using. So we
want to capture all of those unconscious visual
126
00:08:52.190 --> 00:08:56.100
features. And then the question we're
answering is really it's actually it's it of
127
00:08:56.220 --> 00:09:00.740
course correlates to to actions that the
person's going to take, but it's really
128
00:09:00.820 --> 00:09:03.700
not quite about actions at all.
It's about, very directly, about what's
129
00:09:03.700 --> 00:09:07.179
in that person's head, and probably
the best way to think about it is
130
00:09:07.820 --> 00:09:11.169
we answer the question. Let's say
there's an autnomous vehicle, there's a self
131
00:09:11.210 --> 00:09:15.889
driving car, it's approaching a pedestrian
and the question the answers. Okay,
132
00:09:15.929 --> 00:09:18.210
if there were five hundred people sitting
in this car and you ask them to
133
00:09:18.330 --> 00:09:22.289
take a vote about whether that person
wants to cross the street in front of
134
00:09:22.370 --> 00:09:26.200
you, what would the distribution of
their responses look like? So it's kind
135
00:09:26.240 --> 00:09:31.440
of the it's predicting the the wisdom
of crowds answer to what's in that person's
136
00:09:31.480 --> 00:09:35.600
head. And it turns out when
you're driving a vehicle, that's the question
137
00:09:35.720 --> 00:09:39.669
that you need to answer. I'll
give you an example. Let's say you
138
00:09:39.750 --> 00:09:43.950
have an autronomous vehicle that doesn't have
this ability and there's a person standing by
139
00:09:43.029 --> 00:09:48.549
a crosswalk and that person is motionless, and so the autonomous vehicle says,
140
00:09:48.549 --> 00:09:52.299
okay, I can extrapolate from their
path that their motionless, they're going to
141
00:09:52.299 --> 00:09:56.460
stay motionless and I can drive them, drive past them. In fact,
142
00:09:56.580 --> 00:10:01.100
what was happening was that person really
wanted to cross the street and had the
143
00:10:01.139 --> 00:10:03.700
right of way and would have crossed, but then they were like wait,
144
00:10:03.779 --> 00:10:07.730
that aftonos vehicle isn't slowing down at
all. Did that car make the right
145
00:10:07.809 --> 00:10:11.129
decision? I think pretty clearly the
answer is no, and there's a pissed
146
00:10:11.129 --> 00:10:16.009
off pedestrian and scared passengers in the
car who would both agree that that answer
147
00:10:16.129 --> 00:10:18.690
is no. So really what you
need to know is not where is this
148
00:10:18.889 --> 00:10:24.320
person going to be two seconds from
now, but what's in that that person's
149
00:10:24.360 --> 00:10:28.000
head that I need to think about
when I'm making driving decisions, and that's
150
00:10:28.000 --> 00:10:31.240
what we do. So is there
a lot of regional difference with with this
151
00:10:31.360 --> 00:10:35.360
type of you know, sort of
if you're trying to predict this behavior or
152
00:10:35.399 --> 00:10:39.429
get a better understanding of what's in
their head and their future movements? You
153
00:10:39.509 --> 00:10:41.870
know, is it regional to the
point of within the same country, or
154
00:10:41.909 --> 00:10:45.710
it's regional more based on country to
country? Can you tell a little bit
155
00:10:45.750 --> 00:10:48.509
more about what you've found out there? Yeah, that's a great question.
156
00:10:48.669 --> 00:10:54.299
It varies. It varies both living
countries and between countries, but not necessarily
157
00:10:54.379 --> 00:10:58.139
in the ways that that you would
expect. So we've designed our system kind
158
00:10:58.179 --> 00:11:01.299
of with this in mind. Right. So we knew from the beginning that
159
00:11:01.820 --> 00:11:05.250
if we wanted to deploy this system
and let's say Delhi or Mexico City,
160
00:11:05.289 --> 00:11:09.929
kind of no matter what, the
way that these interactions work is going to
161
00:11:09.970 --> 00:11:13.970
be different from the way that these
interactions work in in Boston or in San
162
00:11:13.009 --> 00:11:20.159
Francisco. What we found is that
that's true, but a much bigger driver
163
00:11:20.080 --> 00:11:24.799
in most places, a much bigger
driver of differences in behavior is the the
164
00:11:24.879 --> 00:11:28.960
built environment. So, for instance, we have models that are trained to
165
00:11:28.120 --> 00:11:33.950
work on the US East Coast,
primarily the data from the US East Coast.
166
00:11:33.190 --> 00:11:39.269
If you take those models to southern
California, they work okay, but
167
00:11:39.350 --> 00:11:43.070
they could actually use tuning. And
the reason and the reason is not necessarily
168
00:11:43.190 --> 00:11:48.139
because people have different kind of social
morays and different ways of understanding each other
169
00:11:48.259 --> 00:11:50.340
in California. All that you know. I lived in California. That certainly
170
00:11:50.700 --> 00:11:54.779
certainly to sundry. That's true,
but a big part of it is just
171
00:11:54.940 --> 00:11:58.220
that the roads are wider and the
speeds are fast so if there's someone who
172
00:11:58.299 --> 00:12:01.409
stay ending in an next to a
crosswalk, that's an unsignalized crosswalk and they're
173
00:12:01.409 --> 00:12:05.970
trying to figure out if they can
cross the street, whether or not they
174
00:12:05.490 --> 00:12:09.129
you know, how they make that
decision varies a lot. If the car
175
00:12:09.409 --> 00:12:13.210
is going at forty five miles an
hour in a four lane road versus twenty
176
00:12:13.250 --> 00:12:16.720
miles an hour on a one way
streat yeah, I can see that definitely,
177
00:12:16.919 --> 00:12:20.879
just the differences between California and Arizona. You know, one of them
178
00:12:20.919 --> 00:12:26.120
the drivers are slowing down seeing you
considering crossing the street, and the other
179
00:12:26.159 --> 00:12:30.070
one they're speeding up because they're seeing
your considering to cross the street and they
180
00:12:30.190 --> 00:12:33.909
want to if you're on the side. One of the things we thought about
181
00:12:33.909 --> 00:12:39.389
explicitly. So we have right now
our model has two outputs, intention,
182
00:12:39.429 --> 00:12:41.149
which would been talking about, which
is, you know, does this person
183
00:12:41.309 --> 00:12:45.779
want a cross in front of your
car, and awareness, and we define
184
00:12:45.820 --> 00:12:50.220
awareness pretty specifically as does this person
know that your car is there? One
185
00:12:50.259 --> 00:12:54.620
of the reasons we did that is
that we knew from experience that when people
186
00:12:54.860 --> 00:13:00.490
cross the street in Boston as a
pedestrian, it's very common to kind of
187
00:13:00.570 --> 00:13:05.570
intentionally not look at the on the
driver of the oncoming car because you're doing
188
00:13:05.690 --> 00:13:09.129
this little dance like obviously you want
to know if they're going to stop,
189
00:13:09.409 --> 00:13:11.769
but you don't want them to know
that they know you're there, because then
190
00:13:11.769 --> 00:13:15.600
they're more likely to stop. And
so kind of picking up those subtle social
191
00:13:15.639 --> 00:13:20.559
dynamics is baked into the way that
we think about asking these questions. That's
192
00:13:20.600 --> 00:13:24.519
really interesting. So talk about psychophysics
a little bit. How does that fit
193
00:13:24.600 --> 00:13:26.230
in? You know, what where
does that term? You know, I'm
194
00:13:26.230 --> 00:13:30.110
about that familiar with the so I'd
love to look here. How this pleason?
195
00:13:30.269 --> 00:13:33.830
Yeah, absolutely, so that that
term. I actually I'll tell you.
196
00:13:35.029 --> 00:13:39.269
I didn't use that word for like
the first three years of the company's
197
00:13:39.269 --> 00:13:41.659
existence because I know like no one
knows what that means. And then it
198
00:13:43.139 --> 00:13:48.299
slipped out in a in one conversation
with with a reporter and he said that's
199
00:13:48.340 --> 00:13:50.539
such a cool word. I was
like, oh well, maybe, all
200
00:13:50.539 --> 00:13:52.179
right, no one knows it means, but it sounds really good. So
201
00:13:52.379 --> 00:13:58.889
okay, let's let's lean into it. So psychophysics is a method in human
202
00:13:58.929 --> 00:14:05.370
behavioral science for studying how people perceive
physical things about in the world. Right,
203
00:14:05.409 --> 00:14:11.840
so it's kind of the psychology of
physical thresholds. And what it really
204
00:14:11.000 --> 00:14:20.000
does is it allows you to take
human vision and explore that using parameterized methods,
205
00:14:20.159 --> 00:14:24.120
using variables, and so, for
instance, if you want to know
206
00:14:24.070 --> 00:14:28.950
how people recognize a face, right, so, people, if you show
207
00:14:28.990 --> 00:14:31.990
people a picture of a face,
they will instantly know it's a face.
208
00:14:31.110 --> 00:14:35.870
They're one hundred percent good at this. You can you can even like Pumpkins,
209
00:14:35.990 --> 00:14:37.110
the kind of look like faces,
look all the faces. So what's
210
00:14:37.110 --> 00:14:41.379
going on? So one thing you
could do in psychophysics he could say,
211
00:14:41.379 --> 00:14:43.580
okay, well, we're going to
take a black bar and we're going to
212
00:14:43.700 --> 00:14:48.899
cover progressively more of the picture with
this black bar and we're going to see
213
00:14:50.019 --> 00:14:54.330
what happens to people's responses. And
then you do the Black Bar, you
214
00:14:54.370 --> 00:14:56.450
know, Miss let's say you do
it from top to bottom, then you
215
00:14:56.490 --> 00:14:58.129
do it from left to bright,
you do one at random angles, and
216
00:14:58.370 --> 00:15:01.850
what you end up with there is
it's you say, okay, well,
217
00:15:01.889 --> 00:15:05.090
by changing this parameter of this this
black bar and how much it's covering and
218
00:15:05.250 --> 00:15:09.720
what angle is at, we can
change how good people are recognizing faces.
219
00:15:11.120 --> 00:15:13.279
And then you can back out of
that and you can say, okay,
220
00:15:13.279 --> 00:15:16.440
well, is that because of how
much of the images being covered? Is
221
00:15:16.600 --> 00:15:22.909
that because specifically the eyes are covered? And so it lets you, without
222
00:15:22.350 --> 00:15:28.629
having a strong prior guess at what
visual features people are using to do something
223
00:15:28.669 --> 00:15:31.350
different. And let's see you it, let's see capture a lot of information
224
00:15:31.350 --> 00:15:35.470
about how people, how people look
at the world. And so when I
225
00:15:35.549 --> 00:15:39.179
talked about the way that we train
models and how we don't treat the the
226
00:15:39.460 --> 00:15:43.100
labelers as a black box, that's
really what I mean. We're using the
227
00:15:43.179 --> 00:15:48.340
techniques of psychophysics in order to extract
all of these visual features that people,
228
00:15:48.379 --> 00:15:52.889
people use to make these suctions about
pedestrians that they couldn't couldn't tell you that
229
00:15:52.970 --> 00:15:56.769
they're using. And so, turning
these knobs and fiddling with these parameters,
230
00:15:56.850 --> 00:16:03.850
let's US capture in a really rich
way that that mapping between making a guess
231
00:16:03.850 --> 00:16:07.080
about what somebody's head in somebody's head
and what that person looks like. It's
232
00:16:07.120 --> 00:16:12.039
interesting. How narrow would you say? Obviously you know you're in one specific
233
00:16:12.159 --> 00:16:15.759
area, but how narrow would you
say are the findings? Are you actually
234
00:16:15.799 --> 00:16:22.149
able to glean additional insights that are
that are apvlicable or useful for other purposes,
235
00:16:22.190 --> 00:16:26.509
or it's really very narrow to a
human mirror street for example, and
236
00:16:26.590 --> 00:16:30.350
doing things that they would do around
a street and other cars. You know.
237
00:16:30.549 --> 00:16:37.500
So the approach is is pretty generally
useful. We are doing level four
238
00:16:37.539 --> 00:16:41.820
robotaxis right now because they really need
this right if you have a if you
239
00:16:41.899 --> 00:16:45.539
have a level four vehicle, and
I think frankly they're they're realizing this.
240
00:16:45.820 --> 00:16:49.929
If you don't have something like what
we're providing, you just can't you can't
241
00:16:51.009 --> 00:16:55.009
deploy in the places where you can
make money. We're talking about Arizona earlier
242
00:16:55.090 --> 00:16:59.330
way moos and Chandler Arizona and and
I think they're making amazing progress and it's
243
00:16:59.409 --> 00:17:03.519
cool deployment. But I think one
of the things that's notable about Chandler is
244
00:17:03.840 --> 00:17:08.319
it's not super pedestrian heavy and that's
very much on purpose. So that need
245
00:17:08.480 --> 00:17:12.759
is very stark in robotaxis, which
for us as a company that really needs
246
00:17:12.799 --> 00:17:15.950
a lot of feedback and a lot
of interaction with customers to learn things,
247
00:17:17.190 --> 00:17:22.910
is very helpful for us. But
this general approach really applies to any kind
248
00:17:22.309 --> 00:17:26.829
of nonverbal social interaction between an autonomous
system in a person. You could think
249
00:17:26.829 --> 00:17:34.059
about warehouse robots, factory robots,
sidewalk vehicles, automated crowd monitoring tools,
250
00:17:34.099 --> 00:17:38.019
all of those, these kind of
insights are important. In terms of the
251
00:17:38.180 --> 00:17:44.259
specific experiments we're doing. I would
say that the insights we glean tend to
252
00:17:44.329 --> 00:17:48.329
be fairly I mean, all we're
looking at is pedestrians and cars, so
253
00:17:48.450 --> 00:17:52.809
they tend to be fairly hide to
that. But we definitely have come up
254
00:17:52.849 --> 00:17:59.000
with insights that were surprising, right, and we've learned things about how people
255
00:17:59.039 --> 00:18:03.759
carry themselves and how people you know, wow, this number of communication works
256
00:18:03.880 --> 00:18:06.599
that never would have occurred to me
in advance. And I can give you
257
00:18:06.720 --> 00:18:10.359
one example that I that I love
to talk about. We, after we
258
00:18:10.519 --> 00:18:12.910
trained our first models, and these
are our intention models, right, whether
259
00:18:12.990 --> 00:18:15.990
someone was watched a cross in front
of the vehicle, we did an exercise
260
00:18:17.190 --> 00:18:22.069
where we went and looked at what
parts of images are models cared about,
261
00:18:22.109 --> 00:18:25.349
so you can, you know,
do it, do kind of it a
262
00:18:25.509 --> 00:18:29.140
heat map and if the neural network
really cares about a certain part of the
263
00:18:29.180 --> 00:18:33.220
eventual light up super bright. What
we noticed is that our models lit up
264
00:18:33.220 --> 00:18:37.460
super bright anytime someone had a bag
on their shoulder. It would really pay
265
00:18:37.460 --> 00:18:41.329
attention to the bag and I we
it was like, well, why does
266
00:18:41.369 --> 00:18:45.170
it? Is this like some weird
thing that we have like a you know
267
00:18:45.210 --> 00:18:48.690
it? Does our training data have
an overrepresentation of people crossing roads with bags
268
00:18:48.849 --> 00:18:52.690
versus not? And what we realized
is, no, it's not that.
269
00:18:53.130 --> 00:18:57.839
But if you think about somebody with
a bag on their shoulder that is thinking
270
00:18:57.920 --> 00:19:02.880
they're about to start crossing the road
and you have that just on one shoulder,
271
00:19:02.920 --> 00:19:04.359
what do you do? Will you
just kind of cinch up ever so
272
00:19:04.599 --> 00:19:10.750
slightly so that when you start moving
the bag doesn't go flying every so there's
273
00:19:10.789 --> 00:19:15.390
this little bit of tightening and it's
a very subtle visual feature. But in
274
00:19:15.549 --> 00:19:21.150
retrospect of course that's super diagnostic and
of course we use that as humans,
275
00:19:21.190 --> 00:19:26.180
right. But that insight really came
out of how we collect this training data,
276
00:19:26.259 --> 00:19:27.900
how we treated the models. If
you if you'd ask me an advance
277
00:19:29.019 --> 00:19:32.779
to, you know, write down
what of the visual features that people use
278
00:19:32.859 --> 00:19:34.579
to make these judgments, I never
would have said, you know, how
279
00:19:34.779 --> 00:19:38.569
how tightly somebody has a bag since
to theirs and their shoulder. No,
280
00:19:38.730 --> 00:19:44.009
that's really interesting. So we talked
about sort of features and how you're analyzing
281
00:19:44.130 --> 00:19:47.369
and using their own network. What
do you think some of the biggest challenges
282
00:19:47.410 --> 00:19:49.809
are on the more of the technical
side for you to be able to really
283
00:19:49.849 --> 00:19:56.599
sort of advance your ai and continue
to evolve your technology? So for us,
284
00:19:56.680 --> 00:19:59.519
I would say first of all,
what we're doing is something that,
285
00:20:00.119 --> 00:20:03.240
and I maybe if I'd realized this
a little bit better before I started the
286
00:20:03.279 --> 00:20:07.039
company, but if there's chance of
visually, but this is very, very
287
00:20:07.150 --> 00:20:11.190
hard. We are taking machine learning
research that's very much at the research frontier
288
00:20:11.470 --> 00:20:15.990
and the point is to take that
and get that all the way to a
289
00:20:15.190 --> 00:20:21.940
module that's reliable enough to be deployed
in a vehicle. And one of the
290
00:20:22.180 --> 00:20:26.019
pieces of that that's really important is
the question that we are asking, and
291
00:20:26.099 --> 00:20:29.940
we talked a little bit about this
earlier, but the question we are asking,
292
00:20:30.019 --> 00:20:34.180
which is what's in that person's head
right now as that relates the task
293
00:20:34.299 --> 00:20:38.890
of driving, is very different than
the kinds of questions are customers have typically
294
00:20:38.930 --> 00:20:42.690
ask so for us, we've had
to do a lot of work and build
295
00:20:42.690 --> 00:20:48.130
up a lot of expertise so that
we can make sure that answering that question
296
00:20:48.210 --> 00:20:52.160
in these sort of critical moments when
you have a difficult interaction of with a
297
00:20:52.200 --> 00:20:59.599
pedestrian is reliably and correctly the information
that you need to change the behavior in
298
00:20:59.680 --> 00:21:03.039
the vehicle. So what that means
in practice is we really have to be
299
00:21:03.160 --> 00:21:07.910
the experts in how you integrate a
system like this, how you use the
300
00:21:07.990 --> 00:21:12.670
outputs, how it should modify the
motion plan, even questions what the motion
301
00:21:12.750 --> 00:21:17.470
plan of the vehicle should be.
We've had to develop a lot of expertise
302
00:21:17.549 --> 00:21:21.339
in those and so I think that
as far as the core model goes,
303
00:21:21.900 --> 00:21:23.220
it does what we say. That's
right, like when I gave you that
304
00:21:23.500 --> 00:21:26.579
that five hundred people in the car
kind of analogy. That there really is
305
00:21:26.740 --> 00:21:30.579
what it does and it really does
do that with a high degrade fidelity.
306
00:21:30.099 --> 00:21:33.130
But making sure that then we can
take that, we can say to our
307
00:21:33.210 --> 00:21:37.089
customers, okay, this is where
you plug it in, this is how
308
00:21:37.250 --> 00:21:38.650
you plug it in, this is
what the output should look like, and
309
00:21:38.690 --> 00:21:41.970
if the output looks like that,
looks like that, you absolutely can rely
310
00:21:42.049 --> 00:21:47.400
on it. Has Been a substantial
amount of effort and something that where where
311
00:21:47.440 --> 00:21:49.680
we we know how to do it
now, but making sure that we can
312
00:21:49.759 --> 00:21:55.039
do that broadly is kind of the
next step of our commercial relationships. Yeah,
313
00:21:55.079 --> 00:21:59.200
I mean it always comes down to
implementation and sort of usage, and
314
00:21:59.599 --> 00:22:03.269
it's great. You know, I've
started a lot of companies and often times
315
00:22:03.269 --> 00:22:07.829
it's using data science to find some
sort of opportunity. And you know,
316
00:22:07.869 --> 00:22:10.910
you always think about it in the
abstracts where, once we get this in
317
00:22:10.990 --> 00:22:12.990
the market, you know it's amazing, it works really well, but then
318
00:22:14.029 --> 00:22:18.140
there's only the other pieces and if
for it to really work, it's all
319
00:22:18.140 --> 00:22:19.980
about that integration, and what you
guys are doing is so much more.
320
00:22:21.180 --> 00:22:23.220
You know, they, the teams
are working without to understand what you're doing
321
00:22:23.299 --> 00:22:26.059
as much as you have to understand
what they're doing so that it can actually
322
00:22:26.059 --> 00:22:29.779
do what it's supposed to. Absolutely
and I think that, you know,
323
00:22:29.859 --> 00:22:33.890
the good part is that we've had
a lot of time and a lot of
324
00:22:33.970 --> 00:22:38.569
iteration with customers to figure that out
right. So we have those answers now.
325
00:22:38.809 --> 00:22:41.490
The other part, I would say, I don't know that this is
326
00:22:41.569 --> 00:22:45.000
universally good, but it's helpful to
us, is that you still you need
327
00:22:45.200 --> 00:22:49.599
this right. So not just us
but our customers really need the answer to
328
00:22:49.759 --> 00:22:53.519
these questions, because without answering the
question, it had being able to answer
329
00:22:53.559 --> 00:22:57.589
questions very much like the ones that
were answering. You do nothing else works.
330
00:22:57.710 --> 00:23:00.869
So so there's a lot of incentive
on both sides and we've been able
331
00:23:00.910 --> 00:23:03.589
to do a lot of work to
build out the infrastructor for that. But
332
00:23:03.750 --> 00:23:07.309
it absolutely you know, very early
on we would go to customers and they
333
00:23:07.309 --> 00:23:10.750
would say, well, how do
we integrate this? Now will be like
334
00:23:10.869 --> 00:23:12.619
well, and however you want,
and it's not great because you know,
335
00:23:14.019 --> 00:23:17.740
what do you think like? I
don't know. So kind of kind of
336
00:23:17.819 --> 00:23:22.859
building out that and really a big
part of our team now is oriented around
337
00:23:22.299 --> 00:23:26.019
making sure we can answer those kinds
of questions for customers because because certainly that's
338
00:23:26.099 --> 00:23:30.769
what appropriately they need from us.
Yeah, that's great. Before we go,
339
00:23:30.890 --> 00:23:33.369
I would like to ask you,
you know, what sort of guidance
340
00:23:33.410 --> 00:23:37.210
or advice would you give to someone
who's seeking your career really in data science
341
00:23:37.250 --> 00:23:41.559
and engineering other specific things? From
your background? You have a very unique
342
00:23:41.559 --> 00:23:45.240
experience, clearly as far as great
other things, but I'd look to hear
343
00:23:45.279 --> 00:23:48.839
from you if you have some some
guidance to people who want to get people
344
00:23:48.920 --> 00:23:52.039
into it. Yeah, well,
I'll tell you. So, Emman,
345
00:23:52.160 --> 00:23:55.519
I'm going to say something and I'm
not completely sure that this is actually good
346
00:23:55.680 --> 00:24:03.549
practical advice, but it's my my
strongest intuition about what's happening now in machine
347
00:24:03.589 --> 00:24:10.339
learning and AI and computer vision is
that the hard problems are no longer computer
348
00:24:10.460 --> 00:24:15.299
science problems. So if you think
about training a deep network, and we
349
00:24:15.380 --> 00:24:18.460
get lots of people applying and they're
applying because they want to train deep networks,
350
00:24:18.579 --> 00:24:21.779
frankly, training a deep network per
se is pretty easy. You just
351
00:24:21.940 --> 00:24:23.529
you to give it a data that
you feed it to it. You'd learn
352
00:24:23.609 --> 00:24:27.170
how to evaluate it, you learn
how to do your optimizations later. It's,
353
00:24:27.609 --> 00:24:30.009
you know, kind of a kind
of a there's kind of a cookbook
354
00:24:30.049 --> 00:24:33.450
for it and it's not that hard. On the other hand, knowing how
355
00:24:33.650 --> 00:24:38.400
to ask questions that other people don't
know how to ask the data and knowing
356
00:24:38.519 --> 00:24:44.880
how to think about data in terms
of when you need quality, what quality
357
00:24:44.960 --> 00:24:48.359
means, and sort of thinking about
the rest of the workflow outside of the
358
00:24:48.440 --> 00:24:52.509
model training per se. There's some
very, very interesting and very, very
359
00:24:52.630 --> 00:24:59.710
hard under explore questions in that and
those really aren't computer science questions. They've
360
00:24:59.710 --> 00:25:04.589
resolved to behavioral science questions, to
infrastructure questions. So so I would say
361
00:25:06.150 --> 00:25:08.140
my advice, such as it is, is to have kind of more than
362
00:25:08.259 --> 00:25:12.299
one don't just know about machine learning, don't just learn about data science,
363
00:25:12.460 --> 00:25:17.660
have something else you know and other
things you know about the kind of number
364
00:25:18.099 --> 00:25:21.849
of the questions and the answers that
people are likely to be interested in.
365
00:25:22.009 --> 00:25:23.730
Yeah, well, that's great.
Yeah, always bringing more to the table
366
00:25:23.890 --> 00:25:26.809
helps and get a lot of different
situations. So thank you for that.
367
00:25:27.609 --> 00:25:30.250
Thank you for me and guess in
the show today. We really appreciate it.
368
00:25:30.490 --> 00:25:33.490
Where can the community find you?
What's the best way for the data
369
00:25:33.529 --> 00:25:37.240
science community to sort of reach out
and connect? Absolutely, I mean I
370
00:25:37.279 --> 00:25:41.440
would say check us, check out
at ewwwcers uptive up commentcom. We have
371
00:25:41.680 --> 00:25:45.759
a lot of information up there,
including some some nice visualizations. Is The
372
00:25:45.799 --> 00:25:49.069
black box metaphor we discussed, as
well as job opportunities and and a bunch
373
00:25:49.069 --> 00:25:52.589
of different opporty ways to learn more. Okay, well, great. At
374
00:25:52.630 --> 00:25:56.630
the data standard we are community of
date enthusiasts and data thought leaders. Were
375
00:25:56.710 --> 00:26:00.670
we look forward to engaging with you
on our website, data standard dot Ioh,
376
00:26:00.109 --> 00:26:04.299
where we regularly post a wide range
of podcasts with data executives and offer
377
00:26:04.500 --> 00:26:08.380
regular networking opportunities or more information,
visit the website. Thanks so much,
378
00:26:08.380 --> 00:26:11.299
Sam for joining us today. Thanks
for having
1
00:00:10.310 --> 00:00:13.949
Welcome to the BIGGA standard experience.
I'm your guest host for today, getting
2
00:00:13.990 --> 00:00:18.910
in Reuben. I'm the founder and
CEO of pandio. On our show today
3
00:00:18.949 --> 00:00:23.780
we have Sam Anthony, the CTEO
and Co founder of perceptive automats. Up
4
00:00:23.780 --> 00:00:27.460
Today. We are speaking about the
missing piece for autonomous driving. Welcome to
5
00:00:27.500 --> 00:00:30.100
the show. Thanks for having me. Great I'd love to hear a little
6
00:00:30.100 --> 00:00:34.100
bit more about your background, if
you wouldn't mind starting their sure. So
7
00:00:34.659 --> 00:00:40.530
perceptive ATOMEDA came out of a bunch
of research projects that my cofounders and myself
8
00:00:40.729 --> 00:00:46.130
did when I was a Grad student
actually at Harvard and Vision Sciences Laboratory Harbor.
9
00:00:46.329 --> 00:00:49.770
My background, it's kind of a
stint as a computer hacker, which
10
00:00:49.770 --> 00:00:52.520
we can talk about if you really
want to. But after that my background
11
00:00:52.560 --> 00:00:57.280
is in before Grad School was in
Web Development and and web applications, and
12
00:00:57.399 --> 00:01:03.679
when I got to Harvard, what
I was working on was doing careful psychological
13
00:01:03.799 --> 00:01:10.349
experiments, so careful human behavioral experiments
at Web scale, and this is something
14
00:01:10.390 --> 00:01:12.109
you know. People Talk About Mechanical
Turk and that's kind of a way you
15
00:01:12.310 --> 00:01:19.299
can do this, but what we
pioneered was really taking seriously being doing good,
16
00:01:19.900 --> 00:01:26.180
careful vision science experiments, and what
my cofounders gave Cox and Walter Shire
17
00:01:26.299 --> 00:01:33.049
and I realized was that what I
was doing with these experiments was happening in
18
00:01:33.129 --> 00:01:36.170
a scale where we could start thinking
about using it for machine learning, and
19
00:01:36.969 --> 00:01:41.209
that led us to kind of a
follow up realization that the way everyone else
20
00:01:41.290 --> 00:01:44.569
did machine learning was really missing at
trick, because the way everyone else does
21
00:01:44.689 --> 00:01:51.280
machine learning they think of the labeling
as just this kind of like noisy process
22
00:01:51.319 --> 00:01:56.760
that happens in advance where you're trying
to get your training samples, your things
23
00:01:56.799 --> 00:02:00.280
that you're going to feed your machine
learning model, tagged with like a fact
24
00:02:00.319 --> 00:02:04.950
about the world, and we'll really
realize that that that that process of labeling
25
00:02:05.430 --> 00:02:09.990
was really, in many ways,
exactly the same as running a psychological discernment,
26
00:02:10.069 --> 00:02:14.229
except in the psychological experiment you can
learn much, much more about how
27
00:02:14.270 --> 00:02:19.659
people look at the world. So
we kind of combined these insights from computer
28
00:02:19.740 --> 00:02:23.099
vision and from humanive behavioral science and
ended up with a with a with an
29
00:02:23.099 --> 00:02:30.849
approach for the methodology where that methodology
is capable of building machine learning models that
30
00:02:31.050 --> 00:02:36.729
will answer questions and kind of the
specific category and specific categories things that people
31
00:02:36.770 --> 00:02:40.969
are effortlessly good at and computers are
not good at, and so that's a
32
00:02:42.009 --> 00:02:45.680
pretty useful a pretty interesting set of
problems. And so that's kind of where
33
00:02:45.680 --> 00:02:49.479
we started and after quite a bit
of iteration we got to the problem that
34
00:02:49.599 --> 00:02:53.560
for solving with perceptive Automata, which
is understanding humans in the world from the
35
00:02:53.680 --> 00:02:59.669
perspective of autonomous vehicles, in the
autos robots. In Graduate School. Was
36
00:02:59.710 --> 00:03:04.389
that your first for into the Book
Behavioral Science or had you been studying that
37
00:03:04.550 --> 00:03:07.750
earlier? I studied it an Undergrad. So I had a super weird,
38
00:03:07.909 --> 00:03:13.780
like totally not scalable career path.
I directly out of High School for went
39
00:03:13.939 --> 00:03:16.580
college because that was the the firstcom
boom and I was like, I'll just
40
00:03:16.620 --> 00:03:20.539
go make a lot of money doing
you know, they'll pay me how much
41
00:03:20.580 --> 00:03:23.819
to make a web page, and
I so I did that and I kind
42
00:03:23.819 --> 00:03:27.849
of moved through tech and had kind
of a initial career in tech for about
43
00:03:27.930 --> 00:03:31.289
ten years and I left that because
it felt like the work that I was
44
00:03:31.330 --> 00:03:37.289
doing building web applications was getting commoditized
and went back to college and had the
45
00:03:37.370 --> 00:03:44.199
chance to sort of pick between the
University of California campuses and the way it
46
00:03:44.280 --> 00:03:47.080
work was transferring into the school.
So I had to pick a major to
47
00:03:47.159 --> 00:03:52.439
transfer in and and at every UC
campus except for one, I said I
48
00:03:52.439 --> 00:03:55.389
would major in math. And then, for whatever reason, UCSD I was
49
00:03:55.469 --> 00:03:59.469
like, Oh, there, I
heard their cognitive science program is pretty good.
50
00:03:59.509 --> 00:04:01.229
I'll put down cognitive science on this
one. And I got in not
51
00:04:01.349 --> 00:04:04.870
everywhere, but almost everywhere, and
realize I absolutely did not want to be
52
00:04:04.870 --> 00:04:08.990
a math major, and so I
ended up in the cognitive science program at
53
00:04:09.270 --> 00:04:12.780
UCSD and everything kind of everything kind
of float from their nice. Yeah,
54
00:04:12.780 --> 00:04:15.379
UCS. These actually down the street
from us. I'm in the Carmel Vallege.
55
00:04:15.459 --> 00:04:17.699
Oh, excellent, excellent, excellent. It's I'll tell you what.
56
00:04:17.779 --> 00:04:23.139
I having moved back to Boston,
the I have regrets about the the weather
57
00:04:23.180 --> 00:04:26.329
difference. Well, I do the
reverse, you know. I went to
58
00:04:26.850 --> 00:04:30.449
the College at Brandise University. Of
My brother was a ucks so I spent
59
00:04:30.529 --> 00:04:33.089
some time in summer go as well, you know. And but I grew
60
00:04:33.129 --> 00:04:36.689
up in Arizona. So I went
from Arizona to Boston and I knew,
61
00:04:36.730 --> 00:04:40.399
Oh boyd want to be an Arizona
that. I knew I didn't want to
62
00:04:40.399 --> 00:04:45.439
be a Boston. Yeah, so
I'd love to get in a little more
63
00:04:45.480 --> 00:04:47.240
about what you're working on now.
So can you tell us a little bit
64
00:04:47.279 --> 00:04:51.839
about the black box metaphor and how
this relates to what you're doing? Absolutely,
65
00:04:51.990 --> 00:04:57.310
so, what we're our system does, is our system allows and a
66
00:04:57.389 --> 00:05:01.589
ton of US vehicle to know what's
in the heads of pedestrians. And that's
67
00:05:01.629 --> 00:05:04.269
it like a weird thing to say, right, so, because what's in
68
00:05:04.389 --> 00:05:06.180
my head? Well, it's in
my head, you know. You know,
69
00:05:06.220 --> 00:05:09.579
there's not a telepathy right, how
are you going to figure that out?
70
00:05:09.620 --> 00:05:12.540
But we, as humans, were
very, very social and or very
71
00:05:12.620 --> 00:05:16.579
good at social reasoning. So when
we look at another human we've very quickly
72
00:05:16.740 --> 00:05:20.730
come up with a whole set of
inferences about what's in their head, what
73
00:05:20.850 --> 00:05:26.529
they're thinking, what they want and
whether or not we're right in some absolute
74
00:05:26.649 --> 00:05:30.449
sense, we're right enough to act
on it and for it to for it
75
00:05:30.529 --> 00:05:33.410
to lead to such behavior. Right. By contrast, when you train a
76
00:05:33.529 --> 00:05:39.240
traditional machine learning model, what that
machine learning model is doing is it's trying
77
00:05:39.319 --> 00:05:45.199
to extract a mapping between a set
of pixels and a set of numbers.
78
00:05:45.600 --> 00:05:48.839
Where that mapping comes from, you
don't care. It's irrelevant. What you're
79
00:05:48.839 --> 00:05:51.990
really trying to do is you're trying
to you have some mapping, and again
80
00:05:53.069 --> 00:05:55.629
this goes back to the training process
and your training data. You have a
81
00:05:55.709 --> 00:05:58.470
whole bunch of images and they mapped
a whole bunch of numbers. And let's
82
00:05:58.470 --> 00:06:02.550
say that that mapping is created by
black box, because for all intents and
83
00:06:02.589 --> 00:06:05.980
purposes, that those people think of
it. And then you learn what mapping
84
00:06:06.019 --> 00:06:10.259
goes with what. Then you see
it, see a pedestrian in the world
85
00:06:10.379 --> 00:06:14.459
and you're like, okay, well, which black box corresponds to those pixels
86
00:06:14.459 --> 00:06:15.899
that have seeing in the world?
So one way to think about this is
87
00:06:15.939 --> 00:06:18.569
you take that black box, you
put it on top of the pedestrian and
88
00:06:18.689 --> 00:06:23.410
that black box spits out these numbers, and those numbers are what you can
89
00:06:23.490 --> 00:06:26.889
know. I think probably it's already
a parent that there's a pretty big difference
90
00:06:26.930 --> 00:06:32.279
between between those approaches. And fundamentally, when you're training a machine learning model,
91
00:06:32.319 --> 00:06:39.240
if you are agnostic or if you
don't care about where the mapping between
92
00:06:39.720 --> 00:06:43.639
the images that you have and the
labels comes from, then you're kind of
93
00:06:43.720 --> 00:06:46.040
locked in, like you can have
a black box that's fits out more numbers
94
00:06:46.120 --> 00:06:48.829
you can have a black box.
It's you know, tells you this is
95
00:06:48.910 --> 00:06:53.750
what the person's skeletal polls looks like, this person is carrying a phone,
96
00:06:53.990 --> 00:06:58.629
this person all kinds of external things
about the person that are more or less
97
00:06:58.670 --> 00:07:02.300
helpful. But what it's an impossible
to do actually the traditional supervise learning,
98
00:07:02.339 --> 00:07:08.699
with traditional machine learning, is to
make a useful inference about what's in that
99
00:07:08.860 --> 00:07:13.019
person's head. So when we say
the perceptive a commony removes the black box.
100
00:07:13.540 --> 00:07:15.529
That's what we need. By training
machine learning models the way we do,
101
00:07:16.490 --> 00:07:20.370
we can answer questions that are the
kind of questions a person would answer,
102
00:07:20.410 --> 00:07:25.930
that are questions about what's in that
person's head. And so when you
103
00:07:25.970 --> 00:07:28.889
do that, all of a sudden
you don't have a bunch of black boxes
104
00:07:29.089 --> 00:07:30.639
sitting out numbers, but you can
say, okay, wait, these are
105
00:07:30.800 --> 00:07:35.439
these are people, these are people
with intentions and goals and awareness, and
106
00:07:35.920 --> 00:07:40.319
act around them based on that as
wis. So, at the end of
107
00:07:40.360 --> 00:07:43.920
the day, are you really just
looking at centually people and their actions and
108
00:07:44.040 --> 00:07:47.629
then figuring out those actions are leading
up to certain other actions? Or,
109
00:07:47.870 --> 00:07:50.790
because you talk about being in the
person's head, like how are you connecting
110
00:07:50.829 --> 00:07:54.949
it to actually what's in their head
versus you know their actions, because I
111
00:07:54.949 --> 00:07:58.470
guess that's the ultimate output that you
want, is a better understanding of what
112
00:07:58.819 --> 00:08:01.699
what their actions may be. You
know, based on what's in their head.
113
00:08:01.699 --> 00:08:03.180
Right, yes and no. And
I think to turn that around,
114
00:08:03.500 --> 00:08:07.060
if you think about when you're driving
and you see someone standing by us at
115
00:08:07.139 --> 00:08:09.860
the side of the road and you
say to Yourself, Oh, they're waiting
116
00:08:09.939 --> 00:08:13.410
to see if they can cross in
front of me. How do you know
117
00:08:13.529 --> 00:08:16.889
that? And so the answer is
that we as people, yeah, we're
118
00:08:16.970 --> 00:08:22.050
using visual features that the of the
pedestrian. We're using things about how somebody
119
00:08:22.170 --> 00:08:24.810
looks. Right, we're not talking
to them, we're not we're not just
120
00:08:26.050 --> 00:08:31.199
making a guess, but we are
so good at using those visual features that
121
00:08:31.319 --> 00:08:35.200
it happens essentially unconsciously and what really
pops into our head is, well,
122
00:08:35.279 --> 00:08:39.759
this is what's in that person said. So when we train models, what
123
00:08:39.919 --> 00:08:43.509
we want to do is we want
to have models that first of all pay
124
00:08:43.509 --> 00:08:48.429
attention to all of those visual features
that we, the humans, use but
125
00:08:48.509 --> 00:08:52.149
are unaware of using. So we
want to capture all of those unconscious visual
126
00:08:52.190 --> 00:08:56.100
features. And then the question we're
answering is really it's actually it's it of
127
00:08:56.220 --> 00:09:00.740
course correlates to to actions that the
person's going to take, but it's really
128
00:09:00.820 --> 00:09:03.700
not quite about actions at all.
It's about, very directly, about what's
129
00:09:03.700 --> 00:09:07.179
in that person's head, and probably
the best way to think about it is
130
00:09:07.820 --> 00:09:11.169
we answer the question. Let's say
there's an autnomous vehicle, there's a self
131
00:09:11.210 --> 00:09:15.889
driving car, it's approaching a pedestrian
and the question the answers. Okay,
132
00:09:15.929 --> 00:09:18.210
if there were five hundred people sitting
in this car and you ask them to
133
00:09:18.330 --> 00:09:22.289
take a vote about whether that person
wants to cross the street in front of
134
00:09:22.370 --> 00:09:26.200
you, what would the distribution of
their responses look like? So it's kind
135
00:09:26.240 --> 00:09:31.440
of the it's predicting the the wisdom
of crowds answer to what's in that person's
136
00:09:31.480 --> 00:09:35.600
head. And it turns out when
you're driving a vehicle, that's the question
137
00:09:35.720 --> 00:09:39.669
that you need to answer. I'll
give you an example. Let's say you
138
00:09:39.750 --> 00:09:43.950
have an autronomous vehicle that doesn't have
this ability and there's a person standing by
139
00:09:43.029 --> 00:09:48.549
a crosswalk and that person is motionless, and so the autonomous vehicle says,
140
00:09:48.549 --> 00:09:52.299
okay, I can extrapolate from their
path that their motionless, they're going to
141
00:09:52.299 --> 00:09:56.460
stay motionless and I can drive them, drive past them. In fact,
142
00:09:56.580 --> 00:10:01.100
what was happening was that person really
wanted to cross the street and had the
143
00:10:01.139 --> 00:10:03.700
right of way and would have crossed, but then they were like wait,
144
00:10:03.779 --> 00:10:07.730
that aftonos vehicle isn't slowing down at
all. Did that car make the right
145
00:10:07.809 --> 00:10:11.129
decision? I think pretty clearly the
answer is no, and there's a pissed
146
00:10:11.129 --> 00:10:16.009
off pedestrian and scared passengers in the
car who would both agree that that answer
147
00:10:16.129 --> 00:10:18.690
is no. So really what you
need to know is not where is this
148
00:10:18.889 --> 00:10:24.320
person going to be two seconds from
now, but what's in that that person's
149
00:10:24.360 --> 00:10:28.000
head that I need to think about
when I'm making driving decisions, and that's
150
00:10:28.000 --> 00:10:31.240
what we do. So is there
a lot of regional difference with with this
151
00:10:31.360 --> 00:10:35.360
type of you know, sort of
if you're trying to predict this behavior or
152
00:10:35.399 --> 00:10:39.429
get a better understanding of what's in
their head and their future movements? You
153
00:10:39.509 --> 00:10:41.870
know, is it regional to the
point of within the same country, or
154
00:10:41.909 --> 00:10:45.710
it's regional more based on country to
country? Can you tell a little bit
155
00:10:45.750 --> 00:10:48.509
more about what you've found out there? Yeah, that's a great question.
156
00:10:48.669 --> 00:10:54.299
It varies. It varies both living
countries and between countries, but not necessarily
157
00:10:54.379 --> 00:10:58.139
in the ways that that you would
expect. So we've designed our system kind
158
00:10:58.179 --> 00:11:01.299
of with this in mind. Right. So we knew from the beginning that
159
00:11:01.820 --> 00:11:05.250
if we wanted to deploy this system
and let's say Delhi or Mexico City,
160
00:11:05.289 --> 00:11:09.929
kind of no matter what, the
way that these interactions work is going to
161
00:11:09.970 --> 00:11:13.970
be different from the way that these
interactions work in in Boston or in San
162
00:11:13.009 --> 00:11:20.159
Francisco. What we found is that
that's true, but a much bigger driver
163
00:11:20.080 --> 00:11:24.799
in most places, a much bigger
driver of differences in behavior is the the
164
00:11:24.879 --> 00:11:28.960
built environment. So, for instance, we have models that are trained to
165
00:11:28.120 --> 00:11:33.950
work on the US East Coast,
primarily the data from the US East Coast.
166
00:11:33.190 --> 00:11:39.269
If you take those models to southern
California, they work okay, but
167
00:11:39.350 --> 00:11:43.070
they could actually use tuning. And
the reason and the reason is not necessarily
168
00:11:43.190 --> 00:11:48.139
because people have different kind of social
morays and different ways of understanding each other
169
00:11:48.259 --> 00:11:50.340
in California. All that you know. I lived in California. That certainly
170
00:11:50.700 --> 00:11:54.779
certainly to sundry. That's true,
but a big part of it is just
171
00:11:54.940 --> 00:11:58.220
that the roads are wider and the
speeds are fast so if there's someone who
172
00:11:58.299 --> 00:12:01.409
stay ending in an next to a
crosswalk, that's an unsignalized crosswalk and they're
173
00:12:01.409 --> 00:12:05.970
trying to figure out if they can
cross the street, whether or not they
174
00:12:05.490 --> 00:12:09.129
you know, how they make that
decision varies a lot. If the car
175
00:12:09.409 --> 00:12:13.210
is going at forty five miles an
hour in a four lane road versus twenty
176
00:12:13.250 --> 00:12:16.720
miles an hour on a one way
streat yeah, I can see that definitely,
177
00:12:16.919 --> 00:12:20.879
just the differences between California and Arizona. You know, one of them
178
00:12:20.919 --> 00:12:26.120
the drivers are slowing down seeing you
considering crossing the street, and the other
179
00:12:26.159 --> 00:12:30.070
one they're speeding up because they're seeing
your considering to cross the street and they
180
00:12:30.190 --> 00:12:33.909
want to if you're on the side. One of the things we thought about
181
00:12:33.909 --> 00:12:39.389
explicitly. So we have right now
our model has two outputs, intention,
182
00:12:39.429 --> 00:12:41.149
which would been talking about, which
is, you know, does this person
183
00:12:41.309 --> 00:12:45.779
want a cross in front of your
car, and awareness, and we define
184
00:12:45.820 --> 00:12:50.220
awareness pretty specifically as does this person
know that your car is there? One
185
00:12:50.259 --> 00:12:54.620
of the reasons we did that is
that we knew from experience that when people
186
00:12:54.860 --> 00:13:00.490
cross the street in Boston as a
pedestrian, it's very common to kind of
187
00:13:00.570 --> 00:13:05.570
intentionally not look at the on the
driver of the oncoming car because you're doing
188
00:13:05.690 --> 00:13:09.129
this little dance like obviously you want
to know if they're going to stop,
189
00:13:09.409 --> 00:13:11.769
but you don't want them to know
that they know you're there, because then
190
00:13:11.769 --> 00:13:15.600
they're more likely to stop. And
so kind of picking up those subtle social
191
00:13:15.639 --> 00:13:20.559
dynamics is baked into the way that
we think about asking these questions. That's
192
00:13:20.600 --> 00:13:24.519
really interesting. So talk about psychophysics
a little bit. How does that fit
193
00:13:24.600 --> 00:13:26.230
in? You know, what where
does that term? You know, I'm
194
00:13:26.230 --> 00:13:30.110
about that familiar with the so I'd
love to look here. How this pleason?
195
00:13:30.269 --> 00:13:33.830
Yeah, absolutely, so that that
term. I actually I'll tell you.
196
00:13:35.029 --> 00:13:39.269
I didn't use that word for like
the first three years of the company's
197
00:13:39.269 --> 00:13:41.659
existence because I know like no one
knows what that means. And then it
198
00:13:43.139 --> 00:13:48.299
slipped out in a in one conversation
with with a reporter and he said that's
199
00:13:48.340 --> 00:13:50.539
such a cool word. I was
like, oh well, maybe, all
200
00:13:50.539 --> 00:13:52.179
right, no one knows it means, but it sounds really good. So
201
00:13:52.379 --> 00:13:58.889
okay, let's let's lean into it. So psychophysics is a method in human
202
00:13:58.929 --> 00:14:05.370
behavioral science for studying how people perceive
physical things about in the world. Right,
203
00:14:05.409 --> 00:14:11.840
so it's kind of the psychology of
physical thresholds. And what it really
204
00:14:11.000 --> 00:14:20.000
does is it allows you to take
human vision and explore that using parameterized methods,
205
00:14:20.159 --> 00:14:24.120
using variables, and so, for
instance, if you want to know
206
00:14:24.070 --> 00:14:28.950
how people recognize a face, right, so, people, if you show
207
00:14:28.990 --> 00:14:31.990
people a picture of a face,
they will instantly know it's a face.
208
00:14:31.110 --> 00:14:35.870
They're one hundred percent good at this. You can you can even like Pumpkins,
209
00:14:35.990 --> 00:14:37.110
the kind of look like faces,
look all the faces. So what's
210
00:14:37.110 --> 00:14:41.379
going on? So one thing you
could do in psychophysics he could say,
211
00:14:41.379 --> 00:14:43.580
okay, well, we're going to
take a black bar and we're going to
212
00:14:43.700 --> 00:14:48.899
cover progressively more of the picture with
this black bar and we're going to see
213
00:14:50.019 --> 00:14:54.330
what happens to people's responses. And
then you do the Black Bar, you
214
00:14:54.370 --> 00:14:56.450
know, Miss let's say you do
it from top to bottom, then you
215
00:14:56.490 --> 00:14:58.129
do it from left to bright,
you do one at random angles, and
216
00:14:58.370 --> 00:15:01.850
what you end up with there is
it's you say, okay, well,
217
00:15:01.889 --> 00:15:05.090
by changing this parameter of this this
black bar and how much it's covering and
218
00:15:05.250 --> 00:15:09.720
what angle is at, we can
change how good people are recognizing faces.
219
00:15:11.120 --> 00:15:13.279
And then you can back out of
that and you can say, okay,
220
00:15:13.279 --> 00:15:16.440
well, is that because of how
much of the images being covered? Is
221
00:15:16.600 --> 00:15:22.909
that because specifically the eyes are covered? And so it lets you, without
222
00:15:22.350 --> 00:15:28.629
having a strong prior guess at what
visual features people are using to do something
223
00:15:28.669 --> 00:15:31.350
different. And let's see you it, let's see capture a lot of information
224
00:15:31.350 --> 00:15:35.470
about how people, how people look
at the world. And so when I
225
00:15:35.549 --> 00:15:39.179
talked about the way that we train
models and how we don't treat the the
226
00:15:39.460 --> 00:15:43.100
labelers as a black box, that's
really what I mean. We're using the
227
00:15:43.179 --> 00:15:48.340
techniques of psychophysics in order to extract
all of these visual features that people,
228
00:15:48.379 --> 00:15:52.889
people use to make these suctions about
pedestrians that they couldn't couldn't tell you that
229
00:15:52.970 --> 00:15:56.769
they're using. And so, turning
these knobs and fiddling with these parameters,
230
00:15:56.850 --> 00:16:03.850
let's US capture in a really rich
way that that mapping between making a guess
231
00:16:03.850 --> 00:16:07.080
about what somebody's head in somebody's head
and what that person looks like. It's
232
00:16:07.120 --> 00:16:12.039
interesting. How narrow would you say? Obviously you know you're in one specific
233
00:16:12.159 --> 00:16:15.759
area, but how narrow would you
say are the findings? Are you actually
234
00:16:15.799 --> 00:16:22.149
able to glean additional insights that are
that are apvlicable or useful for other purposes,
235
00:16:22.190 --> 00:16:26.509
or it's really very narrow to a
human mirror street for example, and
236
00:16:26.590 --> 00:16:30.350
doing things that they would do around
a street and other cars. You know.
237
00:16:30.549 --> 00:16:37.500
So the approach is is pretty generally
useful. We are doing level four
238
00:16:37.539 --> 00:16:41.820
robotaxis right now because they really need
this right if you have a if you
239
00:16:41.899 --> 00:16:45.539
have a level four vehicle, and
I think frankly they're they're realizing this.
240
00:16:45.820 --> 00:16:49.929
If you don't have something like what
we're providing, you just can't you can't
241
00:16:51.009 --> 00:16:55.009
deploy in the places where you can
make money. We're talking about Arizona earlier
242
00:16:55.090 --> 00:16:59.330
way moos and Chandler Arizona and and
I think they're making amazing progress and it's
243
00:16:59.409 --> 00:17:03.519
cool deployment. But I think one
of the things that's notable about Chandler is
244
00:17:03.840 --> 00:17:08.319
it's not super pedestrian heavy and that's
very much on purpose. So that need
245
00:17:08.480 --> 00:17:12.759
is very stark in robotaxis, which
for us as a company that really needs
246
00:17:12.799 --> 00:17:15.950
a lot of feedback and a lot
of interaction with customers to learn things,
247
00:17:17.190 --> 00:17:22.910
is very helpful for us. But
this general approach really applies to any kind
248
00:17:22.309 --> 00:17:26.829
of nonverbal social interaction between an autonomous
system in a person. You could think
249
00:17:26.829 --> 00:17:34.059
about warehouse robots, factory robots,
sidewalk vehicles, automated crowd monitoring tools,
250
00:17:34.099 --> 00:17:38.019
all of those, these kind of
insights are important. In terms of the
251
00:17:38.180 --> 00:17:44.259
specific experiments we're doing. I would
say that the insights we glean tend to
252
00:17:44.329 --> 00:17:48.329
be fairly I mean, all we're
looking at is pedestrians and cars, so
253
00:17:48.450 --> 00:17:52.809
they tend to be fairly hide to
that. But we definitely have come up
254
00:17:52.849 --> 00:17:59.000
with insights that were surprising, right, and we've learned things about how people
255
00:17:59.039 --> 00:18:03.759
carry themselves and how people you know, wow, this number of communication works
256
00:18:03.880 --> 00:18:06.599
that never would have occurred to me
in advance. And I can give you
257
00:18:06.720 --> 00:18:10.359
one example that I that I love
to talk about. We, after we
258
00:18:10.519 --> 00:18:12.910
trained our first models, and these
are our intention models, right, whether
259
00:18:12.990 --> 00:18:15.990
someone was watched a cross in front
of the vehicle, we did an exercise
260
00:18:17.190 --> 00:18:22.069
where we went and looked at what
parts of images are models cared about,
261
00:18:22.109 --> 00:18:25.349
so you can, you know,
do it, do kind of it a
262
00:18:25.509 --> 00:18:29.140
heat map and if the neural network
really cares about a certain part of the
263
00:18:29.180 --> 00:18:33.220
eventual light up super bright. What
we noticed is that our models lit up
264
00:18:33.220 --> 00:18:37.460
super bright anytime someone had a bag
on their shoulder. It would really pay
265
00:18:37.460 --> 00:18:41.329
attention to the bag and I we
it was like, well, why does
266
00:18:41.369 --> 00:18:45.170
it? Is this like some weird
thing that we have like a you know
267
00:18:45.210 --> 00:18:48.690
it? Does our training data have
an overrepresentation of people crossing roads with bags
268
00:18:48.849 --> 00:18:52.690
versus not? And what we realized
is, no, it's not that.
269
00:18:53.130 --> 00:18:57.839
But if you think about somebody with
a bag on their shoulder that is thinking
270
00:18:57.920 --> 00:19:02.880
they're about to start crossing the road
and you have that just on one shoulder,
271
00:19:02.920 --> 00:19:04.359
what do you do? Will you
just kind of cinch up ever so
272
00:19:04.599 --> 00:19:10.750
slightly so that when you start moving
the bag doesn't go flying every so there's
273
00:19:10.789 --> 00:19:15.390
this little bit of tightening and it's
a very subtle visual feature. But in
274
00:19:15.549 --> 00:19:21.150
retrospect of course that's super diagnostic and
of course we use that as humans,
275
00:19:21.190 --> 00:19:26.180
right. But that insight really came
out of how we collect this training data,
276
00:19:26.259 --> 00:19:27.900
how we treated the models. If
you if you'd ask me an advance
277
00:19:29.019 --> 00:19:32.779
to, you know, write down
what of the visual features that people use
278
00:19:32.859 --> 00:19:34.579
to make these judgments, I never
would have said, you know, how
279
00:19:34.779 --> 00:19:38.569
how tightly somebody has a bag since
to theirs and their shoulder. No,
280
00:19:38.730 --> 00:19:44.009
that's really interesting. So we talked
about sort of features and how you're analyzing
281
00:19:44.130 --> 00:19:47.369
and using their own network. What
do you think some of the biggest challenges
282
00:19:47.410 --> 00:19:49.809
are on the more of the technical
side for you to be able to really
283
00:19:49.849 --> 00:19:56.599
sort of advance your ai and continue
to evolve your technology? So for us,
284
00:19:56.680 --> 00:19:59.519
I would say first of all,
what we're doing is something that,
285
00:20:00.119 --> 00:20:03.240
and I maybe if I'd realized this
a little bit better before I started the
286
00:20:03.279 --> 00:20:07.039
company, but if there's chance of
visually, but this is very, very
287
00:20:07.150 --> 00:20:11.190
hard. We are taking machine learning
research that's very much at the research frontier
288
00:20:11.470 --> 00:20:15.990
and the point is to take that
and get that all the way to a
289
00:20:15.190 --> 00:20:21.940
module that's reliable enough to be deployed
in a vehicle. And one of the
290
00:20:22.180 --> 00:20:26.019
pieces of that that's really important is
the question that we are asking, and
291
00:20:26.099 --> 00:20:29.940
we talked a little bit about this
earlier, but the question we are asking,
292
00:20:30.019 --> 00:20:34.180
which is what's in that person's head
right now as that relates the task
293
00:20:34.299 --> 00:20:38.890
of driving, is very different than
the kinds of questions are customers have typically
294
00:20:38.930 --> 00:20:42.690
ask so for us, we've had
to do a lot of work and build
295
00:20:42.690 --> 00:20:48.130
up a lot of expertise so that
we can make sure that answering that question
296
00:20:48.210 --> 00:20:52.160
in these sort of critical moments when
you have a difficult interaction of with a
297
00:20:52.200 --> 00:20:59.599
pedestrian is reliably and correctly the information
that you need to change the behavior in
298
00:20:59.680 --> 00:21:03.039
the vehicle. So what that means
in practice is we really have to be
299
00:21:03.160 --> 00:21:07.910
the experts in how you integrate a
system like this, how you use the
300
00:21:07.990 --> 00:21:12.670
outputs, how it should modify the
motion plan, even questions what the motion
301
00:21:12.750 --> 00:21:17.470
plan of the vehicle should be.
We've had to develop a lot of expertise
302
00:21:17.549 --> 00:21:21.339
in those and so I think that
as far as the core model goes,
303
00:21:21.900 --> 00:21:23.220
it does what we say. That's
right, like when I gave you that
304
00:21:23.500 --> 00:21:26.579
that five hundred people in the car
kind of analogy. That there really is
305
00:21:26.740 --> 00:21:30.579
what it does and it really does
do that with a high degrade fidelity.
306
00:21:30.099 --> 00:21:33.130
But making sure that then we can
take that, we can say to our
307
00:21:33.210 --> 00:21:37.089
customers, okay, this is where
you plug it in, this is how
308
00:21:37.250 --> 00:21:38.650
you plug it in, this is
what the output should look like, and
309
00:21:38.690 --> 00:21:41.970
if the output looks like that,
looks like that, you absolutely can rely
310
00:21:42.049 --> 00:21:47.400
on it. Has Been a substantial
amount of effort and something that where where
311
00:21:47.440 --> 00:21:49.680
we we know how to do it
now, but making sure that we can
312
00:21:49.759 --> 00:21:55.039
do that broadly is kind of the
next step of our commercial relationships. Yeah,
313
00:21:55.079 --> 00:21:59.200
I mean it always comes down to
implementation and sort of usage, and
314
00:21:59.599 --> 00:22:03.269
it's great. You know, I've
started a lot of companies and often times
315
00:22:03.269 --> 00:22:07.829
it's using data science to find some
sort of opportunity. And you know,
316
00:22:07.869 --> 00:22:10.910
you always think about it in the
abstracts where, once we get this in
317
00:22:10.990 --> 00:22:12.990
the market, you know it's amazing, it works really well, but then
318
00:22:14.029 --> 00:22:18.140
there's only the other pieces and if
for it to really work, it's all
319
00:22:18.140 --> 00:22:19.980
about that integration, and what you
guys are doing is so much more.
320
00:22:21.180 --> 00:22:23.220
You know, they, the teams
are working without to understand what you're doing
321
00:22:23.299 --> 00:22:26.059
as much as you have to understand
what they're doing so that it can actually
322
00:22:26.059 --> 00:22:29.779
do what it's supposed to. Absolutely
and I think that, you know,
323
00:22:29.859 --> 00:22:33.890
the good part is that we've had
a lot of time and a lot of
324
00:22:33.970 --> 00:22:38.569
iteration with customers to figure that out
right. So we have those answers now.
325
00:22:38.809 --> 00:22:41.490
The other part, I would say, I don't know that this is
326
00:22:41.569 --> 00:22:45.000
universally good, but it's helpful to
us, is that you still you need
327
00:22:45.200 --> 00:22:49.599
this right. So not just us
but our customers really need the answer to
328
00:22:49.759 --> 00:22:53.519
these questions, because without answering the
question, it had being able to answer
329
00:22:53.559 --> 00:22:57.589
questions very much like the ones that
were answering. You do nothing else works.
330
00:22:57.710 --> 00:23:00.869
So so there's a lot of incentive
on both sides and we've been able
331
00:23:00.910 --> 00:23:03.589
to do a lot of work to
build out the infrastructor for that. But
332
00:23:03.750 --> 00:23:07.309
it absolutely you know, very early
on we would go to customers and they
333
00:23:07.309 --> 00:23:10.750
would say, well, how do
we integrate this? Now will be like
334
00:23:10.869 --> 00:23:12.619
well, and however you want,
and it's not great because you know,
335
00:23:14.019 --> 00:23:17.740
what do you think like? I
don't know. So kind of kind of
336
00:23:17.819 --> 00:23:22.859
building out that and really a big
part of our team now is oriented around
337
00:23:22.299 --> 00:23:26.019
making sure we can answer those kinds
of questions for customers because because certainly that's
338
00:23:26.099 --> 00:23:30.769
what appropriately they need from us.
Yeah, that's great. Before we go,
339
00:23:30.890 --> 00:23:33.369
I would like to ask you,
you know, what sort of guidance
340
00:23:33.410 --> 00:23:37.210
or advice would you give to someone
who's seeking your career really in data science
341
00:23:37.250 --> 00:23:41.559
and engineering other specific things? From
your background? You have a very unique
342
00:23:41.559 --> 00:23:45.240
experience, clearly as far as great
other things, but I'd look to hear
343
00:23:45.279 --> 00:23:48.839
from you if you have some some
guidance to people who want to get people
344
00:23:48.920 --> 00:23:52.039
into it. Yeah, well,
I'll tell you. So, Emman,
345
00:23:52.160 --> 00:23:55.519
I'm going to say something and I'm
not completely sure that this is actually good
346
00:23:55.680 --> 00:24:03.549
practical advice, but it's my my
strongest intuition about what's happening now in machine
347
00:24:03.589 --> 00:24:10.339
learning and AI and computer vision is
that the hard problems are no longer computer
348
00:24:10.460 --> 00:24:15.299
science problems. So if you think
about training a deep network, and we
349
00:24:15.380 --> 00:24:18.460
get lots of people applying and they're
applying because they want to train deep networks,
350
00:24:18.579 --> 00:24:21.779
frankly, training a deep network per
se is pretty easy. You just
351
00:24:21.940 --> 00:24:23.529
you to give it a data that
you feed it to it. You'd learn
352
00:24:23.609 --> 00:24:27.170
how to evaluate it, you learn
how to do your optimizations later. It's,
353
00:24:27.609 --> 00:24:30.009
you know, kind of a kind
of a there's kind of a cookbook
354
00:24:30.049 --> 00:24:33.450
for it and it's not that hard. On the other hand, knowing how
355
00:24:33.650 --> 00:24:38.400
to ask questions that other people don't
know how to ask the data and knowing
356
00:24:38.519 --> 00:24:44.880
how to think about data in terms
of when you need quality, what quality
357
00:24:44.960 --> 00:24:48.359
means, and sort of thinking about
the rest of the workflow outside of the
358
00:24:48.440 --> 00:24:52.509
model training per se. There's some
very, very interesting and very, very
359
00:24:52.630 --> 00:24:59.710
hard under explore questions in that and
those really aren't computer science questions. They've
360
00:24:59.710 --> 00:25:04.589
resolved to behavioral science questions, to
infrastructure questions. So so I would say
361
00:25:06.150 --> 00:25:08.140
my advice, such as it is, is to have kind of more than
362
00:25:08.259 --> 00:25:12.299
one don't just know about machine learning, don't just learn about data science,
363
00:25:12.460 --> 00:25:17.660
have something else you know and other
things you know about the kind of number
364
00:25:18.099 --> 00:25:21.849
of the questions and the answers that
people are likely to be interested in.
365
00:25:22.009 --> 00:25:23.730
Yeah, well, that's great.
Yeah, always bringing more to the table
366
00:25:23.890 --> 00:25:26.809
helps and get a lot of different
situations. So thank you for that.
367
00:25:27.609 --> 00:25:30.250
Thank you for me and guess in
the show today. We really appreciate it.
368
00:25:30.490 --> 00:25:33.490
Where can the community find you?
What's the best way for the data
369
00:25:33.529 --> 00:25:37.240
science community to sort of reach out
and connect? Absolutely, I mean I
370
00:25:37.279 --> 00:25:41.440
would say check us, check out
at ewwwcers uptive up commentcom. We have
371
00:25:41.680 --> 00:25:45.759
a lot of information up there,
including some some nice visualizations. Is The
372
00:25:45.799 --> 00:25:49.069
black box metaphor we discussed, as
well as job opportunities and and a bunch
373
00:25:49.069 --> 00:25:52.589
of different opporty ways to learn more. Okay, well, great. At
374
00:25:52.630 --> 00:25:56.630
the data standard we are community of
date enthusiasts and data thought leaders. Were
375
00:25:56.710 --> 00:26:00.670
we look forward to engaging with you
on our website, data standard dot Ioh,
376
00:26:00.109 --> 00:26:04.299
where we regularly post a wide range
of podcasts with data executives and offer
377
00:26:04.500 --> 00:26:08.380
regular networking opportunities or more information,
visit the website. Thanks so much,
378
00:26:08.380 --> 00:26:11.299
Sam for joining us today. Thanks
for having
