#100 Embedded Machine Learning on Edge Devices

WEBVTT

1
00:00:08.679 --> 00:00:13.240
You're listening to data framed, a
podcast by data camp. In this show

2
00:00:13.439 --> 00:00:17.800
you'll hear all the latest trends and
insights in data science. Whether you're just

3
00:00:17.879 --> 00:00:22.359
getting started in your data career or
you're a data leader looking to scale data

4
00:00:22.440 --> 00:00:27.359
driven decisions in your organization, join
us for in depth discussions with data and

5
00:00:27.440 --> 00:00:32.159
analytics leaders at the forefront of the
data revolution. Let's dive right in.

6
00:00:35.240 --> 00:00:39.119
Hi Everyone. As you may be
able to tell from the accent, I'm

7
00:00:39.159 --> 00:00:44.880
not Adele, I am Richie,
a data evangelist at data camp. This

8
00:00:44.920 --> 00:00:49.520
week I'm filling in for a Dell
and I'll shortly begin co hosting the data

9
00:00:49.560 --> 00:00:55.200
framed podcast. I've been working in
data science and data scientif education for nearly

10
00:00:55.200 --> 00:01:00.799
two decades now and I've watched the
field go from being a niche nerdy thing

11
00:01:00.880 --> 00:01:06.040
to well mainstream nerdy thing, and
I spent a lot of my career providing

12
00:01:06.159 --> 00:01:12.000
data science support biologists and chemists and
business people, and my experiences that even

13
00:01:12.040 --> 00:01:19.359
incredibly smart people sometimes get terrified by
having to analyze data. So I really

14
00:01:19.400 --> 00:01:23.120
care about making data feel a little
less scary to work with in the coming

15
00:01:23.159 --> 00:01:29.000
episodes, as well as revisiting the
traditional data frame theme of Helping Organization has

16
00:01:29.000 --> 00:01:33.719
become data driven. I'd also like
to provide some inspiration to you all about

17
00:01:33.959 --> 00:01:38.840
the joys and the possibilities of working
with data. Once you get over the

18
00:01:38.920 --> 00:01:42.519
fear, there really is a lot
of fun to be had. Today we'll

19
00:01:42.519 --> 00:01:49.159
be talking about machine learning on low
powered devices like bones and internet of things

20
00:01:49.239 --> 00:01:53.959
sensors. So there's a really exciting
emerging field and I'm very much looking forward

21
00:01:55.000 --> 00:02:00.519
to finding out about it from Dan
Situny Yuki. Danny's head of machine at

22
00:02:00.680 --> 00:02:05.560
edge impulse and he's the author of
the Book Tangy M L and the forthcoming

23
00:02:05.920 --> 00:02:10.439
ai at the edge. He previously
worked at Google on tensorflow light. So

24
00:02:10.520 --> 00:02:15.960
all this is to say he's a
real expert in the field. Hi Dan,

25
00:02:15.080 --> 00:02:20.159
thanks for joining us today. We're
gonna be talking about machine learning on

26
00:02:20.360 --> 00:02:24.520
edge devices and that terms actually barely
unfamiliar to me. So perhaps for a

27
00:02:24.520 --> 00:02:29.080
bit of context, you could just
start talking to us about well, what

28
00:02:29.319 --> 00:02:32.680
actually is an edge device? Yeah, great question, Richie, and I'm

29
00:02:32.800 --> 00:02:38.199
very excited to be here today.
So an edge device refers to a computer

30
00:02:38.319 --> 00:02:43.120
that's at the edge of the network, and so that sort of makes sense

31
00:02:43.159 --> 00:02:49.039
if you think of the Internet as
this big, sprawling set of interconnected devices,

32
00:02:49.319 --> 00:02:53.599
but it hasn't an edge where there
are devices that are sort of connected

33
00:02:53.680 --> 00:02:59.360
to the grace and network on one
side and then on the other side they're

34
00:02:59.400 --> 00:03:04.719
connected to the real world, basically
through sensors and other sorts of hardware.

35
00:03:04.960 --> 00:03:09.120
So a typical edge device might be, for example, an Internet connected camera,

36
00:03:09.639 --> 00:03:15.039
or it could be a personal fitness
wearable that you wear around your wrist

37
00:03:15.120 --> 00:03:20.240
that logs when you're doing exercise and
uploads data about it to the cloud.

38
00:03:20.560 --> 00:03:25.039
And there are tons of different applications
for edge devices from everywhere, through consumer

39
00:03:25.080 --> 00:03:32.199
electronics through to industrial systems, the
kinds of devices that are monitoring industrial processing

40
00:03:32.599 --> 00:03:38.360
factories, and even the ultimate sort
of edge device is a spacecraft, the

41
00:03:38.400 --> 00:03:44.039
Mars Rovers that are up there driving
around on Mars and they're collecting data and

42
00:03:44.199 --> 00:03:49.439
uploading it and sending it back to
Earth. Those are our furthest edge devices.

43
00:03:49.520 --> 00:03:55.840
Yet that's pretty impressive. There's such
a broad range of different applications.

44
00:03:55.879 --> 00:04:00.919
But can you tell me why do
you have to do machine learning on an

45
00:04:00.000 --> 00:04:05.159
edge device? So I think the
standard thing, if I'm doing machine learning,

46
00:04:05.560 --> 00:04:10.080
is that I'm going to be doing
something on my laptop, so that's

47
00:04:10.120 --> 00:04:13.560
going to be processing the model,
or maybe I'm working in the cloud,

48
00:04:14.400 --> 00:04:19.040
so it's going to be some server
somewhere processing things. Why would you want

49
00:04:19.079 --> 00:04:24.319
to do machine learning on a device
which hasn't got very much power? If

50
00:04:24.360 --> 00:04:29.600
you think about sensor data and data
about the real world, we sort of

51
00:04:30.240 --> 00:04:32.879
I have this idea that we when
we deal with data's data scientists, we're

52
00:04:32.920 --> 00:04:36.519
dealing with data sets that have been
collected and a lot of the time we're

53
00:04:36.519 --> 00:04:43.959
dealing with data sets that are accrued
by our big back end systems and and

54
00:04:44.040 --> 00:04:48.439
the systems that are generating the data
already within our infrastructure, basically. But

55
00:04:48.560 --> 00:04:53.959
if we're collecting sensor data, so
we're collecting data about the real world,

56
00:04:54.319 --> 00:05:00.040
there's essentially like a bottleneck where the
real world has an almost infinite amount of

57
00:05:00.160 --> 00:05:04.439
data. It's just this endless source
of new information. Imagine you have a

58
00:05:04.560 --> 00:05:11.639
device out there that's collecting temperature and
pressure and all sorts of other stuff that's

59
00:05:11.680 --> 00:05:16.959
related to climates or the weather.
There's basically like an infinite stream of data

60
00:05:16.959 --> 00:05:20.360
out there for this thing to collect, but it might not have a very

61
00:05:20.439 --> 00:05:27.959
good connection with which to send that
data to another system that can do analysis

62
00:05:28.000 --> 00:05:30.800
of it. So typically edge devices, because they're on the edge of the

63
00:05:30.879 --> 00:05:36.240
network. They might suffer from things
like unreliable Internet connections or low bandwidth,

64
00:05:36.519 --> 00:05:43.040
or maybe they need to operate under
constrained sort of energy regime, so they're

65
00:05:43.120 --> 00:05:46.079
running off a battery or something like
that, or solar power, and they

66
00:05:46.079 --> 00:05:51.160
can't afford to be burning loads of
energy with network communication all the time.

67
00:05:51.560 --> 00:05:56.279
If you think about like your mobile
phone, the thing that uses the most

68
00:05:56.319 --> 00:06:00.759
battery, apart from the screen of
the phone, is using the radio to

69
00:06:00.879 --> 00:06:05.959
broadcast information. So if you're in
a situation where your device is constrained in

70
00:06:06.000 --> 00:06:11.160
some way and is unable to send
all of that data back at the rate

71
00:06:11.199 --> 00:06:15.560
that you're receiving it, then it
makes sense to put some of this intelligence

72
00:06:15.120 --> 00:06:20.600
on the device rather than depending on
having intelligence running on a cloud service somewhere.

73
00:06:21.279 --> 00:06:25.680
Yeah, I can certainly see how
with a Mars rover, you might

74
00:06:25.720 --> 00:06:30.360
not want to be sending all your
data across space from Mars to somewhere on

75
00:06:30.399 --> 00:06:34.560
earth and then trying to have all
the results and send them back. So

76
00:06:35.360 --> 00:06:42.120
it feels like a slightly novel field
and I'm curious as to how you got

77
00:06:42.120 --> 00:06:46.680
into thinking about edge m l.
yes, so edge ML and sort of

78
00:06:46.800 --> 00:06:50.079
edge ai more broadly, which is
the the idea of doing all sorts of

79
00:06:50.160 --> 00:06:56.199
artificial intelligence on edge devices, has
only existed for a relatively short time and

80
00:06:56.920 --> 00:07:02.279
essentially the thing that's propagated it and
allowed it to develop is the fact that

81
00:07:02.319 --> 00:07:08.759
these edge devices have gotten powerful enough
that they're able to run some of these

82
00:07:09.279 --> 00:07:15.839
heavy lifting algorithms, like deep learning
models on device. But the way that

83
00:07:15.879 --> 00:07:18.959
I got involved with this was I
was working at Google on the tensor flow

84
00:07:19.000 --> 00:07:23.800
team, and tensor flow, as
you're you're probably aware, is Google's big

85
00:07:24.040 --> 00:07:28.600
deep learning framework and there's a part
of tensor flow called tensorflow light, which

86
00:07:28.680 --> 00:07:34.319
is basically designed around the idea of
deploying models down to mobile phones, because

87
00:07:34.480 --> 00:07:40.120
there are like mobile phones are edge
devices right there, these little portable computers

88
00:07:40.120 --> 00:07:43.439
on the edge of the network with
lots of sensors on, and some folks

89
00:07:43.439 --> 00:07:46.959
at Google realized that would be really
cool to be able to run models on

90
00:07:47.120 --> 00:07:51.120
phones to do do all sorts of
cool stuff, and then the sort of

91
00:07:51.199 --> 00:07:57.240
next logical step from that was to
think about like, okay, what kind

92
00:07:57.240 --> 00:08:03.639
of devices are even smaller than mobile
owns and more more edgy and more possible

93
00:08:03.759 --> 00:08:07.519
and lower power than is it possible
to run deep learning models on those.

94
00:08:07.839 --> 00:08:13.720
And I was lucky enough to be
working at Google at the time, where

95
00:08:13.759 --> 00:08:18.839
a new team started by a guy
called Pete warden was starting to build and

96
00:08:18.000 --> 00:08:24.920
investigate this area, and I basically
lucked into being around during this awesome new

97
00:08:24.959 --> 00:08:31.319
technology being launched and I realized it
was the most exciting thing I'd come across

98
00:08:31.360 --> 00:08:35.799
in the tech industry so far.
Combined my fascination around embedded software engineering and

99
00:08:35.919 --> 00:08:41.960
edge devices with my fascination around data
science and machine learning. That's exciting.

100
00:08:41.360 --> 00:08:46.000
And so with the work you've been
doing, like with tensorflow and things like

101
00:08:46.039 --> 00:08:50.120
that, how is it different to
just doing regular ml? How is it

102
00:08:50.159 --> 00:08:56.320
constrained or what makes it hard?
It's really interesting actually, because when you're

103
00:08:56.360 --> 00:09:01.600
dealing with this type of real time
sense today, to that by itself is

104
00:09:03.120 --> 00:09:07.519
a very different paradigm to the sort
of work that you do on data sets

105
00:09:07.559 --> 00:09:11.720
which are more tabular. So,
for example, like we're all very familiar

106
00:09:11.759 --> 00:09:16.440
with this idea of feature engineering for
tabular data. So maybe we're looking at

107
00:09:16.480 --> 00:09:22.799
analyzing the business performance of something or
analyzing some government data about some kind of

108
00:09:24.080 --> 00:09:28.639
phenomena that are going on in our
world. But our typical workflow for identifying

109
00:09:28.679 --> 00:09:35.120
good features to use from that data
uses a set of tools which occupy one

110
00:09:35.399 --> 00:09:41.279
paradigm. But when you think about
real time, live sensor data, so

111
00:09:41.320 --> 00:09:45.720
this is basically like little samples of
information that are coming in at very high

112
00:09:45.759 --> 00:09:52.720
frequency and essentially are quite challenging to
feed raw into a machine learning model,

113
00:09:54.200 --> 00:09:58.639
there's this whole extra field that has
existed for a long time called digital signals

114
00:09:58.639 --> 00:10:03.360
process, a thing which is based
around the idea that there are algorithms that

115
00:10:03.759 --> 00:10:11.720
can make sense of these kinds of
real time streams of sensor data and distill

116
00:10:11.879 --> 00:10:16.720
them down and get the most interesting
components most visible, so that whatever you're

117
00:10:16.720 --> 00:10:20.480
trying to do downstream isn't swamped with
all of this raw data. And these

118
00:10:20.519 --> 00:10:26.720
algorithms look very different from anything that
you do with tabular data, for example

119
00:10:26.840 --> 00:10:31.000
with audio data that you basically,
at a certain frequency, get along stream

120
00:10:31.039 --> 00:10:37.200
of numbers at a very high frequency
which represents the amount of sound pressure.

121
00:10:37.279 --> 00:10:43.639
Almost that's happening, the amount that
your microphones diaphragm is flexing at any given

122
00:10:43.679 --> 00:10:48.159
point and to turn that into something
that a deep learning model can understand.

123
00:10:48.480 --> 00:10:52.679
One of the best ways of doing
it is through building something called a Spectrogram,

124
00:10:52.679 --> 00:11:00.679
which is basically a representation of that
series of signals or series of values

125
00:11:01.120 --> 00:11:07.360
in the frequency domain. So instead
of having a single column with these amplitude

126
00:11:07.440 --> 00:11:13.759
values, you end up with a
two D Matrix which is a representation of

127
00:11:13.080 --> 00:11:18.200
time on the x axis and frequency
on the Y axis. That's bucketed into

128
00:11:18.360 --> 00:11:24.120
into the individual buckets and that is
essentially like an image and that's what you

129
00:11:24.200 --> 00:11:28.879
feed into your deep learning models so
you can use a convolutional neural network to

130
00:11:28.000 --> 00:11:33.200
understand it. So it's all about
transforming signals from their raw form, which

131
00:11:33.240 --> 00:11:39.360
often isn't very convenient to work with, into a maybe higher dimensionality but lower

132
00:11:39.399 --> 00:11:46.639
frequency form which is more convenient to
use these machine learning tools. That's fascinating.

133
00:11:48.159 --> 00:11:52.559
So very sort of distant memories of
learning about fourier transforms and wavelets and

134
00:11:52.639 --> 00:11:58.600
things like that been signal processing from
university, so it's not fresh in my

135
00:11:58.639 --> 00:12:03.799
mind. I think it's something I've
not used in a data science or machine

136
00:12:03.879 --> 00:12:09.720
learning context before. So that sort
of engineering type stuff is always very separate,

137
00:12:09.480 --> 00:12:13.720
and so it's interesting to hear about
how that sort of audio engineering and

138
00:12:15.120 --> 00:12:18.000
machine learning, the two of these
have come together. The thing I love

139
00:12:18.039 --> 00:12:24.000
about it is that there is this
huge field of embedded engineering and signal processing

140
00:12:24.279 --> 00:12:28.879
that's existed for a fairly long time
at this point, and people who are

141
00:12:28.919 --> 00:12:33.200
absolutely incredible experts in this type of
stuff, and what we're finding is that

142
00:12:33.399 --> 00:12:39.200
there's this amazing fusion between the things
that are coming from the machine learning and

143
00:12:39.759 --> 00:12:46.080
deep learning and data science side with
this signal processing expertise, and when you

144
00:12:46.120 --> 00:12:50.919
get people from both sides together,
you can figure out really amazing ways of

145
00:12:50.960 --> 00:12:56.600
interpreting signals and training models. It's
really exciting. And when you've got a

146
00:12:56.720 --> 00:13:01.039
very low powered device and maybe you
haven't got a great into net connection and

147
00:13:01.360 --> 00:13:05.799
maybe you're limited by battery life or
things like that, how does that affect

148
00:13:05.840 --> 00:13:11.440
the machine learning side of things?
So do you have to choose different models,

149
00:13:11.639 --> 00:13:15.639
or do you have to train things
differently, or how does it affect

150
00:13:15.679 --> 00:13:20.399
the actual models themselves? So one
thing that I'd first of all point out

151
00:13:20.440 --> 00:13:24.159
is that when we're talking about machine
learning on embedded devices, the vast majority

152
00:13:24.200 --> 00:13:28.879
of the time we're talking about inference, we're not necessarily talking about training.

153
00:13:30.120 --> 00:13:35.240
So there's nothing stopping you from training
models on these edge devices, and in

154
00:13:35.279 --> 00:13:39.960
some cases this does happen. But
one of the biggest constraints of edge devices

155
00:13:39.039 --> 00:13:43.279
is that you've got limited memory to
work with, and so a lot of

156
00:13:43.320 --> 00:13:48.480
the time when we're talking about training
models in a supervised approach, that just

157
00:13:48.600 --> 00:13:52.440
isn't feasible if you don't have enough
space to store data. Maybe you've only

158
00:13:52.480 --> 00:13:56.919
got tens of killer bytes, which
obviously isn't enough to store much of a

159
00:13:56.000 --> 00:14:01.360
data set. So if you're doing
some kind of unsupervised approach or you've got

160
00:14:01.399 --> 00:14:05.440
some kind of interesting approach to how
you find tuning a model on device,

161
00:14:05.600 --> 00:14:09.759
that can work sometimes, but a
lot of the time we're just talking about

162
00:14:09.879 --> 00:14:13.399
running inference. And then when we're
talking about running inference, there are really

163
00:14:13.440 --> 00:14:20.559
three main constraints. The first constraint
is energy and connected with that is performance,

164
00:14:20.799 --> 00:14:24.960
like the performance of the device,
because if you're trying to get a

165
00:14:26.000 --> 00:14:28.919
device to run on a battery,
for example, you're not going to have

166
00:14:28.000 --> 00:14:35.440
this like two Giga hurts processor,
multi core CPU or like a GPU running

167
00:14:35.440 --> 00:14:39.480
in there. There are edge devices
that are very powerful, but typically you're

168
00:14:39.519 --> 00:14:46.279
going to see more constrained clock speeds
than you will in server side infrastructure for

169
00:14:46.519 --> 00:14:50.159
running machine learning models. So the
issue there is it's quite slow to run

170
00:14:50.399 --> 00:14:54.639
models, and so that means that
you are going to be biased towards training

171
00:14:54.679 --> 00:14:58.200
smaller models, like a smaller model
is going to be able to run more

172
00:14:58.279 --> 00:15:03.720
quickly on advice, use less energy
and fits this set of constraints on the

173
00:15:03.840 --> 00:15:09.000
edge. And then the other two
things are read only memory and Ram.

174
00:15:09.120 --> 00:15:13.360
So the distinction there is that read
only memory is where you're going to be

175
00:15:13.399 --> 00:15:18.679
storing the weights of your model plus
the program that runs on your edge device.

176
00:15:18.120 --> 00:15:24.600
So obviously there's a range, always
in computing hardware, but at the

177
00:15:24.679 --> 00:15:31.000
smallest end you might only have tens
or hundreds of killer bytes to store your

178
00:15:31.039 --> 00:15:35.919
models weights. So if we're thinking
about typical deep learning models for vision,

179
00:15:35.120 --> 00:15:39.840
even something like a mobile net model
that's designed for running on a mobile phone

180
00:15:41.120 --> 00:15:43.799
is going to be too big to
run on some of these smallest edge devices.

181
00:15:43.919 --> 00:15:48.039
So you need to think about how
do we make these models smaller?

182
00:15:48.320 --> 00:15:54.440
And then the ram parts, the
working memory, is just as important because

183
00:15:54.480 --> 00:15:58.360
you need someone to store your signal
when it comes in and you need to

184
00:15:58.440 --> 00:16:03.360
store the forep learning, for example, the intermediate activations of your models layers

185
00:16:03.600 --> 00:16:07.399
while you're working, and if you
don't have enough ram you can't do that.

186
00:16:07.480 --> 00:16:12.279
So these are the three most important
constraints and it typically forces you to

187
00:16:12.320 --> 00:16:18.519
think about smaller models and models which
are designed in a way to be efficient

188
00:16:18.559 --> 00:16:26.159
in terms of ram usage. And
it also encourages you to do as much

189
00:16:26.240 --> 00:16:30.279
signal processing as you can upfront,
because there's often this trade off between you

190
00:16:30.320 --> 00:16:37.879
can basically extract features through signal processing
and use a certain amount of compute and

191
00:16:37.000 --> 00:16:41.080
you can throw things at your deep
learning model, for example, and make

192
00:16:41.120 --> 00:16:47.639
it figure out the feature extraction part
two. And for any given device there's

193
00:16:47.639 --> 00:16:52.759
going to be like an ideal set
point for a particular data set where there's

194
00:16:52.799 --> 00:16:56.360
a perfect trade off between those two
sections where it's the most efficient. But

195
00:16:56.639 --> 00:17:00.600
that's something you have to find on
a per device and per data set basis.

196
00:17:00.879 --> 00:17:06.400
So there's this whole extra contrastraint that
comes in that you don't have to

197
00:17:06.440 --> 00:17:11.759
really think about when you're thinking about
data science on big, powerful developer workstations

198
00:17:11.880 --> 00:17:14.839
or on servers in the back end. And for me. That's what makes

199
00:17:14.839 --> 00:17:18.880
it really exciting. It's always nice
to have a constraint because it forces you

200
00:17:18.920 --> 00:17:22.200
to be creative and think of new
ways of doing stuff. That's incredible.

201
00:17:22.640 --> 00:17:26.799
And so I get the sense that
there's some sort of trade off where you

202
00:17:26.839 --> 00:17:30.480
sacrifice maybe a little bit of predictive
power because you're using a simpler model,

203
00:17:32.039 --> 00:17:36.119
but that's necessary in order to be
able to run it at all because you

204
00:17:36.119 --> 00:17:40.240
don't have the computing power or the
RAM available. Is that about right?

205
00:17:40.680 --> 00:17:42.680
Yeah, I mean that can be
true. So if we're if we're thinking

206
00:17:42.720 --> 00:17:49.240
about larger, more highly dimensional data, so images for example, typically if

207
00:17:49.279 --> 00:17:53.759
you're compressing a model to get it
to run on an embedded device, that

208
00:17:53.839 --> 00:17:57.839
might involve, first of all,
just training a smaller model, like coming

209
00:17:57.880 --> 00:18:03.279
up with an architecture that has less
weights and training that. And so that's

210
00:18:03.359 --> 00:18:07.279
one way and and obviously there's a
trade off there with the fewer weights,

211
00:18:07.319 --> 00:18:12.079
the less representative power the model has, and so potentially you're going to get

212
00:18:12.359 --> 00:18:18.440
performance this isn't as good as you
would on a larger model. The second

213
00:18:18.480 --> 00:18:22.759
thing there that we have at our
disposal is this idea of model compression,

214
00:18:22.799 --> 00:18:26.200
where you can take a model that
was trained in a typical manner as if

215
00:18:26.200 --> 00:18:30.960
it was going to run on on
any device, and then you can do

216
00:18:30.200 --> 00:18:37.519
certain things to it that reduce its
precision, reduce its accuracy potentially, but

217
00:18:37.720 --> 00:18:41.599
make it smaller. So one of
those things is called quantization. So quantization,

218
00:18:41.799 --> 00:18:45.359
which you might have come across in
the back end world as well.

219
00:18:45.400 --> 00:18:51.279
It's a way of getting models to
be smaller by reducing the precision of their

220
00:18:51.319 --> 00:18:56.480
weights. In a typical deep learning
model, for example, the weights are

221
00:18:56.880 --> 00:19:00.000
usually trained in float thirty two format. So we at these thirty two bit

222
00:19:00.079 --> 00:19:04.319
floating point numbers. They're very expressive, they can represent a very large range

223
00:19:04.359 --> 00:19:08.519
of values. But to get them
to run on an embedded device, if

224
00:19:08.559 --> 00:19:15.039
you reduce the number of bits available
to store each weight. So you reduce

225
00:19:15.160 --> 00:19:18.440
them that down to eight bits.
So you're now storing the numbers that make

226
00:19:18.519 --> 00:19:25.039
up your model as eight bit numbers. That's four times smaller model. So

227
00:19:25.079 --> 00:19:27.960
if you can do that and get
away with it without screwing up the models

228
00:19:29.000 --> 00:19:32.960
accuracy too much, that can be
great. And they're actually really smart ways

229
00:19:32.960 --> 00:19:37.200
of doing that where you can really
lose barely any accuracy but reduce the model

230
00:19:37.799 --> 00:19:44.519
by four times or eight times,
or even the smallest precision models are actually

231
00:19:45.359 --> 00:19:51.240
binarized neural networks, B and NS, which have one bit weights. So

232
00:19:51.839 --> 00:19:56.880
literally the weights of the model are
binary and there's some special regimes you need

233
00:19:56.039 --> 00:20:00.720
in order to train those, but
they can work really well. So all

234
00:20:00.759 --> 00:20:06.039
of that comes with trade offs and
it's about managing the trade offs and keeping

235
00:20:06.079 --> 00:20:10.640
the amount of performance that you need
for your application. I think a lot

236
00:20:10.680 --> 00:20:12.839
of people listening will want to know
how do I get involved in this?

237
00:20:14.039 --> 00:20:18.079
It sounds exciting, so how do
you get started with this? There's lots

238
00:20:18.119 --> 00:20:22.640
of different angles to come at this
topic. Some people come at it from

239
00:20:22.680 --> 00:20:26.559
an embedded engineering angle, where they've
worked in embedded engineering before, working with

240
00:20:26.640 --> 00:20:33.160
micro controllers or embedded Linux blores like
raspberry pie, for example, and they've

241
00:20:33.319 --> 00:20:36.759
hit a wall where they want to
make sense of the world and the sense

242
00:20:36.759 --> 00:20:40.440
of data that they're pulling in on
their device, but there's only so much

243
00:20:40.480 --> 00:20:45.880
you can do with hand coded Algorithms
and basic signal processing, so they want

244
00:20:45.920 --> 00:20:48.319
to embed a bit more intelligence in
devices. And then there are people coming

245
00:20:48.359 --> 00:20:53.200
from the other end where maybe they
have a background in data science or machine

246
00:20:53.279 --> 00:20:59.319
learning and they'd like to see,
like how could I take some of this

247
00:21:00.119 --> 00:21:04.119
intelligence that we're able to encapsulate in
models and push it down onto these tiny

248
00:21:04.160 --> 00:21:10.079
devices so that it's available in places
it wasn't available before. So your journey

249
00:21:10.200 --> 00:21:12.440
might be a little bit different depending
on which side you're coming from, because

250
00:21:12.480 --> 00:21:17.200
if you're a data scientist, you're
going to be a lot more familiar with

251
00:21:17.279 --> 00:21:22.680
these data science tools and less familiar
with the embedded side, and if you're

252
00:21:22.680 --> 00:21:26.319
an embedded engineer, then it will
be vice versa. Let's assume that you're

253
00:21:26.359 --> 00:21:30.480
a data scientist and you're excited about
getting started with this kind of thing.

254
00:21:30.559 --> 00:21:34.079
I think a good place to begin
you don't necessarily have to have embedded devices

255
00:21:34.119 --> 00:21:38.440
to start thinking about how do you
train smaller models. It can be interesting

256
00:21:38.480 --> 00:21:45.160
to just start building an awareness of
the relationship between the size of models and

257
00:21:45.319 --> 00:21:49.759
the amount of resources they consume and
just start trying to get into that mindset

258
00:21:49.799 --> 00:21:53.480
and tinker around with things and start
learning like, Oh, if you change

259
00:21:53.559 --> 00:21:57.000
this, what's the impact on performance? And you can start to learn about

260
00:21:57.039 --> 00:22:03.079
things like model compression and quanteization and
begin your journey that way. And the

261
00:22:03.240 --> 00:22:07.319
easiest place to start is probably with
the larger edge devices, things like a

262
00:22:07.440 --> 00:22:11.559
raspberry pie, because you can just
run python on there, so you can

263
00:22:11.599 --> 00:22:15.319
train a tensorflow model and then run
it directly on the device and there's not

264
00:22:15.400 --> 00:22:19.559
that much extra work involved. If
you're going down to micro controllers, so

265
00:22:19.599 --> 00:22:26.720
these are things like the tiny little
processes that live inside of household gadgets and

266
00:22:26.000 --> 00:22:32.559
really are super pervasive throughout our worlds
and are built environments. The workflow for

267
00:22:32.640 --> 00:22:37.799
programming with those is a lot more
complex, so they generally require you to

268
00:22:37.920 --> 00:22:42.920
write code in C or C plus
plus and then use a device specific set

269
00:22:42.960 --> 00:22:48.640
of compilation tools that allow you to
compile your code to run on that particular

270
00:22:48.640 --> 00:22:52.759
device. So are very easy to
use. Set of devices in that area

271
00:22:52.039 --> 00:22:56.799
made by a company called are Doino
and they make micro controllers that are designed

272
00:22:56.839 --> 00:23:00.880
to be easy to program and easy
to use and it can be really good

273
00:23:00.920 --> 00:23:04.799
starting point for embedded machine learning development. But it can be easy to get

274
00:23:04.839 --> 00:23:08.599
lost because there's so much going on
in this field. So what I would

275
00:23:08.640 --> 00:23:12.480
recommend? I'm a bit biased because
I work for a company called edge impulse

276
00:23:12.799 --> 00:23:18.480
and we have this amazing end to
end suite of tools which allows you to

277
00:23:18.920 --> 00:23:25.200
integrate with devices, collect data sets
and experiment with combinations of signal processing and

278
00:23:25.279 --> 00:23:30.359
machine learning models and then deploy to
devices. And I would recommend personally heading

279
00:23:30.400 --> 00:23:36.279
over to our documentation. Just search
for EDGE IMPULSE OR GO TO EDGE IMPULSE

280
00:23:36.359 --> 00:23:40.039
DOT COM and dig in and we've
got loads of tutorials which are designed to

281
00:23:40.480 --> 00:23:45.279
make it easy to jump in and
start learning about how all of these different

282
00:23:45.279 --> 00:23:48.720
moving parts relate to each other and
how you can train a model and deploy

283
00:23:48.759 --> 00:23:56.079
it. Is there a sort of
standard, easy first project, like what's

284
00:23:56.119 --> 00:24:02.559
the hello world equivalent for edge M
L? So I wrote a book along

285
00:24:02.599 --> 00:24:07.920
with Pete Warden from the tensorflow like
micro team a few years back, and

286
00:24:07.960 --> 00:24:12.759
the Hello World Project that we came
up for tiny ml was basically looking at

287
00:24:14.119 --> 00:24:18.200
can you train a tiny model just
to do some very arbitrary things? So

288
00:24:18.279 --> 00:24:23.440
for us it was to predict the
y value given the x value for a

289
00:24:23.480 --> 00:24:30.519
sine wave function. So trained the
simplest possible model and then just go through

290
00:24:30.960 --> 00:24:36.559
the process of converting it into a
form that will run on an embedded device

291
00:24:36.880 --> 00:24:41.119
and then getting it deployed on there
and having all of the parts of this

292
00:24:41.160 --> 00:24:45.039
process are quite difficult. Training an
effective model is difficult, collecting a good

293
00:24:45.119 --> 00:24:48.559
data set is difficult, signal processing
is difficult, deploying the device is difficult.

294
00:24:49.480 --> 00:24:55.160
So it's good to just take something
very easy and deploy it to an

295
00:24:55.160 --> 00:25:00.079
embedded device and get familiar with the
entire flow there. But that's sort of

296
00:25:00.160 --> 00:25:04.440
if you're approaching things from scratch and
doing everything yourself. But what I'd recommend

297
00:25:04.519 --> 00:25:10.480
it really is going to a platform
like edge impulse and following the steps to

298
00:25:10.559 --> 00:25:15.599
do something that trains a model that
runs on a simple time series. So,

299
00:25:15.880 --> 00:25:19.400
for example, a lot of micro
controller development boards will have a built

300
00:25:19.440 --> 00:25:25.440
in accelerometer, and a really good
thing to do is to train a model

301
00:25:25.480 --> 00:25:30.960
that recognizes different types of activity based
on data from an accelerometer. So accelerometer

302
00:25:32.039 --> 00:25:37.240
basically returns a time series of three
values. The three values are the amount

303
00:25:37.279 --> 00:25:41.559
of acceleration on the x axis,
the Y axis and the Z axis.

304
00:25:41.759 --> 00:25:45.440
So you get this idea of motion
in three D space and you can train

305
00:25:45.480 --> 00:25:51.319
a model to recognize different types of
activity, so like jogging versus doing push

306
00:25:51.400 --> 00:25:56.079
ups or something like that. You
collect a little data set with data for

307
00:25:56.240 --> 00:26:02.480
each of these motions and then train
use some signal processing to extract the person

308
00:26:02.559 --> 00:26:06.200
and information, train a little model
and then deploy it to the device,

309
00:26:06.240 --> 00:26:08.920
and then you can like wear this
thing on your arm and it will classify

310
00:26:08.960 --> 00:26:14.880
your activities, and that's really cool
because that's how a smart watch that people

311
00:26:14.920 --> 00:26:19.720
wear for fitness works. that can
understand when you're running versus swimming or something

312
00:26:19.759 --> 00:26:23.119
like that, and you can start
to see how this all fits together end

313
00:26:23.160 --> 00:26:26.559
to end. The tricky part is
if you don't have any embedded experience,

314
00:26:26.599 --> 00:26:32.839
it becomes really difficult even to grab
that initial data from a device. So

315
00:26:33.119 --> 00:26:36.599
that's why these sort of end to
end frameworks, end to end platforms,

316
00:26:36.599 --> 00:26:40.039
are so powerful, because they come
with all the software that you need in

317
00:26:40.119 --> 00:26:42.920
order to grab the data from the
device. Making your own fitness watch does

318
00:26:42.960 --> 00:26:47.640
sound like a pretty cool project,
but I can see how there are a

319
00:26:47.640 --> 00:26:51.920
lot of different steps where you need
to know a bit about a lot of

320
00:26:51.960 --> 00:26:56.319
things in order to make that.
So maybe you talk about when people are

321
00:26:56.319 --> 00:27:00.759
trying to do this, what's a
good idea and what's a bad idea.

322
00:27:00.960 --> 00:27:03.920
So perhaps can you tell me a
bit about what are common mistakes that people

323
00:27:04.039 --> 00:27:07.359
make when they're trying to do this
sort of thing? One of the things

324
00:27:07.440 --> 00:27:15.680
that I often come across is there's
a lot bigger of difference between test data

325
00:27:15.720 --> 00:27:22.279
and training data and real world data
if we're using sensor data versus if we're

326
00:27:22.119 --> 00:27:26.799
training something based on stuff out of
a database that came out of a piece

327
00:27:26.799 --> 00:27:32.680
of cloud software. So the thing
with tabular data that's based on, like,

328
00:27:32.759 --> 00:27:37.359
I don't know, the behavior of
users in a social network or something

329
00:27:37.400 --> 00:27:41.519
like that, is that the data
gets measured in the same way and that

330
00:27:41.680 --> 00:27:47.839
gets collected in the same way during
data collection and during production. It's just

331
00:27:47.920 --> 00:27:52.119
that we sort of took a snapshot
at a certain moment in time and we've

332
00:27:52.119 --> 00:27:57.720
got now a bunch of training and
testing data and after that time all the

333
00:27:57.759 --> 00:28:03.039
real world data is general rated in
exactly the same way. But with embedded

334
00:28:03.079 --> 00:28:07.119
machine learning you have a bit more
of a challenge around when you collect data.

335
00:28:07.240 --> 00:28:10.400
Imagine we're going to design a new
product. We're designing a new kind

336
00:28:10.400 --> 00:28:15.240
of smart watch that's supposed to help, I don't know, climbers and and

337
00:28:15.240 --> 00:28:18.839
give them some feedback on their climbing. So this device doesn't actually exist yet

338
00:28:19.319 --> 00:28:26.599
and before we get that device built, we need to design and train this

339
00:28:26.759 --> 00:28:30.359
machine learning system that can give the
climbers their feedback. But because we don't

340
00:28:30.359 --> 00:28:36.160
have any devices built yet, we've
got to somehow collect a data set without

341
00:28:36.200 --> 00:28:41.440
having our real world actual device available. And so to begin with we'll have

342
00:28:41.480 --> 00:28:45.559
to come up with some prototype that
just collects data from some sensors. Maybe

343
00:28:45.559 --> 00:28:48.240
it's sort of roughly the same size
and shape that the real thing will be,

344
00:28:48.599 --> 00:28:53.440
and have some people wear that device
and do some climbing and label the

345
00:28:53.519 --> 00:28:56.799
data in some way. Maybe it's
like, oh, that was a difficult

346
00:28:56.839 --> 00:29:02.599
climb versus an easy climb or something
like at and collect this whole data set

347
00:29:03.240 --> 00:29:06.480
and once we've done that, we
can train a model and we can test

348
00:29:06.519 --> 00:29:11.960
it and we can see, okay, this model can classify accurately whether this

349
00:29:11.039 --> 00:29:15.519
was an easy climb or a difficult
climb. But the problem is when we

350
00:29:15.599 --> 00:29:22.599
then get further through our product development
process and we've got our finished hardware design,

351
00:29:22.000 --> 00:29:26.599
maybe it's a slightly different shape and
size of watch, maybe it has

352
00:29:26.640 --> 00:29:30.400
a slightly different sensor on it.
That's like it's still an accelerometer, but

353
00:29:30.440 --> 00:29:36.279
it's a different model of accelerometer because
that one was cheaper and we needed to

354
00:29:36.319 --> 00:29:40.160
go with a cheaper one to hit
our target for how much the device costs.

355
00:29:40.160 --> 00:29:45.079
So when it gets deployed into the
real world, the device is slightly

356
00:29:45.119 --> 00:29:48.799
different and that means the data coming
off it is going to be slightly different

357
00:29:49.079 --> 00:29:55.079
and that means our model might not
necessarily work very well. And the real

358
00:29:55.200 --> 00:29:57.960
challenge is that, because all the
data is trapped at the edge of the

359
00:29:59.039 --> 00:30:03.799
network and the Divi ice is the
thing that's interpreting it, we don't necessarily

360
00:30:04.119 --> 00:30:07.680
have a way to check whether it
was working well or not. Right we

361
00:30:07.759 --> 00:30:14.880
can't just compare the real world performance
versus the predicted because we don't have that

362
00:30:15.000 --> 00:30:18.720
data and we don't have a way
to label it. So there's this big

363
00:30:18.839 --> 00:30:22.599
risk that you can end up training
a model that looks really good before it

364
00:30:22.680 --> 00:30:25.799
hits production and then when it hits
production, when it gets deployed to a

365
00:30:25.839 --> 00:30:29.319
real device, that doesn't work very
well. And that's obviously a risk in

366
00:30:29.400 --> 00:30:34.359
all of data science, but it's
especially amplified here because we've got less feedback

367
00:30:34.400 --> 00:30:41.440
loops between development and production. So
that's probably the biggest scariest thing that we

368
00:30:41.440 --> 00:30:45.160
we had to work around as a
field. Oh Wow, yeah, I'm

369
00:30:45.200 --> 00:30:48.599
sure that could be a problem,
and so how do companies deal with this

370
00:30:48.680 --> 00:30:52.000
sort of thing? Yeah, I
mean it's it's tricky and it's it's an

371
00:30:52.039 --> 00:30:59.839
ongoing thing that we're evolving systems and
workflows to deal with. But the most

372
00:31:00.039 --> 00:31:04.799
obvious thing is just keeping things as
similar as possible between production and development and,

373
00:31:04.920 --> 00:31:08.680
wherever possible, making sure you use
the same sensors, making sure that

374
00:31:08.720 --> 00:31:14.400
you're collecting data in the same way. But obviously that's not always possible.

375
00:31:14.559 --> 00:31:18.759
So then you've got to think about
how do you set up some ongoing processes

376
00:31:19.119 --> 00:31:26.480
to continue testing and making sure that
the system still works as the real world

377
00:31:26.559 --> 00:31:32.160
changes, because there's also obviously this
concept of drift where the real world is

378
00:31:32.200 --> 00:31:37.799
always changing and your model represents one
frozen moment of that. And that's even

379
00:31:37.839 --> 00:31:44.559
more amplified in this field because we're
dealing directly with real world data that's captured

380
00:31:44.599 --> 00:31:48.880
from sensors. So one of the
things that you can do is periodically send

381
00:31:48.880 --> 00:31:53.759
people out to capture more data and
test your model against that. So that

382
00:31:53.799 --> 00:31:59.319
can be a helpful signal. So
it's not as good as directly looking at

383
00:31:59.359 --> 00:32:02.039
what's happening in the real world,
but it's going to give you a periodic

384
00:32:02.559 --> 00:32:07.279
snapshot of what's happening and you can
see, okay, with this new data

385
00:32:07.319 --> 00:32:10.119
I've collected, the models still working. Another thing you can do is look

386
00:32:10.160 --> 00:32:16.160
at the output of your model.
So presumably these devices are smart watches that

387
00:32:16.440 --> 00:32:21.880
people are wearing for climbing. They
would be reporting data back into some system.

388
00:32:21.960 --> 00:32:25.559
So if they're determined that the climb
was challenging, then that gets logged

389
00:32:25.880 --> 00:32:30.359
in the users accounts somewhere so they
can see it later. So one thing

390
00:32:30.400 --> 00:32:34.319
you can do is look at the
distributions of the outputs of the model over

391
00:32:34.400 --> 00:32:37.799
time, because even if you can't
afford to save all of the raw senser

392
00:32:37.920 --> 00:32:43.519
data, you can potentially afford to
save the output of the model. And

393
00:32:43.720 --> 00:32:49.319
if the distributions of the output are
changing a lot over time or if they're

394
00:32:49.400 --> 00:32:53.000
very different to what they were on
your test data that you originally captured,

395
00:32:53.039 --> 00:32:58.160
for example, maybe that's a sign
that some drift has occurred or that the

396
00:32:58.200 --> 00:33:02.599
real world is occupying a different distribution
than the initial data set. So that

397
00:33:02.640 --> 00:33:06.480
could be a signal that you need
to collect some more data and do more

398
00:33:06.519 --> 00:33:10.839
evaluation. But just going back to
the climbing watch example, that sounds like

399
00:33:10.880 --> 00:33:14.920
it might just be a case of
you get your climbers back in and get

400
00:33:14.920 --> 00:33:19.279
them to climb the same routs before
using the production version of the watch rather

401
00:33:19.319 --> 00:33:24.319
than the prototype exactly. And maybe
you made the mistake of initially collecting data

402
00:33:24.359 --> 00:33:30.440
from professional climbers and then that you
found out that the people who are buying

403
00:33:30.480 --> 00:33:35.400
your watch in the public are just
amateurs, and so maybe you need to

404
00:33:35.440 --> 00:33:38.440
get some amateurs to come in and
collect data from them and that gives you

405
00:33:38.480 --> 00:33:43.359
a better representation of what's out there
in the real world. So all of

406
00:33:43.359 --> 00:33:45.559
this stuff is things you've got to
be really careful about in your sort of

407
00:33:45.599 --> 00:33:51.920
experiment design. Yeah, the amateurs
fall off the rock face. So,

408
00:33:52.240 --> 00:33:57.680
just flipping this around, are there
any particular best practices for working with edge?

409
00:33:58.240 --> 00:34:00.359
It's a big, big field,
so there's a lot of different individual

410
00:34:00.400 --> 00:34:07.079
places, but I think that the
most important thing for me is to try

411
00:34:07.079 --> 00:34:09.119
and figure out, because this is
such a big area, there are so

412
00:34:09.280 --> 00:34:14.719
so many steps in the workflow of
going from a data set to a model

413
00:34:14.760 --> 00:34:20.360
deployed on a device. Coming up
with a good understanding of all of the

414
00:34:20.440 --> 00:34:25.480
different moving parts at the very beginning
is very important. So understanding what is

415
00:34:25.519 --> 00:34:29.760
the hardware we're going to be deploying
this too. So much of these things

416
00:34:29.760 --> 00:34:34.880
are based around like figuring out a
good compromise based on the hardware that you'll

417
00:34:34.920 --> 00:34:38.119
be deploying to. And so is
that known ahead of time? Do we

418
00:34:38.159 --> 00:34:42.719
get to choose the hardware based on
what kind of model we end up training,

419
00:34:42.800 --> 00:34:45.360
or is it more the other way
around, that we've got a device

420
00:34:45.400 --> 00:34:49.880
that already exists and we need to
fit the model in its spare memory?

421
00:34:50.239 --> 00:34:53.039
So getting that pinned down at the
beginning is really really important and that will

422
00:34:53.119 --> 00:34:58.559
inform the entire rest of the project. The second thing that's really important,

423
00:34:58.599 --> 00:35:02.039
I think, is just having an
understanding of what are we trying to achieve

424
00:35:02.159 --> 00:35:08.159
here. What's the minimum thing we
can do that will result in a useful

425
00:35:08.199 --> 00:35:15.920
product, because sometimes it's good to
aim for that basic improvement rather than saying

426
00:35:15.960 --> 00:35:20.360
like we want to do all these
different things and make this sort of all

427
00:35:20.400 --> 00:35:24.679
singing, all dancing, magical ai
product and then trying and failing. It's

428
00:35:24.719 --> 00:35:30.199
better to choose one simple, straightforward
thing which is going to be helpful to

429
00:35:30.320 --> 00:35:37.199
your customers and then deliver on that
and then work up from there and gradually

430
00:35:37.360 --> 00:35:40.360
add more features and improve things,
but try and bite off something simple to

431
00:35:40.440 --> 00:35:45.400
begin with, because this can be
a really challenging field. I'd like to

432
00:35:45.440 --> 00:35:49.599
talk a little bit more about some
of the use cases. So you mentioned

433
00:35:49.599 --> 00:35:53.320
that really high tech thing with the
Mars rover, and then there's a sort

434
00:35:53.320 --> 00:35:58.239
of consumer product idea of this climbing
watch, and then he talked a bit

435
00:35:58.280 --> 00:36:02.039
about centers and things. So can
you just give me some more examples of

436
00:36:02.079 --> 00:36:07.559
where this might get used? Edge
impulse we have customers across pretty much any

437
00:36:07.599 --> 00:36:15.239
space you can imagine. It covers
everything from consumer products that use a little

438
00:36:15.239 --> 00:36:19.880
bit of intelligence to do something smarter
in the home, all the way through

439
00:36:19.880 --> 00:36:25.039
to industrial systems that are understanding what's
going on in a factory, maybe analyzing

440
00:36:25.079 --> 00:36:30.639
products for quality after they've been manufactured
so you can understand whether there are flaws

441
00:36:30.639 --> 00:36:37.119
in your manufacturing process. Healthcare is
an interesting area, interpreting bio signals using

442
00:36:37.400 --> 00:36:44.199
machine learning to understand how someone's body
is functioning or the progression of a disease.

443
00:36:44.679 --> 00:36:50.840
Agricultures are really interesting one, understanding
how plants are growing or how animal

444
00:36:50.880 --> 00:36:57.760
health is or understanding climate. So
anywhere where there's rex connection to the real

445
00:36:57.880 --> 00:37:01.280
world and tangible things that are going
on and we need to know more about

446
00:37:01.320 --> 00:37:07.199
it. Any field you can think
of there are applications for embedded machine learning.

447
00:37:07.719 --> 00:37:10.679
Yeah, it's amazing machine learnings are
sort of invading every part of our

448
00:37:10.719 --> 00:37:15.960
lives. So, yeah, it's
interesting that. That's something I've really find

449
00:37:15.360 --> 00:37:22.400
exciting about this is that until now
machine learning has been something that's existed in

450
00:37:22.440 --> 00:37:28.599
a constrained set of places that it
will fit right. We've got these big,

451
00:37:28.679 --> 00:37:35.239
huge models running on powerful computers,
but now we're able to push machine

452
00:37:35.320 --> 00:37:39.039
learning down to smaller and smaller devices
so it can fit more neatly into little

453
00:37:39.079 --> 00:37:45.000
corners of our lives. It democratizes
things a little bit. Rather than needing

454
00:37:45.039 --> 00:37:49.960
these one size fits all giant models
that are running in the cloud at some

455
00:37:50.000 --> 00:37:53.679
big tech companies data center, we
can have little, tiny models that are

456
00:37:53.719 --> 00:38:00.760
trained by people who are working in
little corners of the world old whether it's

457
00:38:00.800 --> 00:38:05.880
in some niche field or a place
that has particular problems and challenges that we

458
00:38:05.880 --> 00:38:08.480
can address as machine learning. And
it's a lot more feasible to get good

459
00:38:08.480 --> 00:38:15.480
performance out of these smaller models in
little constrained contexts rather than trying to create

460
00:38:15.519 --> 00:38:21.559
these big giant models that do everything. So lots of tiny, tiny things

461
00:38:21.599 --> 00:38:27.079
on tiny machines. So that's cool. This is very much an emerging field

462
00:38:27.280 --> 00:38:30.360
and I get the feeling that there's
lots of progress being made, but maybe

463
00:38:30.840 --> 00:38:35.920
there are some sorts of limitations at
the moment. So perhaps you can talk

464
00:38:35.960 --> 00:38:39.880
a bit about where the sticking points
are, particularly with tiny m l where

465
00:38:40.199 --> 00:38:45.119
I imagine you've got even stronger constraints. So one of the biggest things is

466
00:38:45.159 --> 00:38:54.239
around tooling and essentially, if you
are working and embedded software or hardware design,

467
00:38:54.719 --> 00:39:00.280
chances are if you're building something in
agriculture, for example, you're eam

468
00:39:00.320 --> 00:39:04.639
is going to have a lot of
experience and a lot of expertise around agriculture

469
00:39:04.679 --> 00:39:09.039
and you're using that to design and
build an effective product that works in that

470
00:39:09.079 --> 00:39:14.239
domain. So if you are working
in that domain and you're you're already a

471
00:39:14.320 --> 00:39:20.239
deep expert in your domain and you're
a deep expert in embedded engineering, you

472
00:39:20.320 --> 00:39:25.320
probably don't have the bandwidth to also
become a deep expert in machine learning.

473
00:39:25.920 --> 00:39:30.280
If you want to use machine learning
as part of a project to begin with,

474
00:39:30.320 --> 00:39:36.679
those this huge really difficult learning curve
where you've got to become an expert

475
00:39:36.679 --> 00:39:39.519
in m l and then apply that
knowledge to the problem that you're trying to

476
00:39:39.559 --> 00:39:45.880
solve. So the big problem and
the big challenges how do we make it

477
00:39:45.000 --> 00:39:51.599
so that people who are domain experts
in embedded engineering and in whatever field they're

478
00:39:51.639 --> 00:39:57.599
working in can use machine learning without
having to become a machine learning expert,

479
00:39:57.639 --> 00:40:00.920
without having to take years to study
field and learn the INS and outs of

480
00:40:01.000 --> 00:40:05.880
data science? So that's where there's
a huge amount of opportunity in this field

481
00:40:05.960 --> 00:40:12.000
is building tools that empower people who
are not from machine learning and data science

482
00:40:12.000 --> 00:40:16.760
background and obviously to build those tools
requires data science and machine learning expertise.

483
00:40:17.119 --> 00:40:23.360
So there's there's a huge opportunity and
a huge challenge there in adjusting and adapting

484
00:40:23.480 --> 00:40:30.239
these tool chains so that people who
are experts in one little subject area that's

485
00:40:30.360 --> 00:40:37.920
very important can make use of them. That agriculture example really resonates with me.

486
00:40:37.199 --> 00:40:42.280
So my father in law was a
sheep farmer and he had decades and

487
00:40:42.320 --> 00:40:45.440
decades and decades of farming experience.
I don't think he ever touched computer in

488
00:40:45.440 --> 00:40:51.280
his life and so getting him to
think about machine learning would very much be

489
00:40:51.320 --> 00:40:53.880
a no go area. But trying
to get all that domain knowledge you had

490
00:40:53.880 --> 00:40:59.360
into some kind of model or would
be a very interesting challenge. So I

491
00:40:59.360 --> 00:41:02.039
can see how it would be a
difficult thing. Yeah, absolutely, and

492
00:41:02.119 --> 00:41:08.159
that's even something we've directly seen.
People using edge ai models to recognize when

493
00:41:08.480 --> 00:41:14.440
sheep are in different parts of a
farm or walking across the road, or

494
00:41:15.039 --> 00:41:20.239
devices that are a warn around an
animal's neck that keep track of their bio

495
00:41:20.320 --> 00:41:23.559
signals and their activity so that you
can understand how healthy they are. So

496
00:41:23.920 --> 00:41:29.679
how are things progressing at the moment? What are the most important challenges that

497
00:41:29.719 --> 00:41:34.599
are being worked on? So right
now I think some of the biggest,

498
00:41:35.400 --> 00:41:40.280
most important work is in this field
of opening up access to people and allowing

499
00:41:40.320 --> 00:41:46.920
people to understand the machine learning workflow
without being a machine learning expert, basically

500
00:41:46.960 --> 00:41:52.760
building tools to understand sense of data
and be able to look at how good

501
00:41:53.039 --> 00:41:58.639
a data set is, of sense
of data and understand quality and give people

502
00:41:58.719 --> 00:42:01.719
feedback around like, Oh, you
should be collecting more examples of this type

503
00:42:01.719 --> 00:42:07.519
of thing. One of the biggest
difficulties is that collecting labeled data can be

504
00:42:07.639 --> 00:42:13.119
very difficult. If you think about
sensor data, essentially it's like big,

505
00:42:13.199 --> 00:42:17.639
massive time series of numbers and it
can be quite inscrutable. You can't necessarily

506
00:42:19.039 --> 00:42:22.599
look at a table of all of
these rows, like a thousand rows per

507
00:42:22.719 --> 00:42:29.599
second, of some arbitrary value that
gets read from a sensor and make any

508
00:42:29.639 --> 00:42:32.760
sense of it as a person.
So we need to come up with techniques

509
00:42:32.800 --> 00:42:37.280
that allow people to take a big
data set of like hundreds of hours of

510
00:42:37.519 --> 00:42:45.039
unlabeled high frequency sensor data and label
it and then be able to use it

511
00:42:45.079 --> 00:42:49.599
for training or perhaps be able to
do some kind of training that doesn't depend

512
00:42:49.639 --> 00:42:52.559
on labels. So that's that's a
really big thing. Another big thing is

513
00:42:52.760 --> 00:43:00.559
understanding the performance of models on this
type of data and explaining how models are

514
00:43:00.559 --> 00:43:05.920
performing and where they're working well and
where they aren't, across a big data

515
00:43:06.000 --> 00:43:10.119
set of this type of difficult to
interpret data. One thing that's quite surprising

516
00:43:10.159 --> 00:43:16.199
in this field that we found is
that the device constraints around memory and latency

517
00:43:16.280 --> 00:43:22.440
and so on aren't as big an
obstacle as you would think, because it

518
00:43:22.480 --> 00:43:29.400
turns out if you use some intelligent
signal processing and you train a small model,

519
00:43:29.760 --> 00:43:34.039
you can often do a lot with
very minimal hardware. We're used to

520
00:43:34.239 --> 00:43:37.320
turning things up to the Max and
going with the biggest model that you can

521
00:43:37.360 --> 00:43:40.880
get away with without overfitting and and
all of that type of thing. But

522
00:43:42.079 --> 00:43:45.280
if you're thinking about fitting models onto
these devices, there's a surprising amount you

523
00:43:45.280 --> 00:43:50.000
can do with what little space you
have. There are a lot of people

524
00:43:50.039 --> 00:43:53.559
in the field looking into compression and
how can you fit bigger and bigger models

525
00:43:53.559 --> 00:43:59.119
onto smaller and smaller devices, but
it's quite rare for us that we have

526
00:43:59.159 --> 00:44:02.800
a customer who needs to do something
that requires a model that's too big to

527
00:44:02.840 --> 00:44:07.280
fit on an embedded device. It's
more often that they don't have a good

528
00:44:07.400 --> 00:44:12.159
enough data set to be able to
train that model in the first place.

529
00:44:13.440 --> 00:44:17.000
That seems to be a common thing
with machine learning, is that actually the

530
00:44:17.159 --> 00:44:22.000
hard part is getting good data in
order to run the model, rather than

531
00:44:22.199 --> 00:44:27.760
the actual modeling part itself. Absolutely, and especially with sense of data,

532
00:44:27.880 --> 00:44:32.960
where, if you're looking at things
like natural language or image data, there

533
00:44:34.000 --> 00:44:37.719
are huge data sets out there that
have been scraped from the Internet that people

534
00:44:37.239 --> 00:44:43.199
used to be able to use for
creating and benchmarking new types of models.

535
00:44:43.239 --> 00:44:47.800
But with sensor data there just isn't
that much out there and people who have

536
00:44:47.880 --> 00:44:55.320
access to sensor data are quite wary
about sharing it because it represents significant intellectual

537
00:44:55.360 --> 00:45:00.440
property on their part. If you
have a company that man factures some kind

538
00:45:00.480 --> 00:45:05.519
of widget and you've got all sorts
of sensors on all of your equipment that

539
00:45:05.920 --> 00:45:12.519
monitor how this manufacturing is going,
that represents valuable data that your company can

540
00:45:12.639 --> 00:45:17.440
use to potentially improve its processes and
you don't necessarily want your competitors getting ahold

541
00:45:17.440 --> 00:45:22.199
of it. So that's a challenge
as well. Is like the availability of

542
00:45:22.280 --> 00:45:29.000
data sets and the issues around privacy
and sharing of data sets. Yeah,

543
00:45:29.039 --> 00:45:31.239
I can see how that would be
a problem. Maybe if there are any

544
00:45:31.320 --> 00:45:37.119
listeners who have sensor data sets that
they can publicly share, then I encourage

545
00:45:37.159 --> 00:45:39.880
you to do that if you can
do it without causing any privacy problems or

546
00:45:40.000 --> 00:45:45.000
IP problems. So this whole thing
seems like it's at a point where it

547
00:45:45.119 --> 00:45:52.039
might be about to explode in popularity. It feels like the toolings coming together

548
00:45:52.320 --> 00:45:57.639
and the techniques are coming together and
some of the processes. So what do

549
00:45:57.679 --> 00:46:01.679
you think is going to drive adoption
of Edge Mul over the next few years?

550
00:46:02.559 --> 00:46:07.760
Yeah, the way we think about
this is sort of there's this massive

551
00:46:07.880 --> 00:46:15.280
untapped potential based on sensor data that
we've got. This this huge, incredibly

552
00:46:15.400 --> 00:46:21.840
valuable pools of data basically, that
exist in the real world that are completely

553
00:46:22.639 --> 00:46:27.880
discarded right now. Most data that's
connected, collected by IOT devices, for

554
00:46:27.920 --> 00:46:31.760
example, isn't really used for anything. And we've got all this technology around

555
00:46:31.840 --> 00:46:37.440
sensors and embedded devices that can collect
lots of data about real world situations,

556
00:46:37.639 --> 00:46:42.440
but we're basically not making any use
of it yet because of these bandwidth and

557
00:46:42.960 --> 00:46:46.840
latency constraints, things like that.
So what's going to drive this is that

558
00:46:46.920 --> 00:46:55.039
there's huge untapped value in this data
throughout industry and consumer devices and everywhere,

559
00:46:57.119 --> 00:47:00.440
and suddenly we've got the tools that
allow us to make use of it to

560
00:47:00.840 --> 00:47:07.079
build better products and do things more
intelligently and be more efficient and create higher

561
00:47:07.159 --> 00:47:13.400
quality products on a manufacturing line,
for example, or to monitor our natural

562
00:47:13.480 --> 00:47:19.079
world to understand how ecologies and the
environment are doing and how how the climate

563
00:47:19.239 --> 00:47:22.679
is changing. So there's a huge
need to interpret all of this data so

564
00:47:22.719 --> 00:47:27.599
that we can make use of it, and this is the technology that allows

565
00:47:27.679 --> 00:47:30.639
us to do that. Exciting Times. So people who want to give us

566
00:47:30.679 --> 00:47:36.639
a try that maybe don't necessarily want
to go it alone. Are there any

567
00:47:36.679 --> 00:47:42.119
good communities for people who want to
get started? There's a really awesome group

568
00:47:42.159 --> 00:47:46.239
called the tiny ml foundation, which
has existed since quite early on in this

569
00:47:46.320 --> 00:47:52.599
field, who run a series of
meetup groups across the world. There's dozens

570
00:47:52.639 --> 00:47:58.480
of them where you can go to
local meetups with other people who are interested

571
00:47:58.519 --> 00:48:01.320
in this field. So if you
go to tiny ml dot org you can

572
00:48:01.360 --> 00:48:06.920
find some links to those. And
the tiny email foundation also run a conference

573
00:48:07.280 --> 00:48:10.840
or cut a few conferences every year
in different parts of the world, and

574
00:48:12.119 --> 00:48:16.039
we do these weekly or bi weekly
talks as well that you can watch.

575
00:48:16.159 --> 00:48:21.079
That a broadcast live and they're also
on Youtube and so on. There's also

576
00:48:21.119 --> 00:48:23.880
a few forums around, so edge
impulse forum is a really good place to

577
00:48:23.920 --> 00:48:28.000
go. If you go to the
EDGE IMPULSE DOT COM website, there's a

578
00:48:28.039 --> 00:48:31.519
link there and there's a ton of
people hanging out talking about embedded machine learning.

579
00:48:31.840 --> 00:48:36.199
Another good thing you can do is
there there are a few courses available.

580
00:48:36.239 --> 00:48:40.400
So there's a coursera course which was
put together by some folks at edge

581
00:48:40.440 --> 00:48:46.079
impulse, which basically gives you an
introduction to how to start working with this

582
00:48:46.159 --> 00:48:52.920
technology. There's also a Harvard Ed
x course which goes into the more sort

583
00:48:52.920 --> 00:48:59.519
of in depth theory behind things.
So if you're excited about the behind the

584
00:48:59.599 --> 00:49:04.760
scene part, the more like Matthew
stuff, the Harvard course is a really

585
00:49:04.800 --> 00:49:07.639
good place to go. If you're
excited about the application stuff and how do

586
00:49:07.679 --> 00:49:13.000
I, just as quickly as possible, start building things with this technology,

587
00:49:13.159 --> 00:49:16.679
then the coursera course is really good. I should probably add that for anyone

588
00:49:16.679 --> 00:49:22.280
who's interested in analyzing censor data,
we do have a data camp course as

589
00:49:22.320 --> 00:49:27.280
well on analyzing the Internet of things
data in Python. That sounds like a

590
00:49:27.320 --> 00:49:30.159
really good place to start as well. Yeah, it's a very gentle introduction

591
00:49:30.559 --> 00:49:36.199
and lots of time series analysis and
a bit of machine learning. So before

592
00:49:36.239 --> 00:49:39.000
we finish, do you have any
final advice for people who want to try

593
00:49:39.039 --> 00:49:43.920
this themselves? Yeah, so don't
be scared, like this is a big

594
00:49:44.000 --> 00:49:46.159
field. There's a lot of moving
parts, a lot of complexity, but

595
00:49:46.480 --> 00:49:52.800
it's definitely possible to focus on the
bit that you care about and sort of

596
00:49:52.840 --> 00:49:57.320
ignore the rest of the complexity if
you use the right tools. So I

597
00:49:57.360 --> 00:50:02.239
wouldn't recommend most people asking from scratch
and trying to learn all of these little

598
00:50:02.719 --> 00:50:07.639
pieces and all the moving parts at
the same time. If you're already interested

599
00:50:07.639 --> 00:50:14.199
in data science and machine learning,
then find a platform that automates the embedded

600
00:50:14.239 --> 00:50:16.119
side of it for you, so
it makes it easy to collect some data

601
00:50:16.159 --> 00:50:21.360
sets, makes it easy to deploy
to advice and then you can focus on

602
00:50:21.400 --> 00:50:24.840
the machine learning part. Otherwise,
if you're an embedded engineer and you're excited

603
00:50:24.840 --> 00:50:30.679
about doing that part and you don't
necessarily want to spend years becoming a data

604
00:50:30.719 --> 00:50:36.199
science expert, the same sort of
platforms will help you focus on the embedded

605
00:50:36.239 --> 00:50:38.960
part and building the products you want
without having to go back to school.

606
00:50:39.440 --> 00:50:45.320
So definitely don't be afraid to use
tools to help you and don't try and

607
00:50:45.440 --> 00:50:50.000
start everything from scratch, because it's
very difficult. Okay, thank you very

608
00:50:50.079 --> 00:50:52.880
much for answering all the questions.
It's been a real pleasure having you on

609
00:50:52.880 --> 00:50:55.880
the podcast. Thank you very much, Dan. Likewise, thank you,

610
00:50:57.000 --> 00:51:05.159
Richie. It's been awesome chatting with
you today. Ye, you've been listening

611
00:51:05.159 --> 00:51:08.480
to data framed, a podcast by
data camp. Keep connected with us by

612
00:51:08.519 --> 00:51:13.920
subscribing to the show in your favorite
podcast player. Please give us a rating,

613
00:51:14.159 --> 00:51:17.840
leave a comment and share episodes.
You love that helps us keep delivering

614
00:51:17.920 --> 00:51:22.960
insights into all things data. Thanks
for listening. Until next time,