1
00:00:00,000 --> 00:00:03,003
and spent a decade in quantitative
finance.
2
00:00:03,003 --> 00:00:06,506
Really building machine
learning and deep learning models
3
00:00:06,506 --> 00:00:09,042
to trade financial futures stocks.
4
00:00:09,042 --> 00:00:12,912
And really kind of the same space
where we're anomaly detecting here
5
00:00:13,046 --> 00:00:14,080
at Buzz Solutions.
6
00:00:14,080 --> 00:00:18,718
That's a subset of this kind of huge,
broad umbrella of what can you use AI for?
7
00:00:19,019 --> 00:00:21,221
So AI doesn't replace the team.
8
00:00:21,221 --> 00:00:24,190
AI is is now on the team.
9
00:00:24,591 --> 00:00:24,924
Yeah.
10
00:00:24,924 --> 00:00:26,393
Or just elevates the team.
11
00:00:26,393 --> 00:00:29,129
Really the team gets to do their job
and they get to apply
12
00:00:36,903 --> 00:00:37,737
Welcome to the
13
00:00:37,737 --> 00:00:40,707
maiden Voyage of the and beyond podcast.
14
00:00:40,974 --> 00:00:43,977
I'm Stephen Ward, the marketing manager
for Bus Solutions.
15
00:00:44,344 --> 00:00:47,347
And today's guest is Nick libertine.
16
00:00:47,480 --> 00:00:50,717
He's the machine
learning team leader for buzz.
17
00:00:51,117 --> 00:00:53,820
And our topic is going to be leading
transformation.
18
00:00:53,820 --> 00:00:56,589
Thanks for joining us today, Nick.
19
00:00:56,589 --> 00:00:56,790
Yeah.
20
00:00:56,790 --> 00:00:59,292
Thanks, David, and excited to be here.
21
00:00:59,292 --> 00:01:01,728
So you've been involved in,
22
00:01:01,728 --> 00:01:04,831
in AI and machine learning,
since the beginning.
23
00:01:04,831 --> 00:01:06,499
So you started out you started out.
24
00:01:06,499 --> 00:01:09,502
You've got a degree in mechanical
engineering
25
00:01:09,502 --> 00:01:12,939
and economics
and then a masters in finance.
26
00:01:12,939 --> 00:01:15,942
So you're kind of a dual threat
technical and business.
27
00:01:17,510 --> 00:01:17,844
Yeah.
28
00:01:17,844 --> 00:01:19,245
Like you said, I've been doing,
29
00:01:19,245 --> 00:01:23,316
you know, machine learning and AI stuff
for the better part of 17 years now.
30
00:01:23,316 --> 00:01:27,587
And yeah, I did get a degree in finance
after I got my engineering degree.
31
00:01:27,620 --> 00:01:31,825
I was super passionate
and excited about the financial markets
32
00:01:31,825 --> 00:01:34,828
and spent a decade in quantitative
finance.
33
00:01:34,828 --> 00:01:38,331
Really building machine
learning and deep learning models
34
00:01:38,331 --> 00:01:41,334
to trade financial futures stocks.
35
00:01:41,434 --> 00:01:43,837
Again, the technology was a lot
newer then.
36
00:01:43,837 --> 00:01:47,407
We weren't using as big models
as complex models as we were today,
37
00:01:47,407 --> 00:01:50,176
but definitely
been doing this for a while now.
38
00:01:50,176 --> 00:01:50,877
Yeah. Okay.
39
00:01:50,877 --> 00:01:53,580
So you started with time series data
40
00:01:53,580 --> 00:01:57,817
and looking for results
that would help make money pretty much.
41
00:01:58,284 --> 00:02:00,353
Right? Yeah.
42
00:02:00,353 --> 00:02:04,624
Most of what we were doing was analyzing
the recent history of stocks
43
00:02:04,624 --> 00:02:09,028
and financial futures, looking back over
the last few weeks and months
44
00:02:09,028 --> 00:02:13,066
and trying to identify
or teach the model to find patterns
45
00:02:13,066 --> 00:02:16,503
in the data, that would be reproducible,
trades that we would take.
46
00:02:16,503 --> 00:02:19,973
And so a lot of what we were doing
was if the stocks go down
47
00:02:19,973 --> 00:02:23,343
over a few days, what's the probability
they're going to be up the next day?
48
00:02:23,343 --> 00:02:26,613
Or if this basket of stocks
deviates from this basket?
49
00:02:26,880 --> 00:02:29,516
Is there a convergence trade
or something like that.
50
00:02:29,516 --> 00:02:33,086
So we were building models
to exploit those
51
00:02:33,453 --> 00:02:35,922
and putting in real dollars
at risk really.
52
00:02:35,922 --> 00:02:38,758
And then in 2021,
53
00:02:38,758 --> 00:02:41,728
it looks like you moved to the, at MIT.
54
00:02:41,895 --> 00:02:43,730
Is it the Lincoln Laboratory?
55
00:02:45,632 --> 00:02:46,799
Yeah, I did
56
00:02:46,799 --> 00:02:50,036
a decade in quantitative finance
and had a lot of fun.
57
00:02:50,370 --> 00:02:53,439
But at that point in time,
I saw an opportunity
58
00:02:53,439 --> 00:02:56,976
to kind of pivot into the DoD
or intelligence space.
59
00:02:57,410 --> 00:03:02,148
Lincoln Lab is, a partnership
between the Department of Defense and MIT,
60
00:03:02,282 --> 00:03:05,685
and they do a lot of research
for defense initiatives.
61
00:03:05,685 --> 00:03:08,888
And so I got an opportunity
to go work on a team there.
62
00:03:08,888 --> 00:03:12,058
We were building
complex computer vision algorithms
63
00:03:12,358 --> 00:03:15,361
to try and solve
some national security problems.
64
00:03:15,395 --> 00:03:16,496
Okay.
65
00:03:16,496 --> 00:03:19,933
We we won't go into details on these,
of course, because they're confidential.
66
00:03:19,933 --> 00:03:22,101
But, I did hear computer vision.
67
00:03:22,101 --> 00:03:25,271
So this is when you started
applying, machine learning
68
00:03:25,271 --> 00:03:28,274
and AI to imagery?
69
00:03:28,775 --> 00:03:30,977
Yeah, it was a bit of a career shift,
70
00:03:30,977 --> 00:03:34,747
you know, switching from financial markets
into this space.
71
00:03:34,747 --> 00:03:36,716
But honestly,
there was a lot of carryover.
72
00:03:36,716 --> 00:03:41,254
I mean, a lot of the foundations
of building models for financial markets,
73
00:03:41,721 --> 00:03:44,757
wanting to make sure
that what you're building on
74
00:03:44,757 --> 00:03:47,894
historical data will work in production,
carried over here as well.
75
00:03:47,894 --> 00:03:50,496
You know, we're building models,
and we want to make sure that
76
00:03:50,496 --> 00:03:51,497
what we were building in
77
00:03:51,497 --> 00:03:55,568
the lab would carry out into the field
and work for the warfighter.
78
00:03:55,768 --> 00:03:59,572
Okay, so after the Lincoln lab,
then you joined Buzz Solutions,
79
00:03:59,572 --> 00:04:03,376
and now you're working on, imagery
for critical infrastructure.
80
00:04:03,376 --> 00:04:04,811
Yeah, yeah.
81
00:04:04,811 --> 00:04:08,448
It was kind of a natural transition
there as far as leading a team
82
00:04:08,448 --> 00:04:11,184
at Lincoln Lab
to now leading this fantastic team here
83
00:04:11,184 --> 00:04:14,187
at both Solutions,
we're solving very challenging problems.
84
00:04:14,187 --> 00:04:18,091
Like we were at Lincoln Lab
using a lot of the same tools
85
00:04:18,091 --> 00:04:21,094
as far as imagery captured from drones
and that type of stuff.
86
00:04:21,327 --> 00:04:25,198
And really kind of the same space
where we're anomaly detecting here
87
00:04:25,331 --> 00:04:26,366
upon solution.
88
00:04:26,366 --> 00:04:31,004
That's a subset of this kind of huge,
broad umbrella of what can you use AI for?
89
00:04:31,204 --> 00:04:34,707
One of the things you can use
is training models to find anomalies
90
00:04:35,041 --> 00:04:36,409
in these image data sets.
91
00:04:36,409 --> 00:04:37,877
And you really want to make sure
92
00:04:37,877 --> 00:04:40,947
that when you're training those
and they're working on your kind of
93
00:04:41,281 --> 00:04:44,817
training data set, that they'll also
be applicable and find stuff in the field
94
00:04:44,817 --> 00:04:48,388
when you actually run your models out
and try and put them into practice.
95
00:04:48,688 --> 00:04:53,926
So across your career, you across finance
and the national security
96
00:04:53,926 --> 00:04:56,296
and critical infrastructure,
what are some of the key lessons
97
00:04:56,296 --> 00:04:59,666
you've learned about applying machine
learning to these data sets?
98
00:05:01,267 --> 00:05:03,670
Yeah,
I think one of the biggest through lines
99
00:05:03,670 --> 00:05:07,407
that's kind of carried through
my entire career is you really,
100
00:05:07,640 --> 00:05:11,878
when you're training a model,
whether it's to trade stocks or,
101
00:05:12,612 --> 00:05:16,082
to predict anomalies,
you really have to have trust that
102
00:05:16,449 --> 00:05:19,852
when you're done training it,
that it's learned something useful
103
00:05:20,053 --> 00:05:23,423
that will be applicable to data
it's never seen during training.
104
00:05:23,456 --> 00:05:27,527
It's very easy to fool yourself
when you're building, you know, kind of
105
00:05:27,527 --> 00:05:32,899
black box artificial intelligence models
that, hey, this model has 90% accurate.
106
00:05:33,032 --> 00:05:35,668
It's performing
really good on my training set.
107
00:05:35,668 --> 00:05:39,572
But that's not the data
that your model is going to be used on.
108
00:05:39,572 --> 00:05:42,642
It's actually going to be used on data
it's likely never seen before.
109
00:05:42,942 --> 00:05:46,212
And so you have to build up
a set of robust evaluation
110
00:05:46,212 --> 00:05:49,215
criteria
to say, look, we've trained this model.
111
00:05:49,282 --> 00:05:52,919
It's really good at detecting this type
of anomaly on power line equipment,
112
00:05:53,353 --> 00:05:56,889
and it actually works on new imagery
that we've never seen before.
113
00:05:56,889 --> 00:05:57,657
And so building
114
00:05:57,657 --> 00:06:01,494
that strong validation gives
you confidence that the model will work
115
00:06:02,028 --> 00:06:06,299
in the field or in production when you're
going to apply it to new imagery.
116
00:06:07,333 --> 00:06:08,201
Okay.
117
00:06:08,201 --> 00:06:10,570
Now now was critical infrastructure.
118
00:06:10,570 --> 00:06:12,572
It seems to me
there's a bit of a difference,
119
00:06:12,572 --> 00:06:16,476
because what you're trying to do with
critical infrastructure is not necessarily
120
00:06:16,476 --> 00:06:22,081
be, 100% precise, but you're trying
not to miss something important, right.
121
00:06:22,081 --> 00:06:25,084
So how does that affect the model
development?
122
00:06:25,885 --> 00:06:28,020
Yeah, yeah, it's a great point.
123
00:06:28,020 --> 00:06:31,858
I think that there's a lot of different
areas that you can optimize for a model.
124
00:06:32,158 --> 00:06:34,961
One of those things that you can do,
like you mentioned,
125
00:06:34,961 --> 00:06:38,398
is optimize the model's ability
to find anomalies.
126
00:06:38,398 --> 00:06:41,734
In that case, you're trying
to find those needles in the haystack,
127
00:06:42,268 --> 00:06:45,171
and it's more important
that you acknowledge that, hey, maybe
128
00:06:45,171 --> 00:06:49,642
there's only 5 or 6 of these anomalies out
in my data of thousands of images.
129
00:06:50,042 --> 00:06:51,177
I want to make sure I catch
130
00:06:51,177 --> 00:06:54,981
as many of those,
even if I have some false detections.
131
00:06:54,981 --> 00:06:57,817
Hey, this looks like what I'm looking for,
but it's not quite.
132
00:06:57,817 --> 00:07:02,789
And so there you're optimizing for recall,
and you really have to coach the models
133
00:07:03,089 --> 00:07:06,659
to try and think
and look for those anomalies
134
00:07:06,659 --> 00:07:10,696
in, in all the different ways
they can manifest among the huge pile
135
00:07:10,696 --> 00:07:14,867
of nominal data that you're training,
your data or your models on.
136
00:07:15,368 --> 00:07:19,138
And that's completely usually inverse
of what you're doing in, say, finance
137
00:07:19,138 --> 00:07:23,776
or quantitative training,
where a lot of the focus is on precision.
138
00:07:24,243 --> 00:07:28,648
You have an opportunity
to trade any minute, any day in the hour.
139
00:07:28,915 --> 00:07:31,918
So there's a lot of opportunity
to put capital at risk.
140
00:07:32,185 --> 00:07:36,055
But you want to make sure that when you do
deploy capital and your model predicts,
141
00:07:36,055 --> 00:07:40,026
hey, hey, take this trade, that it's right
more often than not, but
142
00:07:40,026 --> 00:07:44,263
it has high precision that you're putting
your capital at risk in good situations.
143
00:07:44,263 --> 00:07:46,466
So they're kind of inverse of each other.
144
00:07:46,466 --> 00:07:51,003
And the way you coach the models
to be receptive to either high precision
145
00:07:51,237 --> 00:07:55,775
situations or high recall situations
is really baked into how the machine
146
00:07:55,775 --> 00:07:59,612
learning team or team trains their models,
and how they validate their models
147
00:07:59,612 --> 00:08:00,112
as well.
148
00:08:01,047 --> 00:08:01,814
Yeah, very
149
00:08:01,814 --> 00:08:05,351
interesting because a lot of buzz about
using synthetic data to train models.
150
00:08:05,351 --> 00:08:08,354
What are your thoughts
on the use of synthetic data?
151
00:08:09,188 --> 00:08:11,791
Yeah, I think, you know, in this
152
00:08:11,791 --> 00:08:14,260
era of generative AI,
I mean, we all see it.
153
00:08:14,260 --> 00:08:16,229
We're exposed to it on social media.
154
00:08:16,229 --> 00:08:17,964
We see it come across our feed.
155
00:08:17,964 --> 00:08:21,901
It's so impressive
what the models can do these days.
156
00:08:21,901 --> 00:08:24,704
They can generate very realistic scenes.
157
00:08:24,704 --> 00:08:29,542
They can augment scenes and take
a summer image and turn it into winter.
158
00:08:29,876 --> 00:08:34,046
They can really transform
how you retrain these models, because
159
00:08:34,447 --> 00:08:38,684
we may not have all those different angles
and views captured from the drone.
160
00:08:38,684 --> 00:08:42,021
Maybe we have one view of the power line,
but now we want another view.
161
00:08:42,054 --> 00:08:44,757
Maybe we have one season
or one lighting condition.
162
00:08:44,757 --> 00:08:48,628
So I think they're extremely powerful
and I think people should consider
163
00:08:48,628 --> 00:08:50,062
using them.
164
00:08:50,062 --> 00:08:52,498
I think one caution is
165
00:08:52,498 --> 00:08:55,735
it really goes back to this through line
is how do you trust that
166
00:08:55,735 --> 00:08:59,805
your models learn something
that will generalize in production?
167
00:09:00,339 --> 00:09:03,976
And in that situation,
if you're using synthetic data,
168
00:09:03,976 --> 00:09:07,947
what we found is, is it should be
restricted to your training data set.
169
00:09:08,981 --> 00:09:09,916
And the reason is,
170
00:09:09,916 --> 00:09:13,819
is there might be subtle patterns
that we can't perceive.
171
00:09:13,819 --> 00:09:18,324
But this, you know, the data generator
has put in these fake images
172
00:09:18,791 --> 00:09:21,794
and they're kind of like a watermark
or a fingerprint
173
00:09:21,861 --> 00:09:25,665
that these other classifier
detector models will pick up on.
174
00:09:26,032 --> 00:09:29,535
I mean, these models, even though
their black box, even though they found
175
00:09:29,869 --> 00:09:33,506
they seem really complex, what they're
really doing is looking for patterns.
176
00:09:33,873 --> 00:09:38,110
And so if the generator puts
in some pattern, that's really noise,
177
00:09:38,110 --> 00:09:41,314
but it's kind of a fingerprint
that, hey, I'm a synthetic image,
178
00:09:41,781 --> 00:09:45,685
the other classifier is going to pick up
on that potentially
179
00:09:45,952 --> 00:09:48,955
and make it very easy to hack the problem.
180
00:09:48,988 --> 00:09:52,792
So if you use that data in your validation
181
00:09:52,792 --> 00:09:56,629
set, you might fool yourself saying,
wow, we added a bunch of synthetic images.
182
00:09:56,629 --> 00:09:59,098
Our model performance is going way,
way up.
183
00:09:59,098 --> 00:10:00,466
We're looking really good.
184
00:10:00,466 --> 00:10:03,502
So instead what you need to do
is restrict that only to training.
185
00:10:03,836 --> 00:10:08,441
And in that situation, it has to,
you know, learn generalizable patterns
186
00:10:08,441 --> 00:10:12,778
because those those synthetic watermarks
don't show up in your validation data set.
187
00:10:13,112 --> 00:10:15,848
And so I think it's very powerful,
this synthetic
188
00:10:15,848 --> 00:10:18,517
data, these generative models,
I think you should be using them.
189
00:10:18,517 --> 00:10:21,754
But you have to be very cautious
not to let them leak into what
190
00:10:21,754 --> 00:10:26,425
you're using to evaluate the models
and making that final deployment decision.
191
00:10:26,492 --> 00:10:29,629
What about, you know, for
some of our customers out there, let's say
192
00:10:29,629 --> 00:10:33,633
they're in the grid modernization team
or the asset management team,
193
00:10:34,133 --> 00:10:39,071
and they're trying to modernize
their inspection processes
194
00:10:39,071 --> 00:10:42,074
with AI enabled software
like Bus Solutions.
195
00:10:42,241 --> 00:10:45,444
What bits of advice
would you give these folks at the utility?
196
00:10:47,713 --> 00:10:48,314
Yeah, I think.
197
00:10:48,314 --> 00:10:52,685
One of the biggest lever points
that I see for this type of technology
198
00:10:52,685 --> 00:10:56,122
is just the forced multiplication
aspect of it.
199
00:10:56,322 --> 00:10:58,758
In this day and age,
we have great capability
200
00:10:58,758 --> 00:11:01,761
to capture high quality
imagery of our assets.
201
00:11:01,861 --> 00:11:04,864
We're almost overwhelmed
with that amount of data.
202
00:11:05,097 --> 00:11:09,101
These types of tools, they're really not,
in my opinion, at a point
203
00:11:09,101 --> 00:11:13,172
where they're going to replace
subject matter experts that are looking at
204
00:11:13,639 --> 00:11:18,110
these images, have deep knowledge
about the failure modes, the asset types.
205
00:11:18,477 --> 00:11:21,447
But we have to acknowledge that these
206
00:11:21,447 --> 00:11:24,417
the subject matter
experts, are few and far between,
207
00:11:24,550 --> 00:11:27,253
and they simply cannot
look at the hundreds,
208
00:11:27,253 --> 00:11:30,056
if not millions of images
that we're capturing.
209
00:11:30,056 --> 00:11:34,427
And so the best place to use these
is when you marry them with subject matter
210
00:11:34,427 --> 00:11:39,065
experts, you deploy these models
to kind of sift through all of that data,
211
00:11:39,331 --> 00:11:43,302
highlight kind of a high recall situation
where they kind, they scoop
212
00:11:43,302 --> 00:11:46,806
as many potential anomalies
as potential defects.
213
00:11:47,206 --> 00:11:51,210
These are the types of things that are
high value for then the human to review.
214
00:11:51,544 --> 00:11:56,582
And now the human can apply their subject
matter expertise to the highest value
215
00:11:56,582 --> 00:12:01,320
images confirm what the model's seeing,
and then all the other
216
00:12:01,420 --> 00:12:06,125
nominal images can be safely ignored, and
we can get the best of both worlds there.
217
00:12:06,125 --> 00:12:09,128
So that would be
my advice is look for those areas where
218
00:12:09,261 --> 00:12:12,198
the subject matter
experts are overwhelmed by data.
219
00:12:12,198 --> 00:12:13,199
There's a lot of data.
220
00:12:13,199 --> 00:12:17,036
Apply the models there
to reduce the amount of imagery that your
221
00:12:17,036 --> 00:12:19,972
your people are looking at.
You get a lot of value that way.
222
00:12:21,207 --> 00:12:21,607
Okay.
223
00:12:21,607 --> 00:12:22,875
Very good. That's interesting.
224
00:12:22,875 --> 00:12:25,077
So AI doesn't replace the team.
225
00:12:25,077 --> 00:12:28,047
AI is is now on the team.
226
00:12:28,447 --> 00:12:28,781
Yeah.
227
00:12:28,781 --> 00:12:30,249
Or just elevates the team.
228
00:12:30,249 --> 00:12:34,053
Really the team gets to do their job
and they get to apply
229
00:12:34,086 --> 00:12:37,890
their skills to the highest value items
in that data set.
230
00:12:38,390 --> 00:12:42,628
And it really makes their job count
even more, because you've sifted through
231
00:12:42,628 --> 00:12:45,798
all the other images that they shouldn't
be wasting their time on.
232
00:12:45,898 --> 00:12:48,267
Now, what about,
you know, a lot of the bigger utilities?
233
00:12:48,267 --> 00:12:50,603
They have their own analytics team, right?
234
00:12:50,603 --> 00:12:52,805
And, they want to build their own models.
235
00:12:52,805 --> 00:12:56,876
What advice do you have to
those teams about make versus by
236
00:12:57,076 --> 00:13:00,246
when should they build and when should
they use something existing?
237
00:13:00,412 --> 00:13:03,916
I think that there's a lot of smart people
working in utilities.
238
00:13:03,949 --> 00:13:05,317
The I've, I've seen
239
00:13:05,317 --> 00:13:07,453
and I've talked
to a lot of the analytics teams
240
00:13:07,453 --> 00:13:09,522
and there's a lot of great people out
there.
241
00:13:09,522 --> 00:13:12,625
And I have no doubt that
any one of those teams
242
00:13:12,625 --> 00:13:15,995
can build models
that are high quality, high caliber.
243
00:13:16,462 --> 00:13:20,599
I think the challenge that we see
is then taking that model
244
00:13:21,000 --> 00:13:25,137
and moving it to production
and applying it day in and day out,
245
00:13:26,405 --> 00:13:27,506
making sure that it's
246
00:13:27,506 --> 00:13:30,509
accessible
across the organization and standardized.
247
00:13:30,676 --> 00:13:34,547
And then as data drifts,
retraining the model, updating the model,
248
00:13:34,680 --> 00:13:37,116
keeping the model available.
249
00:13:37,116 --> 00:13:41,187
Those are the challenges that we find
internal teams sometimes struggle with,
250
00:13:41,187 --> 00:13:45,491
because there's a whole new world
of software
251
00:13:45,758 --> 00:13:51,597
and world of MLOps that goes along with
just deploying and serving your models.
252
00:13:51,597 --> 00:13:55,434
So I think that sometimes
that's the biggest case of,
253
00:13:55,501 --> 00:13:56,836
you know, should we do this in-house
254
00:13:56,836 --> 00:14:01,040
or should we have an out of the box
solution that kind of handles that for us?
255
00:14:01,640 --> 00:14:05,477
Nonetheless, I still think it's important
for internal teams to continue
256
00:14:05,477 --> 00:14:08,614
to experiment
and benchmark with their models,
257
00:14:08,614 --> 00:14:11,350
because if you have something
that's better than what you can get out
258
00:14:11,350 --> 00:14:14,353
there, that's really valuable,
you have to hang on to that.
259
00:14:14,420 --> 00:14:18,357
Even if you're leveraging another solution
to help deploy or bring
260
00:14:18,357 --> 00:14:22,528
that solution to scale or just translate,
hey, we have a really good model.
261
00:14:22,828 --> 00:14:24,597
We know it works with these features.
262
00:14:24,597 --> 00:14:27,333
Can you help us replicate and scale that?
263
00:14:27,333 --> 00:14:30,069
That would be the type of conversation
that I would be looking to have.
264
00:14:31,770 --> 00:14:32,238
Very good.
265
00:14:32,238 --> 00:14:34,807
So so really there's
266
00:14:34,807 --> 00:14:37,076
like a whole lifecycle to a model.
267
00:14:37,076 --> 00:14:40,646
It sounds like that,
they should be thinking about.
268
00:14:41,680 --> 00:14:42,514
Yeah, absolutely.
269
00:14:42,514 --> 00:14:46,819
I think a lot of people kind of think,
okay, well, you train the model, it's good
270
00:14:47,319 --> 00:14:48,120
we're done.
271
00:14:48,120 --> 00:14:52,791
And I think that training the model
at some point might be the easy part.
272
00:14:52,791 --> 00:14:54,827
And it's everything that comes after that.
273
00:14:54,827 --> 00:14:58,597
It's putting the model into production,
serving it day in and day out.
274
00:14:59,064 --> 00:15:02,301
There's a whole host of activities,
a lot of software,
275
00:15:02,301 --> 00:15:05,304
a lot of thought
has to go into those pipelines.
276
00:15:05,771 --> 00:15:07,907
So the model is a key piece.
277
00:15:07,907 --> 00:15:10,910
It's a centerpiece here,
but it's not the only piece.
278
00:15:11,710 --> 00:15:12,478
Yeah.
279
00:15:12,478 --> 00:15:14,613
So I mean, it's funny.
280
00:15:14,613 --> 00:15:17,149
I mean, looking back,
you made a really good decision in,
281
00:15:18,517 --> 00:15:19,151
about 15
282
00:15:19,151 --> 00:15:22,388
years ago to go into this
area of AI and ML.
283
00:15:22,388 --> 00:15:25,391
What's your favorite thing
about working in this space?
284
00:15:25,991 --> 00:15:27,893
I really like that.
285
00:15:27,893 --> 00:15:33,232
It kind of combines this idea of science
and engineering with subject matter
286
00:15:33,232 --> 00:15:36,602
experts across a bunch of different fields
that I never really expected.
287
00:15:36,602 --> 00:15:41,373
If you go into mechanical engineering
is where I got my undergrad degree.
288
00:15:41,373 --> 00:15:44,376
You know, there's interesting problems
to solve, but right now,
289
00:15:44,643 --> 00:15:48,814
I'm usually not the subject matter experts
in the problems that I'm helping solve,
290
00:15:49,148 --> 00:15:53,352
whether it was in national security or,
you know, here on the utility side,
291
00:15:53,585 --> 00:15:57,156
I get to work with a lot of people
that know their craft really well
292
00:15:57,656 --> 00:16:00,893
and build models that scale that craft
293
00:16:01,360 --> 00:16:05,698
to to thousands,
millions of impact points,
294
00:16:05,698 --> 00:16:09,668
which that brings a lot of,
you know, value to them.
295
00:16:10,135 --> 00:16:13,372
And it makes me very satisfied
that, wow, we can do this
296
00:16:13,605 --> 00:16:14,974
with the technology we have today.
297
00:16:14,974 --> 00:16:18,077
So that's what I really enjoy about,
you know, the field of AI.
298
00:16:19,745 --> 00:16:21,380
Great, great great stuff.
299
00:16:21,380 --> 00:16:23,816
Nick, I think we're out of time.
300
00:16:23,816 --> 00:16:27,386
But I really appreciate you joining us
today and sharing all your insights.
301
00:16:27,720 --> 00:16:29,054
Fantastic stuff.
302
00:16:29,054 --> 00:16:32,057
And thanks to the audience
for joining Buzz and Beyond.
303
00:16:32,992 --> 00:16:33,892
Yeah. Thanks, Stephen.
00:00:00,000 --> 00:00:03,003
and spent a decade in quantitative
finance.
2
00:00:03,003 --> 00:00:06,506
Really building machine
learning and deep learning models
3
00:00:06,506 --> 00:00:09,042
to trade financial futures stocks.
4
00:00:09,042 --> 00:00:12,912
And really kind of the same space
where we're anomaly detecting here
5
00:00:13,046 --> 00:00:14,080
at Buzz Solutions.
6
00:00:14,080 --> 00:00:18,718
That's a subset of this kind of huge,
broad umbrella of what can you use AI for?
7
00:00:19,019 --> 00:00:21,221
So AI doesn't replace the team.
8
00:00:21,221 --> 00:00:24,190
AI is is now on the team.
9
00:00:24,591 --> 00:00:24,924
Yeah.
10
00:00:24,924 --> 00:00:26,393
Or just elevates the team.
11
00:00:26,393 --> 00:00:29,129
Really the team gets to do their job
and they get to apply
12
00:00:36,903 --> 00:00:37,737
Welcome to the
13
00:00:37,737 --> 00:00:40,707
maiden Voyage of the and beyond podcast.
14
00:00:40,974 --> 00:00:43,977
I'm Stephen Ward, the marketing manager
for Bus Solutions.
15
00:00:44,344 --> 00:00:47,347
And today's guest is Nick libertine.
16
00:00:47,480 --> 00:00:50,717
He's the machine
learning team leader for buzz.
17
00:00:51,117 --> 00:00:53,820
And our topic is going to be leading
transformation.
18
00:00:53,820 --> 00:00:56,589
Thanks for joining us today, Nick.
19
00:00:56,589 --> 00:00:56,790
Yeah.
20
00:00:56,790 --> 00:00:59,292
Thanks, David, and excited to be here.
21
00:00:59,292 --> 00:01:01,728
So you've been involved in,
22
00:01:01,728 --> 00:01:04,831
in AI and machine learning,
since the beginning.
23
00:01:04,831 --> 00:01:06,499
So you started out you started out.
24
00:01:06,499 --> 00:01:09,502
You've got a degree in mechanical
engineering
25
00:01:09,502 --> 00:01:12,939
and economics
and then a masters in finance.
26
00:01:12,939 --> 00:01:15,942
So you're kind of a dual threat
technical and business.
27
00:01:17,510 --> 00:01:17,844
Yeah.
28
00:01:17,844 --> 00:01:19,245
Like you said, I've been doing,
29
00:01:19,245 --> 00:01:23,316
you know, machine learning and AI stuff
for the better part of 17 years now.
30
00:01:23,316 --> 00:01:27,587
And yeah, I did get a degree in finance
after I got my engineering degree.
31
00:01:27,620 --> 00:01:31,825
I was super passionate
and excited about the financial markets
32
00:01:31,825 --> 00:01:34,828
and spent a decade in quantitative
finance.
33
00:01:34,828 --> 00:01:38,331
Really building machine
learning and deep learning models
34
00:01:38,331 --> 00:01:41,334
to trade financial futures stocks.
35
00:01:41,434 --> 00:01:43,837
Again, the technology was a lot
newer then.
36
00:01:43,837 --> 00:01:47,407
We weren't using as big models
as complex models as we were today,
37
00:01:47,407 --> 00:01:50,176
but definitely
been doing this for a while now.
38
00:01:50,176 --> 00:01:50,877
Yeah. Okay.
39
00:01:50,877 --> 00:01:53,580
So you started with time series data
40
00:01:53,580 --> 00:01:57,817
and looking for results
that would help make money pretty much.
41
00:01:58,284 --> 00:02:00,353
Right? Yeah.
42
00:02:00,353 --> 00:02:04,624
Most of what we were doing was analyzing
the recent history of stocks
43
00:02:04,624 --> 00:02:09,028
and financial futures, looking back over
the last few weeks and months
44
00:02:09,028 --> 00:02:13,066
and trying to identify
or teach the model to find patterns
45
00:02:13,066 --> 00:02:16,503
in the data, that would be reproducible,
trades that we would take.
46
00:02:16,503 --> 00:02:19,973
And so a lot of what we were doing
was if the stocks go down
47
00:02:19,973 --> 00:02:23,343
over a few days, what's the probability
they're going to be up the next day?
48
00:02:23,343 --> 00:02:26,613
Or if this basket of stocks
deviates from this basket?
49
00:02:26,880 --> 00:02:29,516
Is there a convergence trade
or something like that.
50
00:02:29,516 --> 00:02:33,086
So we were building models
to exploit those
51
00:02:33,453 --> 00:02:35,922
and putting in real dollars
at risk really.
52
00:02:35,922 --> 00:02:38,758
And then in 2021,
53
00:02:38,758 --> 00:02:41,728
it looks like you moved to the, at MIT.
54
00:02:41,895 --> 00:02:43,730
Is it the Lincoln Laboratory?
55
00:02:45,632 --> 00:02:46,799
Yeah, I did
56
00:02:46,799 --> 00:02:50,036
a decade in quantitative finance
and had a lot of fun.
57
00:02:50,370 --> 00:02:53,439
But at that point in time,
I saw an opportunity
58
00:02:53,439 --> 00:02:56,976
to kind of pivot into the DoD
or intelligence space.
59
00:02:57,410 --> 00:03:02,148
Lincoln Lab is, a partnership
between the Department of Defense and MIT,
60
00:03:02,282 --> 00:03:05,685
and they do a lot of research
for defense initiatives.
61
00:03:05,685 --> 00:03:08,888
And so I got an opportunity
to go work on a team there.
62
00:03:08,888 --> 00:03:12,058
We were building
complex computer vision algorithms
63
00:03:12,358 --> 00:03:15,361
to try and solve
some national security problems.
64
00:03:15,395 --> 00:03:16,496
Okay.
65
00:03:16,496 --> 00:03:19,933
We we won't go into details on these,
of course, because they're confidential.
66
00:03:19,933 --> 00:03:22,101
But, I did hear computer vision.
67
00:03:22,101 --> 00:03:25,271
So this is when you started
applying, machine learning
68
00:03:25,271 --> 00:03:28,274
and AI to imagery?
69
00:03:28,775 --> 00:03:30,977
Yeah, it was a bit of a career shift,
70
00:03:30,977 --> 00:03:34,747
you know, switching from financial markets
into this space.
71
00:03:34,747 --> 00:03:36,716
But honestly,
there was a lot of carryover.
72
00:03:36,716 --> 00:03:41,254
I mean, a lot of the foundations
of building models for financial markets,
73
00:03:41,721 --> 00:03:44,757
wanting to make sure
that what you're building on
74
00:03:44,757 --> 00:03:47,894
historical data will work in production,
carried over here as well.
75
00:03:47,894 --> 00:03:50,496
You know, we're building models,
and we want to make sure that
76
00:03:50,496 --> 00:03:51,497
what we were building in
77
00:03:51,497 --> 00:03:55,568
the lab would carry out into the field
and work for the warfighter.
78
00:03:55,768 --> 00:03:59,572
Okay, so after the Lincoln lab,
then you joined Buzz Solutions,
79
00:03:59,572 --> 00:04:03,376
and now you're working on, imagery
for critical infrastructure.
80
00:04:03,376 --> 00:04:04,811
Yeah, yeah.
81
00:04:04,811 --> 00:04:08,448
It was kind of a natural transition
there as far as leading a team
82
00:04:08,448 --> 00:04:11,184
at Lincoln Lab
to now leading this fantastic team here
83
00:04:11,184 --> 00:04:14,187
at both Solutions,
we're solving very challenging problems.
84
00:04:14,187 --> 00:04:18,091
Like we were at Lincoln Lab
using a lot of the same tools
85
00:04:18,091 --> 00:04:21,094
as far as imagery captured from drones
and that type of stuff.
86
00:04:21,327 --> 00:04:25,198
And really kind of the same space
where we're anomaly detecting here
87
00:04:25,331 --> 00:04:26,366
upon solution.
88
00:04:26,366 --> 00:04:31,004
That's a subset of this kind of huge,
broad umbrella of what can you use AI for?
89
00:04:31,204 --> 00:04:34,707
One of the things you can use
is training models to find anomalies
90
00:04:35,041 --> 00:04:36,409
in these image data sets.
91
00:04:36,409 --> 00:04:37,877
And you really want to make sure
92
00:04:37,877 --> 00:04:40,947
that when you're training those
and they're working on your kind of
93
00:04:41,281 --> 00:04:44,817
training data set, that they'll also
be applicable and find stuff in the field
94
00:04:44,817 --> 00:04:48,388
when you actually run your models out
and try and put them into practice.
95
00:04:48,688 --> 00:04:53,926
So across your career, you across finance
and the national security
96
00:04:53,926 --> 00:04:56,296
and critical infrastructure,
what are some of the key lessons
97
00:04:56,296 --> 00:04:59,666
you've learned about applying machine
learning to these data sets?
98
00:05:01,267 --> 00:05:03,670
Yeah,
I think one of the biggest through lines
99
00:05:03,670 --> 00:05:07,407
that's kind of carried through
my entire career is you really,
100
00:05:07,640 --> 00:05:11,878
when you're training a model,
whether it's to trade stocks or,
101
00:05:12,612 --> 00:05:16,082
to predict anomalies,
you really have to have trust that
102
00:05:16,449 --> 00:05:19,852
when you're done training it,
that it's learned something useful
103
00:05:20,053 --> 00:05:23,423
that will be applicable to data
it's never seen during training.
104
00:05:23,456 --> 00:05:27,527
It's very easy to fool yourself
when you're building, you know, kind of
105
00:05:27,527 --> 00:05:32,899
black box artificial intelligence models
that, hey, this model has 90% accurate.
106
00:05:33,032 --> 00:05:35,668
It's performing
really good on my training set.
107
00:05:35,668 --> 00:05:39,572
But that's not the data
that your model is going to be used on.
108
00:05:39,572 --> 00:05:42,642
It's actually going to be used on data
it's likely never seen before.
109
00:05:42,942 --> 00:05:46,212
And so you have to build up
a set of robust evaluation
110
00:05:46,212 --> 00:05:49,215
criteria
to say, look, we've trained this model.
111
00:05:49,282 --> 00:05:52,919
It's really good at detecting this type
of anomaly on power line equipment,
112
00:05:53,353 --> 00:05:56,889
and it actually works on new imagery
that we've never seen before.
113
00:05:56,889 --> 00:05:57,657
And so building
114
00:05:57,657 --> 00:06:01,494
that strong validation gives
you confidence that the model will work
115
00:06:02,028 --> 00:06:06,299
in the field or in production when you're
going to apply it to new imagery.
116
00:06:07,333 --> 00:06:08,201
Okay.
117
00:06:08,201 --> 00:06:10,570
Now now was critical infrastructure.
118
00:06:10,570 --> 00:06:12,572
It seems to me
there's a bit of a difference,
119
00:06:12,572 --> 00:06:16,476
because what you're trying to do with
critical infrastructure is not necessarily
120
00:06:16,476 --> 00:06:22,081
be, 100% precise, but you're trying
not to miss something important, right.
121
00:06:22,081 --> 00:06:25,084
So how does that affect the model
development?
122
00:06:25,885 --> 00:06:28,020
Yeah, yeah, it's a great point.
123
00:06:28,020 --> 00:06:31,858
I think that there's a lot of different
areas that you can optimize for a model.
124
00:06:32,158 --> 00:06:34,961
One of those things that you can do,
like you mentioned,
125
00:06:34,961 --> 00:06:38,398
is optimize the model's ability
to find anomalies.
126
00:06:38,398 --> 00:06:41,734
In that case, you're trying
to find those needles in the haystack,
127
00:06:42,268 --> 00:06:45,171
and it's more important
that you acknowledge that, hey, maybe
128
00:06:45,171 --> 00:06:49,642
there's only 5 or 6 of these anomalies out
in my data of thousands of images.
129
00:06:50,042 --> 00:06:51,177
I want to make sure I catch
130
00:06:51,177 --> 00:06:54,981
as many of those,
even if I have some false detections.
131
00:06:54,981 --> 00:06:57,817
Hey, this looks like what I'm looking for,
but it's not quite.
132
00:06:57,817 --> 00:07:02,789
And so there you're optimizing for recall,
and you really have to coach the models
133
00:07:03,089 --> 00:07:06,659
to try and think
and look for those anomalies
134
00:07:06,659 --> 00:07:10,696
in, in all the different ways
they can manifest among the huge pile
135
00:07:10,696 --> 00:07:14,867
of nominal data that you're training,
your data or your models on.
136
00:07:15,368 --> 00:07:19,138
And that's completely usually inverse
of what you're doing in, say, finance
137
00:07:19,138 --> 00:07:23,776
or quantitative training,
where a lot of the focus is on precision.
138
00:07:24,243 --> 00:07:28,648
You have an opportunity
to trade any minute, any day in the hour.
139
00:07:28,915 --> 00:07:31,918
So there's a lot of opportunity
to put capital at risk.
140
00:07:32,185 --> 00:07:36,055
But you want to make sure that when you do
deploy capital and your model predicts,
141
00:07:36,055 --> 00:07:40,026
hey, hey, take this trade, that it's right
more often than not, but
142
00:07:40,026 --> 00:07:44,263
it has high precision that you're putting
your capital at risk in good situations.
143
00:07:44,263 --> 00:07:46,466
So they're kind of inverse of each other.
144
00:07:46,466 --> 00:07:51,003
And the way you coach the models
to be receptive to either high precision
145
00:07:51,237 --> 00:07:55,775
situations or high recall situations
is really baked into how the machine
146
00:07:55,775 --> 00:07:59,612
learning team or team trains their models,
and how they validate their models
147
00:07:59,612 --> 00:08:00,112
as well.
148
00:08:01,047 --> 00:08:01,814
Yeah, very
149
00:08:01,814 --> 00:08:05,351
interesting because a lot of buzz about
using synthetic data to train models.
150
00:08:05,351 --> 00:08:08,354
What are your thoughts
on the use of synthetic data?
151
00:08:09,188 --> 00:08:11,791
Yeah, I think, you know, in this
152
00:08:11,791 --> 00:08:14,260
era of generative AI,
I mean, we all see it.
153
00:08:14,260 --> 00:08:16,229
We're exposed to it on social media.
154
00:08:16,229 --> 00:08:17,964
We see it come across our feed.
155
00:08:17,964 --> 00:08:21,901
It's so impressive
what the models can do these days.
156
00:08:21,901 --> 00:08:24,704
They can generate very realistic scenes.
157
00:08:24,704 --> 00:08:29,542
They can augment scenes and take
a summer image and turn it into winter.
158
00:08:29,876 --> 00:08:34,046
They can really transform
how you retrain these models, because
159
00:08:34,447 --> 00:08:38,684
we may not have all those different angles
and views captured from the drone.
160
00:08:38,684 --> 00:08:42,021
Maybe we have one view of the power line,
but now we want another view.
161
00:08:42,054 --> 00:08:44,757
Maybe we have one season
or one lighting condition.
162
00:08:44,757 --> 00:08:48,628
So I think they're extremely powerful
and I think people should consider
163
00:08:48,628 --> 00:08:50,062
using them.
164
00:08:50,062 --> 00:08:52,498
I think one caution is
165
00:08:52,498 --> 00:08:55,735
it really goes back to this through line
is how do you trust that
166
00:08:55,735 --> 00:08:59,805
your models learn something
that will generalize in production?
167
00:09:00,339 --> 00:09:03,976
And in that situation,
if you're using synthetic data,
168
00:09:03,976 --> 00:09:07,947
what we found is, is it should be
restricted to your training data set.
169
00:09:08,981 --> 00:09:09,916
And the reason is,
170
00:09:09,916 --> 00:09:13,819
is there might be subtle patterns
that we can't perceive.
171
00:09:13,819 --> 00:09:18,324
But this, you know, the data generator
has put in these fake images
172
00:09:18,791 --> 00:09:21,794
and they're kind of like a watermark
or a fingerprint
173
00:09:21,861 --> 00:09:25,665
that these other classifier
detector models will pick up on.
174
00:09:26,032 --> 00:09:29,535
I mean, these models, even though
their black box, even though they found
175
00:09:29,869 --> 00:09:33,506
they seem really complex, what they're
really doing is looking for patterns.
176
00:09:33,873 --> 00:09:38,110
And so if the generator puts
in some pattern, that's really noise,
177
00:09:38,110 --> 00:09:41,314
but it's kind of a fingerprint
that, hey, I'm a synthetic image,
178
00:09:41,781 --> 00:09:45,685
the other classifier is going to pick up
on that potentially
179
00:09:45,952 --> 00:09:48,955
and make it very easy to hack the problem.
180
00:09:48,988 --> 00:09:52,792
So if you use that data in your validation
181
00:09:52,792 --> 00:09:56,629
set, you might fool yourself saying,
wow, we added a bunch of synthetic images.
182
00:09:56,629 --> 00:09:59,098
Our model performance is going way,
way up.
183
00:09:59,098 --> 00:10:00,466
We're looking really good.
184
00:10:00,466 --> 00:10:03,502
So instead what you need to do
is restrict that only to training.
185
00:10:03,836 --> 00:10:08,441
And in that situation, it has to,
you know, learn generalizable patterns
186
00:10:08,441 --> 00:10:12,778
because those those synthetic watermarks
don't show up in your validation data set.
187
00:10:13,112 --> 00:10:15,848
And so I think it's very powerful,
this synthetic
188
00:10:15,848 --> 00:10:18,517
data, these generative models,
I think you should be using them.
189
00:10:18,517 --> 00:10:21,754
But you have to be very cautious
not to let them leak into what
190
00:10:21,754 --> 00:10:26,425
you're using to evaluate the models
and making that final deployment decision.
191
00:10:26,492 --> 00:10:29,629
What about, you know, for
some of our customers out there, let's say
192
00:10:29,629 --> 00:10:33,633
they're in the grid modernization team
or the asset management team,
193
00:10:34,133 --> 00:10:39,071
and they're trying to modernize
their inspection processes
194
00:10:39,071 --> 00:10:42,074
with AI enabled software
like Bus Solutions.
195
00:10:42,241 --> 00:10:45,444
What bits of advice
would you give these folks at the utility?
196
00:10:47,713 --> 00:10:48,314
Yeah, I think.
197
00:10:48,314 --> 00:10:52,685
One of the biggest lever points
that I see for this type of technology
198
00:10:52,685 --> 00:10:56,122
is just the forced multiplication
aspect of it.
199
00:10:56,322 --> 00:10:58,758
In this day and age,
we have great capability
200
00:10:58,758 --> 00:11:01,761
to capture high quality
imagery of our assets.
201
00:11:01,861 --> 00:11:04,864
We're almost overwhelmed
with that amount of data.
202
00:11:05,097 --> 00:11:09,101
These types of tools, they're really not,
in my opinion, at a point
203
00:11:09,101 --> 00:11:13,172
where they're going to replace
subject matter experts that are looking at
204
00:11:13,639 --> 00:11:18,110
these images, have deep knowledge
about the failure modes, the asset types.
205
00:11:18,477 --> 00:11:21,447
But we have to acknowledge that these
206
00:11:21,447 --> 00:11:24,417
the subject matter
experts, are few and far between,
207
00:11:24,550 --> 00:11:27,253
and they simply cannot
look at the hundreds,
208
00:11:27,253 --> 00:11:30,056
if not millions of images
that we're capturing.
209
00:11:30,056 --> 00:11:34,427
And so the best place to use these
is when you marry them with subject matter
210
00:11:34,427 --> 00:11:39,065
experts, you deploy these models
to kind of sift through all of that data,
211
00:11:39,331 --> 00:11:43,302
highlight kind of a high recall situation
where they kind, they scoop
212
00:11:43,302 --> 00:11:46,806
as many potential anomalies
as potential defects.
213
00:11:47,206 --> 00:11:51,210
These are the types of things that are
high value for then the human to review.
214
00:11:51,544 --> 00:11:56,582
And now the human can apply their subject
matter expertise to the highest value
215
00:11:56,582 --> 00:12:01,320
images confirm what the model's seeing,
and then all the other
216
00:12:01,420 --> 00:12:06,125
nominal images can be safely ignored, and
we can get the best of both worlds there.
217
00:12:06,125 --> 00:12:09,128
So that would be
my advice is look for those areas where
218
00:12:09,261 --> 00:12:12,198
the subject matter
experts are overwhelmed by data.
219
00:12:12,198 --> 00:12:13,199
There's a lot of data.
220
00:12:13,199 --> 00:12:17,036
Apply the models there
to reduce the amount of imagery that your
221
00:12:17,036 --> 00:12:19,972
your people are looking at.
You get a lot of value that way.
222
00:12:21,207 --> 00:12:21,607
Okay.
223
00:12:21,607 --> 00:12:22,875
Very good. That's interesting.
224
00:12:22,875 --> 00:12:25,077
So AI doesn't replace the team.
225
00:12:25,077 --> 00:12:28,047
AI is is now on the team.
226
00:12:28,447 --> 00:12:28,781
Yeah.
227
00:12:28,781 --> 00:12:30,249
Or just elevates the team.
228
00:12:30,249 --> 00:12:34,053
Really the team gets to do their job
and they get to apply
229
00:12:34,086 --> 00:12:37,890
their skills to the highest value items
in that data set.
230
00:12:38,390 --> 00:12:42,628
And it really makes their job count
even more, because you've sifted through
231
00:12:42,628 --> 00:12:45,798
all the other images that they shouldn't
be wasting their time on.
232
00:12:45,898 --> 00:12:48,267
Now, what about,
you know, a lot of the bigger utilities?
233
00:12:48,267 --> 00:12:50,603
They have their own analytics team, right?
234
00:12:50,603 --> 00:12:52,805
And, they want to build their own models.
235
00:12:52,805 --> 00:12:56,876
What advice do you have to
those teams about make versus by
236
00:12:57,076 --> 00:13:00,246
when should they build and when should
they use something existing?
237
00:13:00,412 --> 00:13:03,916
I think that there's a lot of smart people
working in utilities.
238
00:13:03,949 --> 00:13:05,317
The I've, I've seen
239
00:13:05,317 --> 00:13:07,453
and I've talked
to a lot of the analytics teams
240
00:13:07,453 --> 00:13:09,522
and there's a lot of great people out
there.
241
00:13:09,522 --> 00:13:12,625
And I have no doubt that
any one of those teams
242
00:13:12,625 --> 00:13:15,995
can build models
that are high quality, high caliber.
243
00:13:16,462 --> 00:13:20,599
I think the challenge that we see
is then taking that model
244
00:13:21,000 --> 00:13:25,137
and moving it to production
and applying it day in and day out,
245
00:13:26,405 --> 00:13:27,506
making sure that it's
246
00:13:27,506 --> 00:13:30,509
accessible
across the organization and standardized.
247
00:13:30,676 --> 00:13:34,547
And then as data drifts,
retraining the model, updating the model,
248
00:13:34,680 --> 00:13:37,116
keeping the model available.
249
00:13:37,116 --> 00:13:41,187
Those are the challenges that we find
internal teams sometimes struggle with,
250
00:13:41,187 --> 00:13:45,491
because there's a whole new world
of software
251
00:13:45,758 --> 00:13:51,597
and world of MLOps that goes along with
just deploying and serving your models.
252
00:13:51,597 --> 00:13:55,434
So I think that sometimes
that's the biggest case of,
253
00:13:55,501 --> 00:13:56,836
you know, should we do this in-house
254
00:13:56,836 --> 00:14:01,040
or should we have an out of the box
solution that kind of handles that for us?
255
00:14:01,640 --> 00:14:05,477
Nonetheless, I still think it's important
for internal teams to continue
256
00:14:05,477 --> 00:14:08,614
to experiment
and benchmark with their models,
257
00:14:08,614 --> 00:14:11,350
because if you have something
that's better than what you can get out
258
00:14:11,350 --> 00:14:14,353
there, that's really valuable,
you have to hang on to that.
259
00:14:14,420 --> 00:14:18,357
Even if you're leveraging another solution
to help deploy or bring
260
00:14:18,357 --> 00:14:22,528
that solution to scale or just translate,
hey, we have a really good model.
261
00:14:22,828 --> 00:14:24,597
We know it works with these features.
262
00:14:24,597 --> 00:14:27,333
Can you help us replicate and scale that?
263
00:14:27,333 --> 00:14:30,069
That would be the type of conversation
that I would be looking to have.
264
00:14:31,770 --> 00:14:32,238
Very good.
265
00:14:32,238 --> 00:14:34,807
So so really there's
266
00:14:34,807 --> 00:14:37,076
like a whole lifecycle to a model.
267
00:14:37,076 --> 00:14:40,646
It sounds like that,
they should be thinking about.
268
00:14:41,680 --> 00:14:42,514
Yeah, absolutely.
269
00:14:42,514 --> 00:14:46,819
I think a lot of people kind of think,
okay, well, you train the model, it's good
270
00:14:47,319 --> 00:14:48,120
we're done.
271
00:14:48,120 --> 00:14:52,791
And I think that training the model
at some point might be the easy part.
272
00:14:52,791 --> 00:14:54,827
And it's everything that comes after that.
273
00:14:54,827 --> 00:14:58,597
It's putting the model into production,
serving it day in and day out.
274
00:14:59,064 --> 00:15:02,301
There's a whole host of activities,
a lot of software,
275
00:15:02,301 --> 00:15:05,304
a lot of thought
has to go into those pipelines.
276
00:15:05,771 --> 00:15:07,907
So the model is a key piece.
277
00:15:07,907 --> 00:15:10,910
It's a centerpiece here,
but it's not the only piece.
278
00:15:11,710 --> 00:15:12,478
Yeah.
279
00:15:12,478 --> 00:15:14,613
So I mean, it's funny.
280
00:15:14,613 --> 00:15:17,149
I mean, looking back,
you made a really good decision in,
281
00:15:18,517 --> 00:15:19,151
about 15
282
00:15:19,151 --> 00:15:22,388
years ago to go into this
area of AI and ML.
283
00:15:22,388 --> 00:15:25,391
What's your favorite thing
about working in this space?
284
00:15:25,991 --> 00:15:27,893
I really like that.
285
00:15:27,893 --> 00:15:33,232
It kind of combines this idea of science
and engineering with subject matter
286
00:15:33,232 --> 00:15:36,602
experts across a bunch of different fields
that I never really expected.
287
00:15:36,602 --> 00:15:41,373
If you go into mechanical engineering
is where I got my undergrad degree.
288
00:15:41,373 --> 00:15:44,376
You know, there's interesting problems
to solve, but right now,
289
00:15:44,643 --> 00:15:48,814
I'm usually not the subject matter experts
in the problems that I'm helping solve,
290
00:15:49,148 --> 00:15:53,352
whether it was in national security or,
you know, here on the utility side,
291
00:15:53,585 --> 00:15:57,156
I get to work with a lot of people
that know their craft really well
292
00:15:57,656 --> 00:16:00,893
and build models that scale that craft
293
00:16:01,360 --> 00:16:05,698
to to thousands,
millions of impact points,
294
00:16:05,698 --> 00:16:09,668
which that brings a lot of,
you know, value to them.
295
00:16:10,135 --> 00:16:13,372
And it makes me very satisfied
that, wow, we can do this
296
00:16:13,605 --> 00:16:14,974
with the technology we have today.
297
00:16:14,974 --> 00:16:18,077
So that's what I really enjoy about,
you know, the field of AI.
298
00:16:19,745 --> 00:16:21,380
Great, great great stuff.
299
00:16:21,380 --> 00:16:23,816
Nick, I think we're out of time.
300
00:16:23,816 --> 00:16:27,386
But I really appreciate you joining us
today and sharing all your insights.
301
00:16:27,720 --> 00:16:29,054
Fantastic stuff.
302
00:16:29,054 --> 00:16:32,057
And thanks to the audience
for joining Buzz and Beyond.
303
00:16:32,992 --> 00:16:33,892
Yeah. Thanks, Stephen.
