Taming Voice Complexity with Dynamic Ensembles at Modulate

WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 02/08/2026 21:02:53
Duration: 3565.304
Channels: 1
1
00:00:05.120 --> 00:00:08.000
Hello, and welcome to the AI Engineering Podcast,
2
00:00:08.000 --> 00:00:12.160
your guide to the fast moving world of building scalable and maintainable AI systems.
3
00:00:19.685 --> 00:00:24.485
Unlock the full potential of your AI workloads with a seamless and composable data infrastructure.
4
00:00:25.060 --> 00:00:32.739
Bruin is an open source framework that streamlines integration from the command line, allowing you to focus on what matters most, building intelligent systems.
5
00:00:33.300 --> 00:00:41.255
Write Python code for your business logic and let Bruin handle the heavy lifting of data movement, lineage tracking, data quality monitoring, and governance enforcement.
6
00:00:41.735 --> 00:00:48.935
With native support for ML and AI workloads, Bruin empowers data teams to deliver faster, more reliable, and scalable AI solutions.
7
00:00:49.495 --> 00:00:55.255
Harness Bruin's connectors for hundreds of platforms, including popular machine learning frameworks like TensorFlow and PyTorch.
8
00:00:55.910 --> 00:01:00.230
Build end to end AI workflows that integrate seamlessly with your existing tech stack.
9
00:01:00.550 --> 00:01:05.510
Join the ranks of forward thinking organizations that are revolutionizing their data engineering with Bruin.
10
00:01:05.750 --> 00:01:09.575
Get started today at aiengineeringpodcast.com/bruin.
11
00:01:09.575 --> 00:01:14.135
And for dbt cloud customers, enjoy a $1,000 credit to migrate to Bruin Cloud.
12
00:01:14.455 --> 00:01:27.340
Your host is Tobias Maci, and today I'm interviewing Carter Huffman about his work in building an ensemble approach to low latency voice AI. So, Carter, can you start by introducing yourself? Hey. I'm Carter Hoffman. I'm the CTO and cofounder over at Modulate.
13
00:01:27.659 --> 00:01:29.340
We do proactive
14
00:01:29.340 --> 00:01:33.820
voice analysis understanding in Frontier Voice AI. Super nice to meet you.
15
00:01:34.140 --> 00:01:36.939
And do you remember how you first got started in the ML and AI space?
16
00:01:37.465 --> 00:01:42.985
Absolutely. Yeah. It was actually over at the Jet Propulsion Laboratory in Pasadena, California,
17
00:01:43.465 --> 00:01:47.305
which is one of the NASA centers focused on the robotic exploration of space.
18
00:01:47.625 --> 00:01:52.825
And I've been really into space, really into robots my whole life. And so when a recruiter came to JPL
19
00:01:53.170 --> 00:02:11.565
to my school, and I, you know, kind of talked to him, and he was like, hey, you know, my job is landing robots on Mars. You want to come interview? I was like, oh, yeah, sure. You know, I don't do a ton of, like, space robotics. But yeah, absolutely. I'll come apply. And I got hired into the the machine learning and instrument autonomy group. So I helped make the spacecraft smarter. That's where I started.
20
00:02:11.965 --> 00:02:22.110
And so in the context of what you're working on now, I know that you recently had an announcement about some of the restructuring and refactoring
21
00:02:22.110 --> 00:02:25.310
of at least some component of your VoiceAI
22
00:02:25.870 --> 00:02:26.910
capabilities.
23
00:02:26.910 --> 00:02:36.805
And before we get too much into the details of that, I'm wondering if you could just unpack a little bit about what you mean by that term voice AI and how it is maybe differentiated
24
00:02:36.805 --> 00:02:41.045
from a lot of the other AI applications that folks are likely using today.
25
00:02:41.685 --> 00:02:45.285
Sure thing. Absolutely. So by voice AI, I really mean
26
00:02:45.630 --> 00:02:48.430
machine learning and artificial intelligence models
27
00:02:48.510 --> 00:02:50.190
that are built towards
28
00:02:50.270 --> 00:02:51.310
understanding,
29
00:02:51.310 --> 00:02:52.270
manipulating,
30
00:02:52.270 --> 00:02:53.230
and producing
31
00:02:53.550 --> 00:02:55.310
human voice signals.
32
00:02:55.630 --> 00:02:58.990
And the reason why voice AI is kind of its own category
33
00:02:59.435 --> 00:03:03.195
is because voice is a really interesting form of media.
34
00:03:03.355 --> 00:03:17.730
You have text. You have images. You have video. You have voice. And if you've been following the trajectory of AI just in general over the past twenty years, you notice that people build models that tackle text, images, and video first.
35
00:03:18.130 --> 00:03:20.050
Voice kinda trails behind,
36
00:03:20.370 --> 00:03:24.850
and that's because voice is really, really technically difficult. There's a lot of data there,
37
00:03:26.085 --> 00:03:28.965
but it all encodes
38
00:03:28.965 --> 00:03:37.765
a ton of nuanced information that's very hard for us as human beings to verbalize and really recognize. Like, you can tell my tone is off
39
00:03:38.180 --> 00:03:45.300
just because you're, like, evolutionarily built to notice that stuff. And when you try to reproduce that or manipulate that with a machine,
40
00:03:45.460 --> 00:03:48.020
it's very, very difficult to get it perfect.
41
00:03:48.420 --> 00:03:52.020
And if you do voice AI slightly less than perfect,
42
00:03:52.515 --> 00:03:56.675
it becomes super noticeable and super, like, disconcerting.
43
00:03:56.754 --> 00:04:08.034
Contrast with, like, generating an image of a face, you see a face in anything. So it's very easy to make a face even if there's a ton of cool research going on. So that's kind of why voice AI is its own separate thing.
44
00:04:09.050 --> 00:04:09.770
And
45
00:04:10.250 --> 00:04:12.570
another aspect of that is that,
46
00:04:13.050 --> 00:04:15.210
actually, maybe one of the earlier
47
00:04:15.370 --> 00:04:22.650
models that was introduced in this current era of generative models and transformer architectures
48
00:04:22.085 --> 00:04:27.125
was the whisper model from open AI, which is very focused on voice
49
00:04:27.125 --> 00:04:28.005
with the
50
00:04:28.325 --> 00:04:45.820
caveat that it's only trying to convert that to a textual representation to increase the amount of corpora available for training of these language models. And I'm wondering if you can add a bit more nuance to some of the detail of what you're talking about with those elements of inflection and tone
51
00:04:45.900 --> 00:04:51.580
and how that's differentiated from purely transcription based or being able to even do
52
00:04:52.045 --> 00:05:00.045
speech to speech where I just wanna be able to take that voice signal, convert it to text, feed it to an LLM, generate some output, and then convert it back to voice.
53
00:05:00.685 --> 00:05:01.565
Absolutely.
54
00:05:01.565 --> 00:05:02.445
Yeah. So
55
00:05:02.765 --> 00:05:08.680
doing that flow speech to text to speech works reasonably well when
56
00:05:08.760 --> 00:05:09.960
the emotional
57
00:05:09.960 --> 00:05:14.280
content sentiment or kind of long term context of a conversation
58
00:05:14.520 --> 00:05:16.360
is relatively unimportant.
59
00:05:16.360 --> 00:05:23.175
So you can get kind of reasonable responses and reasonable back and forth on short time scales in that situation.
60
00:05:23.414 --> 00:05:28.615
But voice contains a ton more signal than just the, like, textual content
61
00:05:28.935 --> 00:05:43.510
of of what you're saying. How you say it is super, super important. So, like, for example, you know, I I say, like, hey. I'm super excited to be here. I'm thrilled. Or I can say, like, yeah. I'm super excited to be here. I'm thrilled.
62
00:05:43.750 --> 00:05:55.105
The exact same words. And if you transcribe it and feed it to an LLM, it will do exactly the same thing with those two signals. But you and I know that you would react to that statement extremely
63
00:05:55.105 --> 00:05:55.985
differently.
64
00:05:56.465 --> 00:06:02.865
And that's just one of the aspects. And there's a ton of, like, social relationship and context packed into
65
00:06:03.720 --> 00:06:09.320
why am I expressing that emotion in that way. So that was what makes it difficult,
66
00:06:09.560 --> 00:06:18.680
and that's kind of what makes it very important to be able to capture those other signals because you have to react almost in opposite ways depending on the tone and emotion of the voice.
67
00:06:19.775 --> 00:06:21.695
And now digging into
68
00:06:21.855 --> 00:06:25.775
some of the announcement of your ensemble listening approach,
69
00:06:25.935 --> 00:06:27.695
what are some of the
70
00:06:28.575 --> 00:06:36.090
challenges or constraints that are specific to voice AI that led you to putting in the effort to actually
71
00:06:36.890 --> 00:06:49.955
researching and developing this approach to how to actually manage some of that input processing and some of the general expectations and requirements around how it's supposed to operate that led you down that path?
72
00:06:50.595 --> 00:06:51.395
Absolutely.
73
00:06:51.395 --> 00:06:53.955
So I would say there's two main constraints
74
00:06:53.955 --> 00:06:58.275
that factored into our decision to pursue a totally new architecture.
75
00:06:58.595 --> 00:07:03.030
Constraint number one is cost and processing power required.
76
00:07:03.509 --> 00:07:05.030
Just transcribing,
77
00:07:05.110 --> 00:07:06.790
analyzing a waveform,
78
00:07:07.110 --> 00:07:09.750
even doing simple relatively simple operations
79
00:07:09.750 --> 00:07:19.065
like detecting accent or detecting whether there's music in the background or what kind of background noise there is, those kinds of operations on an audio stream,
80
00:07:19.865 --> 00:07:21.225
you can get
81
00:07:21.385 --> 00:07:28.110
a little bit of the way there with relatively simple and cheap models. But to do a very good job,
82
00:07:28.430 --> 00:07:33.950
you have to deploy much more powerful and expensive models. Like, there are transcription
83
00:07:33.950 --> 00:07:40.465
services out there that cost, you know, anywhere from 10¢ to a dollar to process an hour of audio.
84
00:07:40.705 --> 00:07:42.705
If you're a large social platform,
85
00:07:43.025 --> 00:07:44.865
you could have 50,000,000,
86
00:07:44.865 --> 00:07:46.065
a 100,000,000,
87
00:07:46.305 --> 00:07:51.185
three billion hours of voice content on your platform each month.
88
00:07:51.665 --> 00:07:52.385
And
89
00:07:52.810 --> 00:07:54.810
you're not really gonna pay $3,000,000,000
90
00:07:54.810 --> 00:08:03.530
a month to analyze that audio if you wanna do something useful with it. So on the one hand, you have to be really, really cost effective and efficient
91
00:08:03.610 --> 00:08:08.090
at processing those audio signals if you're gonna do anything useful at scale.
92
00:08:08.705 --> 00:08:16.065
So that's problem statement number one. Problem statement number two is that audio and especially voice conversations
93
00:08:16.065 --> 00:08:18.545
in particular are very kind of like a
94
00:08:18.865 --> 00:08:21.345
yeah. I I call it like a lumpy multimodal
95
00:08:21.345 --> 00:08:22.385
distribution.
96
00:08:22.625 --> 00:08:23.665
Right? Like, you have
97
00:08:24.210 --> 00:08:25.810
different pockets
98
00:08:26.130 --> 00:08:26.770
of
99
00:08:27.010 --> 00:08:28.370
kinds of audio,
100
00:08:28.610 --> 00:08:32.050
but within those pockets, there's a ton of shared similarities.
101
00:08:32.290 --> 00:08:32.770
So
102
00:08:33.330 --> 00:08:35.810
we're doing, like, a podcast conversation.
103
00:08:35.810 --> 00:08:40.714
Right? We're in relatively high quality audio environment, relatively low background noise.
104
00:08:40.795 --> 00:08:42.154
We're enunciating
105
00:08:42.315 --> 00:08:49.195
relatively clearly or at least you are. I hope I am. And we have kind of these properties that are shared
106
00:08:49.435 --> 00:08:50.235
among
107
00:08:50.635 --> 00:08:53.274
other very similar audio environments.
108
00:08:53.620 --> 00:09:01.860
And then contrast that with calling into a customer support center, and I'm on my phone. It's over telephony, so it's, like, eight kilohertz audio,
109
00:09:01.940 --> 00:09:13.495
and I'm on a bus, and there are people talking in the background. And there are a ton of also calls like that, but that's a very different distribution. You take a model like Whisper and you apply it to that distribution,
110
00:09:13.735 --> 00:09:16.215
it's gonna produce absolute junk.
111
00:09:16.375 --> 00:09:17.015
Right?
112
00:09:17.335 --> 00:09:20.055
So you have these different pockets
113
00:09:20.310 --> 00:09:29.590
where you have a lot of voice conversations that are very similar to each other, but the pockets are very, very different and require very different modeling approaches.
114
00:09:29.750 --> 00:09:31.110
And if you wanna build
115
00:09:31.510 --> 00:09:37.555
a big model that can handle all of those different pockets, all of the different modes of that distribution,
116
00:09:37.954 --> 00:09:39.714
it has to be extremely
117
00:09:39.714 --> 00:09:41.235
flexible and powerful
118
00:09:41.315 --> 00:09:43.955
to handle those different data distributions
119
00:09:44.035 --> 00:09:48.620
equally well. If you build a model that's extremely powerful,
120
00:09:48.699 --> 00:09:51.980
it's gonna be super cost effective, so you violate the first constraint.
121
00:09:52.060 --> 00:09:54.060
So it's a really, really tricky situation,
122
00:09:54.060 --> 00:09:55.660
which is why it required new tech.
123
00:09:56.139 --> 00:09:58.220
And so digging into
124
00:09:58.300 --> 00:10:01.339
the specifics of this Ensemble architecture,
125
00:10:01.420 --> 00:10:03.394
what are some of the
126
00:10:04.035 --> 00:10:10.435
high level details about how it actually addresses some of those complexities around the
127
00:10:10.675 --> 00:10:17.555
high cost as well as high latency issues that come up when you're dealing with these large multipurpose
128
00:10:17.555 --> 00:10:18.274
models
129
00:10:18.590 --> 00:10:29.150
and then also tries to address some of the accuracy issues that you run into when you go to a smaller or more quantized model to address some of those latency considerations?
130
00:10:30.030 --> 00:10:30.990
Absolutely.
131
00:10:30.990 --> 00:10:34.350
Great question. So there are really two main properties
132
00:10:34.565 --> 00:10:38.565
that led us to the Ensemble model architecture.
133
00:10:38.885 --> 00:10:43.125
Property number one is that when we're talking about those multimodal
134
00:10:43.125 --> 00:10:44.245
distributions,
135
00:10:44.725 --> 00:10:50.940
because the similarities are so close to each other within a single kind of mode of that distribution,
136
00:10:51.980 --> 00:10:54.779
all of the data is relatively
137
00:10:54.779 --> 00:10:55.580
similar,
138
00:10:55.980 --> 00:11:01.180
and so you don't need a ton of capacity. A small model could get you very far.
139
00:11:01.675 --> 00:11:05.035
Small specialized models can perform with high accuracy
140
00:11:05.115 --> 00:11:06.955
in certain circumstances,
141
00:11:07.435 --> 00:11:10.155
but you lose a lot in terms of generalizability
142
00:11:10.155 --> 00:11:13.435
if you are focused on a small model.
143
00:11:14.339 --> 00:11:15.060
However,
144
00:11:16.019 --> 00:11:17.620
the other insight
145
00:11:17.620 --> 00:11:20.740
is that these distributions
146
00:11:21.060 --> 00:11:24.660
continue on over the course of the conversation.
147
00:11:24.899 --> 00:11:29.140
So you and I are having this high audio quality back and forth conversation.
148
00:11:29.705 --> 00:11:36.105
We're very unlikely to switch to something like a noisy bus, low audio quality environment
149
00:11:36.345 --> 00:11:40.265
halfway through the conversation. Or even more importantly, we're unlikely to be swapping
150
00:11:40.505 --> 00:11:45.950
back and forth all the time. And you might have different distributions for different participants,
151
00:11:46.110 --> 00:11:48.350
but they remain relatively
152
00:11:48.350 --> 00:11:51.230
static over the course of a conversation.
153
00:11:51.709 --> 00:11:54.750
So the way to bring these two things together
154
00:11:54.990 --> 00:11:57.149
is that if you can isolate
155
00:11:57.149 --> 00:11:58.670
quickly and efficiently
156
00:11:59.245 --> 00:12:00.925
which distribution
157
00:12:01.165 --> 00:12:06.045
you're in or which distribution each participant of the conversation is in,
158
00:12:06.525 --> 00:12:09.005
then if you also have
159
00:12:09.325 --> 00:12:15.630
a small model or a set of small models that are tuned specifically for that distribution,
160
00:12:16.269 --> 00:12:20.190
you can pick the right model to apply to that distribution,
161
00:12:20.670 --> 00:12:25.230
trust it will mostly do a good job over the whole course of the conversation,
162
00:12:25.550 --> 00:12:29.470
and then course correct if something does change as it as it occasionally might.
163
00:12:30.185 --> 00:12:33.225
So this is sort of like how a really big model
164
00:12:33.945 --> 00:12:39.945
will find paths through the weights in the large network that are appropriate for different distributions.
165
00:12:39.945 --> 00:12:46.290
But instead of having to run all the weights in the network or at least a large subset of the weights in the network,
166
00:12:46.690 --> 00:12:47.570
you are
167
00:12:47.890 --> 00:13:00.545
pre isolating down to a very, very small subset of the overall Ensemble model that you know ahead of time or know based on the first analysis of the data in the conversation
168
00:13:00.625 --> 00:13:02.865
is going to be applicable to
169
00:13:03.025 --> 00:13:06.385
likely the whole rest of the conversation distribution.
170
00:13:07.345 --> 00:13:08.465
Architecturally,
171
00:13:08.465 --> 00:13:09.185
having
172
00:13:09.425 --> 00:13:12.529
more models increases the complexity
173
00:13:12.529 --> 00:13:21.890
and potential for failure as well as the variety of failure modes as is the case with any complex distributed system. And so that definitely requires
174
00:13:21.890 --> 00:13:22.610
having
175
00:13:22.930 --> 00:13:25.649
a high degree of confidence
176
00:13:25.649 --> 00:13:26.930
in the
177
00:13:27.215 --> 00:13:28.575
orchestration,
178
00:13:28.655 --> 00:13:30.255
the failure recovery,
179
00:13:30.495 --> 00:13:44.769
and I imagine also some measure of fallback capability for the situation where the ensemble just catastrophically fails. And I'm just wondering if you can talk to some of the ways that you thought through and proved out this overall
180
00:13:45.009 --> 00:13:47.410
engineering effort and what your
181
00:13:47.889 --> 00:13:55.170
calculus was as far as which pieces you were able to pull off the shelf and what are the pieces that you had to engineer from whole cloth.
182
00:13:56.765 --> 00:13:57.645
Great question.
183
00:13:58.125 --> 00:13:58.685
So
184
00:14:00.125 --> 00:14:04.685
one of the things that surprised us most when we started deploying
185
00:14:04.845 --> 00:14:07.005
these Ensemble models
186
00:14:07.565 --> 00:14:08.925
was really that
187
00:14:09.580 --> 00:14:11.100
as they were
188
00:14:11.260 --> 00:14:12.700
running inference,
189
00:14:13.899 --> 00:14:17.580
we found it very hard to predict actually
190
00:14:17.899 --> 00:14:20.140
which models were important
191
00:14:20.140 --> 00:14:21.100
to the results
192
00:14:21.455 --> 00:14:33.135
and which models were relatively unimportant to the results and when a model was doing a good job or when a model was doing a bad job. So the actually, the initial
193
00:14:34.360 --> 00:14:36.600
idea, the initial architecture,
194
00:14:36.600 --> 00:14:38.279
sort of the v one
195
00:14:38.519 --> 00:14:41.240
of these ELMs in our particular
196
00:14:41.560 --> 00:14:43.560
model Velma, sort of v one,
197
00:14:43.879 --> 00:14:45.480
is much more of a static
198
00:14:46.345 --> 00:14:47.385
ensemble
199
00:14:47.465 --> 00:14:49.625
that is preselected
200
00:14:50.105 --> 00:14:52.505
at conversation initialization.
201
00:14:52.505 --> 00:14:56.665
So you would have, like, okay. We have a big ensemble of a bunch of different models.
202
00:14:56.905 --> 00:14:58.025
As you mentioned,
203
00:14:58.265 --> 00:15:07.090
a ton of models leads to a complex system, so you have to select down to a very sparse set of these models to actually deploy at runtime
204
00:15:07.170 --> 00:15:09.570
other for a given conversation.
205
00:15:10.290 --> 00:15:10.770
Otherwise,
206
00:15:11.595 --> 00:15:23.035
complexity goes up, which means the cost and difficulty of orchestration goes up, which also means that the cost in general of running the system goes up. And we talked about cost being one of the big constraints earlier on.
207
00:15:23.435 --> 00:15:37.250
So the initial idea was to statically select, like, okay. We know for a given language and a given, like, social chat environment or interview call environment or other things like that,
208
00:15:37.490 --> 00:15:38.930
we can kind of test
209
00:15:39.195 --> 00:15:45.515
based on some ground truth labeled data sets which subsets of models work best on these environments
210
00:15:45.755 --> 00:15:51.275
and then just kind of do a lookup table and and statically look up which models to run and run those for the whole conversation.
211
00:15:51.920 --> 00:15:53.440
And what we found
212
00:15:53.680 --> 00:15:55.519
was that when that fails,
213
00:15:55.680 --> 00:15:57.920
it fails very, very, very poorly
214
00:15:58.000 --> 00:15:59.760
because these models are small.
215
00:16:00.000 --> 00:16:10.685
And, individually, they don't generalize well. The whole ensemble generalizes well, but, individually, the models don't generalize well. If you pick the wrong small model, you're going to get very bad results.
216
00:16:11.165 --> 00:16:15.805
So this led us to doing a ton of research into how do you cost effectively
217
00:16:16.204 --> 00:16:17.005
monitor
218
00:16:17.805 --> 00:16:19.805
the results of the models,
219
00:16:20.259 --> 00:16:26.180
How do you spot check whether or not the models are performing well? And how do you course correct
220
00:16:26.420 --> 00:16:36.795
and select different subsets of models if you know these aren't performing well? And this is the idea of the dynamic ensemble block, which is one of the cool pieces
221
00:16:37.035 --> 00:16:39.675
of the ELM architecture that we invented,
222
00:16:40.155 --> 00:16:43.275
where we realized that this is actually an optimization problem.
223
00:16:43.515 --> 00:16:46.555
Each model has a cost to run-in
224
00:16:46.555 --> 00:16:53.510
a given ensemble, like our ensemble of transcription models or an ensemble of emotion detection models,
225
00:16:54.070 --> 00:16:56.310
each has an expected
226
00:16:56.310 --> 00:16:57.269
accuracy
227
00:16:57.670 --> 00:16:58.709
given
228
00:16:58.870 --> 00:17:02.550
the distribution of audio that you're currently looking at.
229
00:17:03.110 --> 00:17:03.750
And
230
00:17:04.525 --> 00:17:07.804
they each have a degree of confidence
231
00:17:07.804 --> 00:17:10.524
that you can have about their outputs.
232
00:17:11.165 --> 00:17:15.404
And the optimization problem is that as you're running data through,
233
00:17:15.885 --> 00:17:21.360
you're getting results from the models that you selected in the Ensemble.
234
00:17:21.680 --> 00:17:26.559
You compare those results to each other to the rest of the context of the conversation
235
00:17:26.720 --> 00:17:29.280
to figure out how much you trust those results.
236
00:17:29.914 --> 00:17:35.994
And then the optimization problem is around the next part of the conversation, say, the next minute that you're analyzing.
237
00:17:36.235 --> 00:17:37.674
For that next minute,
238
00:17:37.914 --> 00:17:41.674
how do you optimize for accuracy while minimizing cost?
239
00:17:41.914 --> 00:17:44.840
And that transforms that whole problem
240
00:17:45.000 --> 00:17:55.000
down into a very well known, like, multi armed bandit exploration versus exploitation problem so you can deploy a ton of well known machinery to solving that optimally.
241
00:17:55.355 --> 00:18:00.155
That was the really, really cool thing once we found out this surprise that
242
00:18:00.635 --> 00:18:03.034
when the models go off the rails, they go really bad.
243
00:18:03.515 --> 00:18:05.595
Yeah. As you're describing
244
00:18:05.595 --> 00:18:06.955
the architecture,
245
00:18:06.955 --> 00:18:09.995
it brings to mind two, I guess, complementary
246
00:18:11.059 --> 00:18:32.614
aspects of prior art where you're talking about the optimization question. It brings to mind a lot of the cost based optimizers that go into SQL query planners, particularly for these large scale out distributed database engines or things like the Trino engine that allows you to be able to do predicate pushdowns based on what is the actual underlying storage.
247
00:18:33.335 --> 00:18:37.654
And then in terms of being able to determine which particular
248
00:18:38.560 --> 00:18:53.440
model or module you want to actually route to, it brings up a lot of the conversation that's happening right now about how to actually implement these multi agent systems and how to manage communication and routing between those different agents to be able to achieve the desired outcome.
249
00:18:55.125 --> 00:18:59.765
Exactly. And I think that's what makes it so exciting and so
250
00:19:00.885 --> 00:19:02.565
I don't wanna say possible,
251
00:19:02.565 --> 00:19:04.565
you know, but at least tractable
252
00:19:04.805 --> 00:19:05.205
to
253
00:19:06.320 --> 00:19:07.120
develop
254
00:19:07.120 --> 00:19:09.760
this Ensemble listening model architecture
255
00:19:09.760 --> 00:19:14.000
as a, you know, what, 45 person start up, is that
256
00:19:14.960 --> 00:19:16.480
as you described,
257
00:19:16.640 --> 00:19:19.120
cost optimization problems are everywhere.
258
00:19:19.725 --> 00:19:27.325
And we're really, really like, as a species, we're really good at solving cost optimization problems. So as soon as we found out, oh,
259
00:19:27.725 --> 00:19:31.565
this really tricky part of analyzing conversations
260
00:19:31.565 --> 00:19:33.165
cost effectively and accurately
261
00:19:33.540 --> 00:19:37.220
can be transformed into a cost optimization problem.
262
00:19:37.460 --> 00:19:57.464
That was the eureka where we were like, okay. We have the tools to actually deal with this. And you can deal with that in terms of, like, you know, what is your overall cost budget, but how sensitive are you to that? You can add in things like how many GPUs or clusters of GPUs we have available. Because one of the exciting things about cost optimization around
263
00:19:57.784 --> 00:19:59.304
our small models
264
00:19:59.465 --> 00:20:08.809
based architecture here is that you don't need the biggest and best GPUs. You can run on we're running on, you know, five, six year old GPUs sometimes because they're the most cost effective.
265
00:20:09.050 --> 00:20:12.650
So you can make the optimization problem even more complicated,
266
00:20:12.890 --> 00:20:18.234
but we have the optimizers to do that. And we've had them for decades, and they're really, really, really good.
267
00:20:19.355 --> 00:20:24.075
The the routing question, I think, is a really, really fun research problem.
268
00:20:24.315 --> 00:20:28.955
And the thing that makes multi agent systems writing so hard, in my opinion,
269
00:20:29.195 --> 00:20:32.390
is that agents are so flexible.
270
00:20:33.590 --> 00:20:37.990
Any agent could, in theory, do a wide variety of things,
271
00:20:38.470 --> 00:20:42.070
take in a wide variety of inputs, produce a wide variety of outputs.
272
00:20:42.544 --> 00:20:51.104
It's very flexible subcomponents of the system. So figuring out how to route to which one is optimal when many of them can do many of the tasks
273
00:20:51.265 --> 00:20:52.705
can be very tricky.
274
00:20:52.865 --> 00:20:54.385
This is an advantage to
275
00:20:54.940 --> 00:21:05.660
the ensemble of small models approach, at least when trying to solve the routing problem, is that we know exactly what all of these small models do. We know exactly what their cost is. They don't have a thinking budget.
276
00:21:06.300 --> 00:21:10.414
They don't really have a very flexible compute graph.
277
00:21:10.495 --> 00:21:12.654
Each of these individual models
278
00:21:12.815 --> 00:21:16.334
are relatively small and self contained and predictable.
279
00:21:16.335 --> 00:21:22.414
And we also, because they're small, have a very good understanding of their output distribution
280
00:21:22.850 --> 00:21:29.650
across different kinds of inputs that they might be given and what kind of accuracy ranges we can expect.
281
00:21:30.050 --> 00:21:33.010
So in terms of routing between the different models,
282
00:21:34.130 --> 00:21:37.330
making each of the individual nodes super predictable,
283
00:21:37.765 --> 00:21:42.004
as we have turns that routing problem into
284
00:21:42.405 --> 00:21:44.085
a pretty deterministic
285
00:21:44.085 --> 00:21:54.220
kind of thing as opposed to something with agents where it's very easy for the agents to, like, get themselves into a loop or something like that. Or we're a very much a, like, directed graph
286
00:21:54.300 --> 00:21:56.299
with some feedback loops,
287
00:21:56.380 --> 00:21:57.340
asynchronous,
288
00:21:57.340 --> 00:21:58.379
low priority
289
00:21:58.460 --> 00:22:04.780
to do things like optimize selecting which models. But that doesn't get in the way of the feed forward processing of the data.
290
00:22:05.965 --> 00:22:09.964
It also reminds me of at least my understanding
291
00:22:09.965 --> 00:22:12.364
of the GPT five
292
00:22:12.365 --> 00:22:13.245
category
293
00:22:13.245 --> 00:22:18.445
of model deployments where my understanding is that it's not actually one monolithic
294
00:22:18.445 --> 00:22:19.164
model.
295
00:22:19.404 --> 00:22:20.524
It's actually several
296
00:22:20.929 --> 00:22:26.370
sub models that are specialized for different cases, and they abstract that underneath
297
00:22:26.370 --> 00:22:41.475
a routing layer that determines which model to actually use under the covers. And I'm wondering what are some of the aspects of prior art that you were able to pull from either anything that they've published or any other similar product launches that have happened over the past several months.
298
00:22:42.275 --> 00:22:43.315
Absolutely.
299
00:22:43.955 --> 00:22:46.595
We've been working with ensembles
300
00:22:46.595 --> 00:22:48.115
of models for years.
301
00:22:48.610 --> 00:22:51.489
So the prior art that we pull from
302
00:22:51.809 --> 00:22:54.049
is mostly more from
303
00:22:54.370 --> 00:22:57.249
traditional mixture of experts models
304
00:22:57.570 --> 00:22:59.249
and traditional
305
00:22:59.570 --> 00:23:01.169
hybrid systems
306
00:23:01.250 --> 00:23:02.049
between
307
00:23:02.395 --> 00:23:04.235
rules based and
308
00:23:04.635 --> 00:23:07.514
neural network or black box kind of architectures
309
00:23:07.674 --> 00:23:09.674
and combining those things together.
310
00:23:09.915 --> 00:23:10.794
I also
311
00:23:11.275 --> 00:23:15.674
it was really exciting. Got to pull on some of my work from JPL
312
00:23:15.960 --> 00:23:16.759
where
313
00:23:17.640 --> 00:23:20.760
we were doing a lot of investigations
314
00:23:20.760 --> 00:23:25.320
into how can you deploy AI systems safely on spacecraft.
315
00:23:25.960 --> 00:23:26.600
And
316
00:23:26.919 --> 00:23:28.039
on a spacecraft,
317
00:23:29.205 --> 00:23:30.725
doing something unpredictable
318
00:23:30.725 --> 00:23:40.085
is often quite fatal. Right? If you accidentally turn your solar panels away from the sun, it's hard to recover from that if your battery goes dead.
319
00:23:40.565 --> 00:23:42.885
And if you accidentally turn off your radio,
320
00:23:43.549 --> 00:23:46.190
unless you have a fallback system, which they do,
321
00:23:47.230 --> 00:23:50.429
it's hard to tell the spacecraft to turn it back on again.
322
00:23:51.070 --> 00:23:55.070
So when you're deploying an AI system onto a spacecraft,
323
00:23:55.150 --> 00:23:57.870
the number one lesson is bounding
324
00:23:58.264 --> 00:23:59.464
the effects
325
00:23:59.465 --> 00:24:01.624
that that one model can have
326
00:24:01.865 --> 00:24:13.919
on the rest of your system. So no matter what the network says, like, hey. We need to go look at this rock next. It's the most important rock I've ever seen in my entire life. Ultimately,
327
00:24:13.919 --> 00:24:17.919
the outputs of that model can be perfect or they can be garbage,
328
00:24:18.159 --> 00:24:22.559
but they're all fed into more of a rules based, well understood,
329
00:24:23.280 --> 00:24:24.559
constrained
330
00:24:24.720 --> 00:24:25.520
optimization
331
00:24:25.520 --> 00:24:26.480
algorithm
332
00:24:26.785 --> 00:24:31.184
such that even if the AI encounters something completely unpredictable,
333
00:24:31.905 --> 00:24:34.145
you've already got the predetermined
334
00:24:34.145 --> 00:24:36.144
bounds to keep it from
335
00:24:36.545 --> 00:24:40.785
completely crashing everything else. So this actually related back to
336
00:24:41.539 --> 00:24:42.980
how we orchestrated
337
00:24:42.980 --> 00:24:48.739
some of our models because we're using those input distribution features
338
00:24:48.980 --> 00:24:52.099
and the results from some of our sub ensembles
339
00:24:52.100 --> 00:24:56.100
to figure out, are we doing a good job at understanding this conversation?
340
00:24:56.764 --> 00:25:03.244
But you might see the problem with that of we're using our models to figure out if our models are doing a good job.
341
00:25:03.725 --> 00:25:04.364
And
342
00:25:04.845 --> 00:25:07.084
if they're confused but confident,
343
00:25:07.164 --> 00:25:09.484
you can mess up the entire conversation.
344
00:25:09.485 --> 00:25:12.860
And some of the stuff we're doing is actually pretty,
345
00:25:13.260 --> 00:25:19.660
like, high risk, high high reward kind of stuff. You know? We're deployed in social game stopping harassment.
346
00:25:19.900 --> 00:25:29.315
You accidentally banned the victim instead of the aggressor in our harassment situation. That can cause a lot of real harm to a lot of real people. We look for extremism,
347
00:25:29.315 --> 00:25:46.190
child safety incidents, other things like that. It's like, it can be relatively high cost if you get the stuff wrong. Of course, you have other safeguards on it, but it's still an important problem to get right. So if your models are just confident in their own abilities and you don't have a check on them, you can run into serious problems.
348
00:25:46.670 --> 00:25:55.615
So that's why you were asking a little bit about off the shelf models earlier. There's actually a place for the large, highly general
349
00:25:56.415 --> 00:25:57.215
models
350
00:25:57.375 --> 00:26:05.455
in an ELM framework as kind of a supervisor or a check. So we actually have a suite of models in our different ensembles
351
00:26:05.720 --> 00:26:08.760
that are very high capability,
352
00:26:08.760 --> 00:26:10.600
high cost generalists.
353
00:26:10.920 --> 00:26:14.600
And we will occasionally and this is, again, an optimization problem.
354
00:26:15.080 --> 00:26:17.800
We will occasionally run them to check
355
00:26:18.465 --> 00:26:19.665
the outputs
356
00:26:19.665 --> 00:26:22.225
of our smaller, more specialized models
357
00:26:22.385 --> 00:26:28.065
and make sure that they agree to a reasonable degree. And you can't run it too often or it gets expensive,
358
00:26:28.145 --> 00:26:34.860
but you can't run that not at all. Otherwise, you fall into this confidence problem. So you have this kind of checking feedback loop,
359
00:26:35.100 --> 00:26:39.179
and that's how you constrain your overall system
360
00:26:39.340 --> 00:26:49.085
to do a pretty good job. But, again, a lot of that priority is coming from stuff that people were working on one, two decades ago. Some of these ideas, they're not super, super new.
361
00:26:50.045 --> 00:26:51.005
And then
362
00:26:51.245 --> 00:26:56.125
generalizing beyond the voice use case, what are some of the
363
00:26:56.980 --> 00:26:58.980
environments or situations
364
00:26:58.980 --> 00:27:03.379
that you would advocate for teams to explore,
365
00:27:03.779 --> 00:27:05.299
either building their own
366
00:27:05.700 --> 00:27:14.735
Ensemble approach or if there are any other off the shelf frameworks that allow for building this style of Ensemble model capability,
367
00:27:14.735 --> 00:27:15.455
again,
368
00:27:16.095 --> 00:27:17.534
not including the
369
00:27:17.775 --> 00:27:21.775
multi agent use case because of the inherent flexibility involved.
370
00:27:24.370 --> 00:27:25.330
Great question.
371
00:27:26.450 --> 00:27:27.169
The
372
00:27:27.570 --> 00:27:28.450
applicability
373
00:27:28.450 --> 00:27:30.050
of Ensemble model
374
00:27:30.530 --> 00:27:32.130
well, first, I would say
375
00:27:33.090 --> 00:27:35.169
anybody working in a
376
00:27:35.410 --> 00:27:36.370
regime
377
00:27:36.450 --> 00:27:37.970
where you are going to do
378
00:27:38.605 --> 00:27:39.884
a similar
379
00:27:40.044 --> 00:27:41.004
task
380
00:27:41.085 --> 00:27:42.924
many, many times over,
381
00:27:43.804 --> 00:27:45.804
I would recommend looking into
382
00:27:46.044 --> 00:27:48.044
small models in general
383
00:27:48.205 --> 00:27:49.485
and ensembles
384
00:27:49.485 --> 00:27:50.684
of small models
385
00:27:50.870 --> 00:27:54.470
in the cases where you need a lot of general flexibility,
386
00:27:55.110 --> 00:27:57.830
but all around a similar kind of task.
387
00:27:58.070 --> 00:28:00.470
So the reason why ELMs were so
388
00:28:01.270 --> 00:28:03.750
directly applicable to voice conversations
389
00:28:03.830 --> 00:28:05.430
is because voice conversations
390
00:28:05.985 --> 00:28:07.904
have a ton of shared properties.
391
00:28:08.145 --> 00:28:20.705
You're always wanna gonna wanna get a transcript. You're always gonna wanna figure out who are these people that are talking. You're always gonna wanna figure out what emotion are they using, what accent are they using. Give me some basic demographics so I can understand the context.
392
00:28:21.260 --> 00:28:24.300
Always gonna wanna understand the topics of discussion,
393
00:28:24.540 --> 00:28:31.420
the intents of the people involved, like what are these people trying to achieve when they're discussing in order to understand that conversation.
394
00:28:31.660 --> 00:28:38.504
And there are tens of billions of conversations that happen on this planet every single day, but they all fall into that structure.
395
00:28:38.825 --> 00:28:40.744
And if you contrast with something like
396
00:28:41.145 --> 00:28:42.904
just a large language model,
397
00:28:43.304 --> 00:28:51.959
there's none of that structure of how must I analyze this data built into it. Even though the space of conversations,
398
00:28:52.039 --> 00:28:54.599
all voice conversations is very, very general,
399
00:28:55.559 --> 00:28:58.599
it's still a much, much more constrained
400
00:28:59.240 --> 00:29:13.475
space of what kinds of things do you want to do to analyze this data than just say an arbitrary stream of text or an arbitrary stream of data, which is what the big foundation models are applicable to. So if you can identify that structure
401
00:29:13.635 --> 00:29:14.674
and say,
402
00:29:15.635 --> 00:29:22.710
I'm gonna be tackling a very diverse set of data, but they all need to be analyzed in similar ways,
403
00:29:23.669 --> 00:29:24.309
then
404
00:29:25.110 --> 00:29:30.789
deploying small models lets you take advantage of the cost efficiencies of those small models.
405
00:29:30.870 --> 00:29:37.404
You can break the problem down into several pieces, like let's analyze emotion. Let's analyze transcription.
406
00:29:37.485 --> 00:29:42.684
Let's do these other things. And if you still want to maintain the flexibility
407
00:29:43.005 --> 00:29:44.285
and generality
408
00:29:44.285 --> 00:29:47.245
of applying to a bunch of different distributions,
409
00:29:47.405 --> 00:29:48.845
like different kinds of conversations,
410
00:29:49.600 --> 00:29:52.399
then that's where you'll want ensembles of those models.
411
00:29:52.720 --> 00:30:00.479
So that's that's where I would suggest looking into using ensemble models instead of just going straight for a big foundation model.
412
00:30:00.960 --> 00:30:01.919
And then
413
00:30:02.295 --> 00:30:03.015
the
414
00:30:03.415 --> 00:30:13.735
large open question that I see a lot of people trying to deal with right now as a broader set of people are entering this ML and AI ecosystem
415
00:30:13.895 --> 00:30:21.549
is how do I actually verify that anything that I changed didn't just completely destroy everything that I've been working on for the past six months?
416
00:30:21.790 --> 00:30:27.390
And so one of the approaches that has gained a lot of popularity is the idea of evaluations.
417
00:30:28.110 --> 00:30:31.470
There are also the observability aspects
418
00:30:31.470 --> 00:30:32.270
of doing
419
00:30:32.794 --> 00:30:38.634
real time analysis of model performance and outputs to determine if that is actually
420
00:30:39.034 --> 00:30:40.634
producing the desired
421
00:30:41.115 --> 00:30:41.754
result.
422
00:30:42.235 --> 00:30:43.355
And as
423
00:30:43.355 --> 00:30:45.434
you add this Ensemble
424
00:30:45.434 --> 00:30:47.275
aspect to the overall
425
00:30:48.450 --> 00:30:49.889
request path,
426
00:30:50.049 --> 00:31:02.130
what are some of the ways that that complicates that question of how do I actually validate the overall functionality of the system as well as being able to do more isolated validation of those sub models?
427
00:31:03.395 --> 00:31:04.514
It gives you
428
00:31:04.755 --> 00:31:06.994
added complexity you need to handle,
429
00:31:07.395 --> 00:31:11.634
but it also gives you a lot of power and tools to solve that complexity.
430
00:31:12.115 --> 00:31:13.874
So the added complexity,
431
00:31:14.115 --> 00:31:24.660
as you might guess, is the extraordinary number of different routes through the system that your data can take. So if I have some small models that are specialized to
432
00:31:24.900 --> 00:31:26.340
high quality audio
433
00:31:26.660 --> 00:31:27.380
and
434
00:31:27.685 --> 00:31:32.565
my ensemble picks a model specialized to low quality audio instead,
435
00:31:33.045 --> 00:31:34.565
the rest of the system,
436
00:31:34.805 --> 00:31:37.684
all bets are off, anything downstream from that.
437
00:31:37.925 --> 00:31:42.005
And because you're doing this dynamic optimization that I was talking about earlier,
438
00:31:42.325 --> 00:31:44.480
where the results of models
439
00:31:44.480 --> 00:31:47.519
running earlier in the conversation feed back
440
00:31:48.080 --> 00:31:52.719
into the system to help it pick which models to run-in the future.
441
00:31:53.759 --> 00:31:55.199
If you've got garbage
442
00:31:55.200 --> 00:31:58.000
coming out of your models early in the conversation,
443
00:31:58.505 --> 00:32:04.184
you can poison that decision making in those choices for later on in the conversation.
444
00:32:04.985 --> 00:32:08.985
So it makes it really, really hard because of the sheer number of paths
445
00:32:08.985 --> 00:32:11.065
the data can take through your Ensemble.
446
00:32:11.750 --> 00:32:16.309
And the flip side is the Ensemble approach also gives you a lot of tools.
447
00:32:16.629 --> 00:32:17.269
So
448
00:32:17.750 --> 00:32:21.669
part one is the structure that I was talking about earlier.
449
00:32:21.830 --> 00:32:23.750
So we know that a conversation
450
00:32:24.705 --> 00:32:28.864
has these different pieces of structure that we're looking for.
451
00:32:29.105 --> 00:32:30.465
And all of the
452
00:32:30.705 --> 00:32:32.144
structured outputs,
453
00:32:32.145 --> 00:32:36.465
like for example, the text content and emotion
454
00:32:36.705 --> 00:32:40.470
content of what I'm saying are related.
455
00:32:40.870 --> 00:32:42.230
They're not identical.
456
00:32:42.310 --> 00:32:47.669
One's just not a strict function of the other. Otherwise, you wouldn't need the multiple models, but they're related.
457
00:32:48.070 --> 00:32:51.590
And so for observability and debugging purposes,
458
00:32:51.830 --> 00:32:54.764
one thing you can do, which we in fact do,
459
00:32:55.005 --> 00:32:57.885
is check sentiment extracted
460
00:32:57.885 --> 00:32:59.085
from text
461
00:32:59.245 --> 00:33:01.325
and check emotional tone
462
00:33:01.325 --> 00:33:07.485
and see if they're matching up relatively often. It's a statistical approach, not a deterministic
463
00:33:07.485 --> 00:33:08.605
debugging tactic,
464
00:33:09.000 --> 00:33:10.760
But you expect
465
00:33:11.400 --> 00:33:15.399
different outputs from different pieces of your ensemble
466
00:33:15.400 --> 00:33:17.480
to correlate in certain ways.
467
00:33:17.800 --> 00:33:21.800
And so if I identify that for certain kinds of conversations,
468
00:33:22.345 --> 00:33:24.825
like, let's say, low quality audio calls,
469
00:33:25.545 --> 00:33:30.985
no matter what text is being output, my emotion model always thinks people are angry,
470
00:33:31.545 --> 00:33:35.065
then I know that something's likely to be wrong.
471
00:33:35.550 --> 00:33:37.710
And in certain situations,
472
00:33:37.710 --> 00:33:38.350
again,
473
00:33:38.910 --> 00:33:43.390
the emotion and the text might not match. But if they're completely uncorrelated
474
00:33:43.390 --> 00:33:46.190
over a long a large amount of data,
475
00:33:46.510 --> 00:33:48.830
then that gives you a pointer for what to look into.
476
00:33:50.485 --> 00:33:56.004
And digging now into some of the specifics of these voice models,
477
00:33:56.165 --> 00:34:03.445
I have a generalized understanding of the ways that large language models work in terms of these transformer architectures,
478
00:34:03.679 --> 00:34:11.920
the attention mechanisms that are used, the variance there, the idea of these mixture of experts models for the case where you want to be able to incorporate more reasoning.
479
00:34:12.160 --> 00:34:24.795
And I'm just curious how much of that is analogous in this case of audio and particularly voice focused models, and how much of the actual underlying architecture needs to shift because of the modality
480
00:34:24.795 --> 00:34:27.115
and the specifics of speech?
481
00:34:28.075 --> 00:34:30.155
It's actually very complementary.
482
00:34:30.474 --> 00:34:31.115
So
483
00:34:31.515 --> 00:34:32.635
when you're tackling
484
00:34:33.430 --> 00:34:34.710
an audio
485
00:34:34.710 --> 00:34:35.510
analysis
486
00:34:35.510 --> 00:34:36.390
problem,
487
00:34:36.950 --> 00:34:38.230
you're looking at
488
00:34:38.790 --> 00:34:41.430
both how do I get the right
489
00:34:41.910 --> 00:34:42.710
results
490
00:34:42.710 --> 00:34:45.030
out of the underlying
491
00:34:45.030 --> 00:34:45.510
data?
492
00:34:45.965 --> 00:34:52.045
And then how do I interpret those to make broader sense of the conversation and do something useful with it?
493
00:34:52.685 --> 00:34:55.485
The techniques that you're talking about around attention
494
00:34:55.885 --> 00:34:59.165
and all sorts of other capabilities
495
00:34:59.165 --> 00:35:00.365
for these models,
496
00:35:01.390 --> 00:35:08.670
audio is a very long time scale kind of data, right? Like, you've got a 48 kilohertz audio signal.
497
00:35:09.070 --> 00:35:09.950
That's
498
00:35:09.950 --> 00:35:30.800
a ton of different data points for just a second where you might utter, like, you know, what, one or a couple of different phones make a couple of different sounds. So you're you're mapping this very, very, very high dimensional audio signal down to, even if you're doing emotion and all the nuance and everything, a much, much, much lower dimensionality
499
00:35:30.800 --> 00:35:31.600
space.
500
00:35:32.800 --> 00:35:34.880
Technologies like transformers
501
00:35:34.880 --> 00:35:44.175
and even older school stuff from the twenty tens, you know, earlier twenty tens, like dilated convolutions and wave nets and things like that, All of these different strategies
502
00:35:44.815 --> 00:35:46.415
and architectures
503
00:35:46.815 --> 00:35:47.775
perform
504
00:35:47.934 --> 00:35:57.295
very, very well on these longer sets of data. And a lot of the research that's been going into these very large models has been increasing their context windows
505
00:35:57.510 --> 00:36:00.550
and how to efficiently handle and select
506
00:36:00.950 --> 00:36:04.390
in extremely large data contexts
507
00:36:04.550 --> 00:36:07.270
what parts of the data to pay attention to.
508
00:36:07.510 --> 00:36:09.750
All of that is applicable in extracting
509
00:36:10.425 --> 00:36:14.825
different pieces of information accurately from these audio signals.
510
00:36:15.385 --> 00:36:16.025
But
511
00:36:16.345 --> 00:36:21.225
the next layer up is really where a lot of the Ensemble
512
00:36:21.225 --> 00:36:22.025
technology
513
00:36:22.025 --> 00:36:24.185
that I've been talking about comes in.
514
00:36:24.425 --> 00:36:24.985
So
515
00:36:25.450 --> 00:36:28.090
when you're building an ensemble model,
516
00:36:28.970 --> 00:36:30.730
if you can slot in
517
00:36:31.130 --> 00:36:33.770
a better emotion detection model,
518
00:36:33.850 --> 00:36:42.535
the whole ensemble's gonna work better. And the better emotion detection model might be replacing, say, a standard dilated convolution network
519
00:36:42.535 --> 00:36:45.975
with a network that employs an attention mechanism
520
00:36:46.135 --> 00:36:46.935
or
521
00:36:47.174 --> 00:36:48.695
some other thing like that.
522
00:36:51.174 --> 00:36:58.260
And that you have to change in terms of the Ensemble model is your priors on the accuracy of that model, assuming it gets more accurate,
523
00:36:58.420 --> 00:37:07.859
and your cost parameters of the model, assuming it might cost more to run. And then once you've changed that, the rest of the Ensemble just works as it normally did. So they're very complementary.
524
00:37:09.444 --> 00:37:13.125
Speaking to that longer horizon and the longer
525
00:37:13.684 --> 00:37:15.845
window of attention that's necessary,
526
00:37:15.924 --> 00:37:28.200
it also brings up the question of another trend that's becoming very popular in agentic contexts is the idea of agentic memory and being able to store and retrieve certain aspects of information
527
00:37:28.440 --> 00:37:29.400
for later
528
00:37:29.880 --> 00:37:30.680
usage.
529
00:37:30.760 --> 00:37:33.080
And particularly if you're dealing with
530
00:37:33.480 --> 00:37:34.280
a
531
00:37:34.359 --> 00:37:37.125
minutes to hours long conversation
532
00:37:37.125 --> 00:37:38.885
or even monologue.
533
00:37:39.125 --> 00:37:54.820
Obviously, there are cases where you're going to overflow the available context window for even an ensemble model, and I'm just curious how you take advantage of some of these storage and retrieval questions for being able to do that
534
00:37:55.060 --> 00:37:55.700
more
535
00:37:55.940 --> 00:37:57.140
long horizon
536
00:37:57.140 --> 00:37:58.020
analysis
537
00:37:58.020 --> 00:37:59.860
of this high
538
00:38:00.100 --> 00:38:01.620
information density
539
00:38:01.860 --> 00:38:02.420
medium.
540
00:38:03.795 --> 00:38:06.355
This is where knowing the structure
541
00:38:06.435 --> 00:38:26.950
of your data is so important. It's like what's what's old is new again. Right? It's always been in machine learning, like, understand what data your models are going to operate on. Bring in domain expertise as well as do backprop through the neural net. Right? For something like a conversation, you could have a five hour long conversation,
542
00:38:27.109 --> 00:38:32.595
but you have priors, very good priors on what data is actually important
543
00:38:32.915 --> 00:38:34.515
to that conversation.
544
00:38:34.675 --> 00:38:37.795
Right? You wanna know who are these people,
545
00:38:37.955 --> 00:38:43.155
what are their intents right now, what have their intents and goals been through the course of the conversation,
546
00:38:43.670 --> 00:38:53.270
How are they acting towards each other? What's the distribution of emotions? Like, for example, I sound pretty excited all the time. So when I deviate from that norm,
547
00:38:53.510 --> 00:38:55.590
then that's information that's important.
548
00:38:56.505 --> 00:39:00.345
By digging into, like, how do humans understand conversations,
549
00:39:00.345 --> 00:39:03.145
like, what kinds of tasks are you trying to achieve
550
00:39:03.385 --> 00:39:05.385
by analyzing these conversations,
551
00:39:05.464 --> 00:39:06.825
you can actually pre
552
00:39:07.145 --> 00:39:09.065
kind of preregister,
553
00:39:09.065 --> 00:39:10.185
precreate,
554
00:39:10.505 --> 00:39:11.464
preallocate
555
00:39:11.980 --> 00:39:16.940
the the the pieces of information that are going to be broadly,
556
00:39:17.740 --> 00:39:19.020
generally relevant
557
00:39:19.100 --> 00:39:19.900
to
558
00:39:19.900 --> 00:39:39.570
the conversation. If I'm gonna make if I'm if I'm gonna make a super, super simplified example, it's like, you know, pinning the header or pinning the rows in an Excel spreadsheet or something like that. It's like, yeah, you can kinda scroll back and look at a sentence that happened, like, three hours ago or a tab that's, you know or like a cell that's, like, you know, 200
559
00:39:39.570 --> 00:39:50.930
rows over. But you always know that some of this data is going to be important and should be front and center and visible to your models at all times. So instead of for our use case, so instead of
560
00:39:51.250 --> 00:39:53.090
trying to have some extremely
561
00:39:53.170 --> 00:39:54.050
very flexible
562
00:39:54.725 --> 00:39:55.605
dynamic
563
00:39:56.005 --> 00:40:01.845
storage with absolutely no priors or structure on it, we have a ton of preallocated
564
00:40:01.845 --> 00:40:04.965
structure around what's a running summary of the conversation.
565
00:40:05.045 --> 00:40:06.405
Who are the participants?
566
00:40:06.485 --> 00:40:11.280
What do they sound like? What's their general distribution of emotions
567
00:40:11.440 --> 00:40:13.200
across the conversation?
568
00:40:13.200 --> 00:40:15.760
All of these different things. And those are preregistered
569
00:40:15.760 --> 00:40:16.880
as kind of a
570
00:40:17.920 --> 00:40:22.480
fixed memory block that serves as the starting point for context
571
00:40:22.875 --> 00:40:26.155
for all of the individual models in the ensemble.
572
00:40:26.555 --> 00:40:31.595
And of course, some of the models can add more context and go in dynamically.
573
00:40:31.755 --> 00:40:43.020
And of course, those things like the summary and the participant roles and the behaviors they're exhibiting, they change over time. But we always know that those pieces of information are going to be relevant. So it massively,
574
00:40:43.020 --> 00:40:45.420
massively, massively simplifies
575
00:40:45.500 --> 00:40:47.260
the memory problem
576
00:40:47.500 --> 00:40:48.700
for this system,
577
00:40:48.860 --> 00:40:49.980
and it also
578
00:40:49.980 --> 00:40:50.860
reduces
579
00:40:50.860 --> 00:40:51.260
the
580
00:40:51.785 --> 00:40:54.265
ability to make weird errors.
581
00:40:54.265 --> 00:40:56.985
Right? If you're thinking about a system with memory,
582
00:40:57.065 --> 00:40:59.785
the system might be doing something really, really stupid.
583
00:41:00.025 --> 00:41:13.190
And you're like, why is it doing this? And it's because, oh, it didn't have the right information in memory. Why didn't it have the right information in memory? Now you're going down a very long debugging path that is going to be very complicated.
584
00:41:13.589 --> 00:41:14.150
And
585
00:41:14.550 --> 00:41:15.190
by
586
00:41:16.069 --> 00:41:21.785
changing how the system stores things in memory, you could change a ton of other behaviors without knowing it.
587
00:41:22.184 --> 00:41:23.145
Preregistering
588
00:41:23.145 --> 00:41:24.265
the structure
589
00:41:24.505 --> 00:41:31.944
of the memory or even just a subset of the memory ahead of time means that your models will always have the relevant information
590
00:41:32.400 --> 00:41:38.160
and reduces a ton of the errors that these very, very, very dynamics and flexible systems can have.
591
00:41:39.359 --> 00:41:40.880
As you have been
592
00:41:41.920 --> 00:41:44.000
exploring this architecture,
593
00:41:44.079 --> 00:41:46.160
developing your own implementation
594
00:41:46.160 --> 00:41:47.119
of it,
595
00:41:47.440 --> 00:41:47.599
and
596
00:41:48.615 --> 00:41:51.575
keeping an eye on just the overall ecosystem
597
00:41:51.575 --> 00:41:53.415
of how people are
598
00:41:53.575 --> 00:41:54.695
dealing with
599
00:41:55.015 --> 00:41:56.775
these model deployments,
600
00:41:56.855 --> 00:41:58.215
model optimization.
601
00:41:58.615 --> 00:42:00.215
What are some of the complexities
602
00:42:00.215 --> 00:42:03.895
that you as an engineering team have had to
603
00:42:04.250 --> 00:42:10.170
come to terms with and address to be able to achieve this objective and some of the
604
00:42:10.569 --> 00:42:12.570
useful learnings that you
605
00:42:13.609 --> 00:42:17.690
developed in the process of actually getting to delivering this as a feature set?
606
00:42:19.705 --> 00:42:22.585
Probably the biggest and most obvious
607
00:42:22.905 --> 00:42:25.705
complexity that we've had to deal with and overcome
608
00:42:26.025 --> 00:42:29.224
has been the fact that this Ensemble
609
00:42:29.224 --> 00:42:31.945
listening model architecture by necessity
610
00:42:32.025 --> 00:42:33.785
is a very distributed system.
611
00:42:34.329 --> 00:42:42.810
In general, if you're going to run a neural network on a lot of data, you want to load the weights into the GPU and keep them there. You don't want to be swapping models in and out.
612
00:42:43.290 --> 00:42:45.050
If you're going to do that for
613
00:42:45.690 --> 00:42:51.005
a possibility of over one one hundred different models that you might run on any given piece of data,
614
00:42:51.484 --> 00:42:54.845
some models can be shared in a single GPU.
615
00:42:54.925 --> 00:43:00.285
But you're not going to fit all 100 models into one GPU or even a couple GPUs.
616
00:43:00.525 --> 00:43:01.885
So you're starting to look at,
617
00:43:02.810 --> 00:43:04.730
in order to run inference
618
00:43:04.810 --> 00:43:06.570
on any individual
619
00:43:06.570 --> 00:43:07.530
data point,
620
00:43:08.010 --> 00:43:12.810
I'm going to need to route that data to a small subset
621
00:43:12.810 --> 00:43:13.450
of
622
00:43:13.610 --> 00:43:15.610
a bunch of different machines
623
00:43:16.025 --> 00:43:18.905
with a bunch of different models loaded onto them.
624
00:43:19.305 --> 00:43:21.385
So it very, very quickly becomes
625
00:43:21.865 --> 00:43:24.105
a distributed computing problem.
626
00:43:24.905 --> 00:43:27.065
When tackling a distributed problem,
627
00:43:27.305 --> 00:43:33.300
the distributed problem becomes a lot harder if you need strong consistency guarantees.
628
00:43:33.859 --> 00:43:37.380
One of the really nice things about modern machine learning
629
00:43:37.460 --> 00:43:40.740
is that a lot of models can be very forgiving
630
00:43:40.819 --> 00:43:47.525
when given errors or gaps in data. And we exploited that very, very heavily in the ELM architecture.
631
00:43:47.845 --> 00:43:49.125
So instead of
632
00:43:49.445 --> 00:43:50.725
asserting that
633
00:43:51.045 --> 00:43:55.125
all of the different models in the ensemble must have their data available
634
00:43:55.770 --> 00:44:00.890
by some sort of time or worse, locking until every single data point has come in,
635
00:44:02.250 --> 00:44:05.210
we make sure that our data flow
636
00:44:05.610 --> 00:44:06.650
goes from
637
00:44:07.450 --> 00:44:10.970
less flexible models to more flexible models
638
00:44:11.155 --> 00:44:13.395
in a pretty strict hierarchy,
639
00:44:13.395 --> 00:44:15.715
at least for that feedforward processing.
640
00:44:16.355 --> 00:44:16.995
So
641
00:44:17.315 --> 00:44:19.315
if our less sophisticated
642
00:44:19.635 --> 00:44:21.635
models, smaller models,
643
00:44:22.355 --> 00:44:24.115
don't all return results
644
00:44:24.115 --> 00:44:26.435
in a required amount of time,
645
00:44:27.279 --> 00:44:31.040
The good news is that data is flowing into a more flexible
646
00:44:31.200 --> 00:44:32.480
and more forgiving,
647
00:44:32.480 --> 00:44:34.480
in some ways, model.
648
00:44:34.799 --> 00:44:38.160
And we have a much greater ability
649
00:44:38.400 --> 00:44:46.735
to account for and deal with missing data than we would if we were feeding into another super tiny, super specialized
650
00:44:47.535 --> 00:44:49.695
model in the Ensemble.
651
00:44:50.175 --> 00:44:55.615
So that's been a very, very strong rule of thumb for us. Always flow data from
652
00:44:56.070 --> 00:44:57.750
less sophisticated,
653
00:44:57.830 --> 00:44:59.030
smaller models
654
00:44:59.350 --> 00:45:00.150
into
655
00:45:00.150 --> 00:45:02.710
more flexible, more general models.
656
00:45:03.270 --> 00:45:07.590
And that helps us account for these distributed computing errors
657
00:45:07.590 --> 00:45:10.150
and still do a good job generally,
658
00:45:10.470 --> 00:45:13.305
even in times of high model dropout.
659
00:45:13.945 --> 00:45:14.585
And
660
00:45:15.385 --> 00:45:17.065
as far as the
661
00:45:17.305 --> 00:45:21.305
cost question of being able to deliver this functionality
662
00:45:21.465 --> 00:45:23.145
more economically,
663
00:45:23.465 --> 00:45:26.345
what are some of the ways that you're tracking the
664
00:45:26.510 --> 00:45:39.869
real world impact of that, particularly given the distributed nature of the actual cost as well as the increased operational overhead and potentially headcount needed to be able to keep the system running?
665
00:45:42.585 --> 00:45:43.465
Good question.
666
00:45:44.105 --> 00:45:46.985
In terms of the additional complexity,
667
00:45:47.785 --> 00:45:49.145
I would say that
668
00:45:49.625 --> 00:45:53.785
most engineering systems have complexity in a variety of different places.
669
00:45:54.345 --> 00:45:54.985
And
670
00:45:55.460 --> 00:45:56.180
by
671
00:45:56.339 --> 00:45:57.380
specifically
672
00:45:57.380 --> 00:46:01.780
focusing on orchestrating a bunch of different models together,
673
00:46:02.420 --> 00:46:03.460
we're solving
674
00:46:04.500 --> 00:46:07.780
a ton of these different orchestration problems,
675
00:46:07.859 --> 00:46:10.180
but they're all very similar problems.
676
00:46:11.165 --> 00:46:11.885
So
677
00:46:13.165 --> 00:46:18.125
it actually pays off for us to have a small number of really, really
678
00:46:19.165 --> 00:46:20.925
deep knowledge experts
679
00:46:21.165 --> 00:46:21.885
in
680
00:46:22.285 --> 00:46:24.045
orchestrating systems of models
681
00:46:24.300 --> 00:46:27.260
and then building reusable tools
682
00:46:27.580 --> 00:46:28.780
to manage
683
00:46:28.940 --> 00:46:29.660
those
684
00:46:29.740 --> 00:46:31.500
different ensembles.
685
00:46:31.820 --> 00:46:33.500
Given that we can do that,
686
00:46:33.900 --> 00:46:39.660
we end up not having a huge amount of what I would call additional overhead
687
00:46:39.325 --> 00:46:40.765
Because the alternative
688
00:46:41.085 --> 00:46:46.125
is if you're working with a very, very, very big foundational model,
689
00:46:47.165 --> 00:46:49.485
now you're trying to shard
690
00:46:49.805 --> 00:46:52.605
that model across a bunch of different GPUs.
691
00:46:53.319 --> 00:46:56.359
And if you're trying to do data communication
692
00:46:56.440 --> 00:46:59.720
between different parts of, say, one neural network,
693
00:47:00.280 --> 00:47:01.560
that's a situation
694
00:47:01.880 --> 00:47:02.520
where
695
00:47:03.319 --> 00:47:08.599
changes in data or dropout or distributed system failures can be very catastrophic.
696
00:47:09.215 --> 00:47:15.535
It's very, very unpredictable what a neural network will do if you just cut out a little chunk of its weights.
697
00:47:16.335 --> 00:47:23.135
And so you have to solve those distributed problems if you're building big, like very big foundation models
698
00:47:23.589 --> 00:47:26.230
in strong consistency ways.
699
00:47:26.470 --> 00:47:30.230
And you have to deploy a ton of really, really capable hardware
700
00:47:30.390 --> 00:47:32.710
and network them in extremely
701
00:47:32.710 --> 00:47:34.630
low latency setups
702
00:47:34.869 --> 00:47:37.270
in order to do that kind of inference
703
00:47:37.385 --> 00:47:40.585
in a way that results in a good customer experience.
704
00:47:41.305 --> 00:47:46.665
By chunking down all of our models to be able to run on single machines or single GPUs,
705
00:47:47.224 --> 00:47:49.865
we actually buy a ton of flexibility.
706
00:47:50.025 --> 00:47:51.705
And so I wouldn't say it's
707
00:47:51.944 --> 00:47:52.744
more complex.
708
00:47:53.360 --> 00:47:55.600
It's a different kind of complexity,
709
00:47:55.840 --> 00:47:58.400
and in some ways, actually more forgiving
710
00:47:58.400 --> 00:48:03.840
than trying to be a model provider that just holds a really big foundational model.
711
00:48:04.560 --> 00:48:06.720
And then
712
00:48:06.065 --> 00:48:08.065
in terms of the
713
00:48:08.945 --> 00:48:11.665
evolution of this style of architecture,
714
00:48:12.065 --> 00:48:14.305
what are some of the impacts
715
00:48:14.305 --> 00:48:15.825
that you foresee
716
00:48:16.465 --> 00:48:19.665
these capabilities having on the broader industry
717
00:48:19.825 --> 00:48:25.340
and the challenges that we're already running into as far as just brute force scaling and
718
00:48:25.500 --> 00:48:32.780
increasing the number of parameters versus these more small specialized models, and then also how that factors into
719
00:48:32.940 --> 00:48:45.215
the push to the edge of having this be a more naturally distributed system where you can have some of these very small models on the edge to determine whether to even propagate that request downstream.
720
00:48:46.575 --> 00:48:48.655
I think that Ensemble
721
00:48:49.135 --> 00:48:49.935
models
722
00:48:50.410 --> 00:48:52.010
are going to
723
00:48:52.250 --> 00:48:53.290
replace
724
00:48:53.370 --> 00:48:56.730
a lot of existing foundation model use cases
725
00:48:56.970 --> 00:48:58.410
and extend
726
00:48:58.730 --> 00:49:02.010
a lot of other foundation model capabilities.
727
00:49:02.490 --> 00:49:05.050
So in the replacing existing use cases,
728
00:49:07.665 --> 00:49:09.505
I see Ensemble models
729
00:49:09.585 --> 00:49:11.425
as being the right
730
00:49:11.745 --> 00:49:12.545
tool
731
00:49:12.865 --> 00:49:13.985
to solve
732
00:49:14.225 --> 00:49:15.185
repeated
733
00:49:15.185 --> 00:49:16.705
structured problems.
734
00:49:17.105 --> 00:49:19.985
Again, even as general as conversation analysis,
735
00:49:20.640 --> 00:49:25.119
that is a very general problem statement but much, much narrower than
736
00:49:25.359 --> 00:49:26.160
arbitrary
737
00:49:26.160 --> 00:49:28.160
operations on streams of data.
738
00:49:28.640 --> 00:49:32.160
So if you're going to do a lot of conversation analysis,
739
00:49:32.775 --> 00:49:35.974
it can pay off in terms of accuracy,
740
00:49:35.974 --> 00:49:37.494
determinism, and cost
741
00:49:37.734 --> 00:49:38.535
to
742
00:49:38.855 --> 00:49:42.455
build out an ensemble to take advantage of that structure.
743
00:49:43.174 --> 00:49:45.175
I think of the parallel as
744
00:49:45.815 --> 00:49:47.575
if you have a novel task,
745
00:49:48.359 --> 00:49:53.320
say, this is the first time I ever wanna just write down a transcript of this podcast.
746
00:49:53.720 --> 00:49:55.800
You just go ask somebody to do it.
747
00:49:56.279 --> 00:49:59.079
But if you need to transcribe 10,000 podcasts,
748
00:49:59.400 --> 00:50:02.519
you don't go ask a human to transcribe 10,000 podcasts.
749
00:50:02.825 --> 00:50:05.545
You ask a specially constructed
750
00:50:06.745 --> 00:50:07.545
system
751
00:50:07.705 --> 00:50:09.945
to do that transcription
752
00:50:09.945 --> 00:50:13.465
because it's going to do that same work deterministically
753
00:50:13.705 --> 00:50:16.740
10,000 times over. In a similar way,
754
00:50:17.140 --> 00:50:19.300
these large model companies,
755
00:50:19.460 --> 00:50:21.300
like OpenAI and Anthropic,
756
00:50:21.460 --> 00:50:22.260
are building
757
00:50:22.500 --> 00:50:24.340
basically artificial people.
758
00:50:25.060 --> 00:50:34.575
And those systems are trying to be extremely, extremely general. But just as you wouldn't ask a real person to do the same repetitive task 10,000 times,
759
00:50:35.055 --> 00:50:40.095
I think you also wouldn't ask an artificial person to do the same repetitive task
760
00:50:40.175 --> 00:50:48.090
10,000 times. You're just wasting a ton of capacity in either case, and you would much rather prefer a deterministic,
761
00:50:48.330 --> 00:50:50.490
accurate, and more cost effective
762
00:50:50.810 --> 00:50:53.610
solution that takes advantage of the structure
763
00:50:53.930 --> 00:50:54.570
and
764
00:50:54.730 --> 00:50:55.610
repeatability
765
00:50:55.610 --> 00:50:57.450
of the problem you're trying to tackle.
766
00:50:58.035 --> 00:50:59.555
So a lot of
767
00:50:59.875 --> 00:51:01.875
businesses and individuals
768
00:51:01.954 --> 00:51:05.475
are using big foundation models to tackle
769
00:51:05.714 --> 00:51:06.755
repetitive,
770
00:51:06.994 --> 00:51:10.194
highly structured tasks just because they're so powerful.
771
00:51:10.960 --> 00:51:17.680
But I think that is going to turn out to be very, very far from the most cost effective and scalable solution.
772
00:51:18.080 --> 00:51:20.480
When you have these repeated situations,
773
00:51:20.720 --> 00:51:26.315
you can deploy systems that are much better adapted to them. So I think that's
774
00:51:26.395 --> 00:51:28.795
how Ensemble models will
775
00:51:29.355 --> 00:51:30.155
replace
776
00:51:30.155 --> 00:51:39.434
some of the applications of the larger foundation models. I think the flip side is they'll actually work in harmony to augment the intelligence and capabilities.
777
00:51:39.690 --> 00:51:45.450
Right? So especially with things like like like chain of thought systems and memory systems,
778
00:51:45.609 --> 00:51:47.130
having more compute
779
00:51:47.450 --> 00:51:50.730
gives you access to more intelligence capabilities.
780
00:51:51.655 --> 00:51:55.015
But if you're trying to do something like media understanding
781
00:51:55.015 --> 00:51:57.335
or voice understanding or conversation
782
00:51:57.335 --> 00:51:58.295
understanding,
783
00:51:58.695 --> 00:52:01.495
you're spending a lot of your compute and your capacity
784
00:52:01.895 --> 00:52:03.335
to process
785
00:52:03.335 --> 00:52:07.460
that stream of data and get to a useful representation
786
00:52:07.460 --> 00:52:11.380
of that data that the rest of the system can work with.
787
00:52:12.020 --> 00:52:13.700
I see Ensemble models
788
00:52:13.780 --> 00:52:14.500
as
789
00:52:14.580 --> 00:52:17.380
basically becoming a suite of extremely
790
00:52:17.380 --> 00:52:18.260
powerful
791
00:52:18.660 --> 00:52:21.380
data processing tools
792
00:52:20.744 --> 00:52:23.545
available to these more flexible agents
793
00:52:23.704 --> 00:52:27.785
so that if you have an incoming data stream or a media stream,
794
00:52:28.025 --> 00:52:31.464
the agent can deploy a tool like an ELM
795
00:52:31.545 --> 00:52:34.480
to understand that stream, save,
796
00:52:34.720 --> 00:52:37.200
again, 100000x
797
00:52:37.280 --> 00:52:41.680
on the compute it would have by itself needed to understand that data stream,
798
00:52:41.920 --> 00:52:46.640
and then take the representation of that data produced by the Ensemble model
799
00:52:47.075 --> 00:52:54.275
and process it with all of that leftover compute that was available to it because it didn't have to understand the data itself.
800
00:52:55.075 --> 00:53:04.839
And the real question I wanna know is when is this Voice AI going to eliminate the need to manually edit my podcasts?
801
00:53:05.799 --> 00:53:10.760
It's coming. It's coming very, very soon. I think 2026
802
00:53:11.000 --> 00:53:12.440
is already
803
00:53:12.805 --> 00:53:15.205
starting off as an extraordinary
804
00:53:15.205 --> 00:53:17.045
year for voice AI models.
805
00:53:17.125 --> 00:53:18.085
You're seeing
806
00:53:18.405 --> 00:53:19.445
increases
807
00:53:19.445 --> 00:53:20.165
from
808
00:53:20.405 --> 00:53:23.125
Ensemble models and other kinds of technologies
809
00:53:23.205 --> 00:53:26.565
of 10, a 100 x in terms of cost,
810
00:53:26.805 --> 00:53:29.040
in terms of scalability.
811
00:53:29.120 --> 00:53:31.200
You're seeing improvements in accuracy,
812
00:53:31.200 --> 00:53:35.200
not just on the normal standard clean data sets, but on tricky
813
00:53:35.440 --> 00:53:40.160
conversation data sets. You're seeing voice AIs that are starting to be able to take instruction
814
00:53:40.815 --> 00:53:44.335
and follow instructions and follow nuances
815
00:53:44.415 --> 00:53:46.575
in order to be much, much more capable.
816
00:53:46.895 --> 00:53:50.575
And you're starting to get to the level of accuracy and capability
817
00:53:50.655 --> 00:53:52.335
that human beings
818
00:53:52.415 --> 00:53:55.840
doing these kinds of tasks also exhibit.
819
00:53:56.080 --> 00:54:11.185
So I think we're going to see a lot of really, really cool tech around voice AI coming out within the next twelve months. 2026 is gonna be a very, very exciting year for voice AI tech. Are there any other aspects of the work that you're doing at Modulate,
820
00:54:11.185 --> 00:54:12.945
this overall Ensemble
821
00:54:12.945 --> 00:54:13.905
architecture,
822
00:54:14.224 --> 00:54:15.585
or the applications
823
00:54:15.585 --> 00:54:19.825
that you're applying it to that we didn't discuss yet that you'd like to cover before we close out the show?
824
00:54:20.145 --> 00:54:28.390
I think we've hit on the main points. I think for a different audience, it could be interesting to go into the determinism
825
00:54:28.390 --> 00:54:29.590
and kind of
826
00:54:30.230 --> 00:54:31.430
ability to
827
00:54:32.150 --> 00:54:33.030
avoid
828
00:54:33.990 --> 00:54:37.670
things like hallucinations better by using these
829
00:54:37.670 --> 00:54:40.155
ensembles of small models.
830
00:54:40.395 --> 00:54:47.275
But I think that's actually more applicable to, like, maybe almost like a a compliance or safety audience,
831
00:54:47.675 --> 00:54:55.850
which is a lot of aspects of the things we're working in. So I think we actually hit on a a ton of the main topics I really wanted to cover. I'm feeling pretty good about this.
832
00:54:56.810 --> 00:55:13.655
Alright. Well, for anybody who wants to get in touch with you and follow along with the work that you and your team are doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gaps in the tooling technology or human training that's available for AI systems today.
833
00:55:14.615 --> 00:55:15.415
The
834
00:55:15.415 --> 00:55:18.775
biggest problem with AI tooling and systems
835
00:55:18.935 --> 00:55:19.415
is
836
00:55:19.750 --> 00:55:20.790
understanding
837
00:55:21.030 --> 00:55:24.710
where they fail and what kinds of failures they can exhibit.
838
00:55:25.190 --> 00:55:26.550
Think of things like
839
00:55:26.710 --> 00:55:30.950
customer support bots and voice bots that will just offer refunds
840
00:55:31.110 --> 00:55:37.105
or even when they're out of policy or reset people's passwords even when they haven't authenticated,
841
00:55:37.345 --> 00:55:37.905
or
842
00:55:38.385 --> 00:55:42.545
will flirt with somebody because they started flirting with it.
843
00:55:43.505 --> 00:55:46.065
These systems can go off the rails
844
00:55:46.145 --> 00:55:47.585
very, very easily
845
00:55:47.825 --> 00:55:49.265
and in unpredictable
846
00:55:49.265 --> 00:55:49.664
ways.
847
00:55:50.349 --> 00:55:51.869
And there is not
848
00:55:52.829 --> 00:55:56.670
These large models are the epitome of a black box algorithm.
849
00:55:57.470 --> 00:55:59.470
Even things like chain of thought,
850
00:55:59.950 --> 00:56:02.590
it gives the illusion that you can understand
851
00:56:03.070 --> 00:56:04.589
how the model operates.
852
00:56:05.505 --> 00:56:06.224
But
853
00:56:06.625 --> 00:56:09.185
there's actually a ton of
854
00:56:11.585 --> 00:56:15.664
different kinds of failure modes and different kinds of behaviors
855
00:56:15.664 --> 00:56:17.744
that these models can unpredictably
856
00:56:17.744 --> 00:56:18.464
exhibit
857
00:56:18.710 --> 00:56:19.430
from
858
00:56:19.670 --> 00:56:27.670
the right random combinations of input data. And it's very, very hard to know or guarantee or even put bounds on
859
00:56:28.150 --> 00:56:31.910
when you will experience the normal happy path operation
860
00:56:32.150 --> 00:56:33.750
and when you will experience
861
00:56:34.464 --> 00:56:35.585
a negative,
862
00:56:35.585 --> 00:56:36.625
unexpected,
863
00:56:36.625 --> 00:56:38.625
or out of distribution operation.
864
00:56:39.425 --> 00:56:40.865
And I think
865
00:56:41.105 --> 00:56:43.265
that in addition to
866
00:56:43.744 --> 00:56:44.545
some
867
00:56:44.865 --> 00:56:47.825
tools for testing and observability,
868
00:56:48.145 --> 00:56:51.040
which are, I think, still relatively
869
00:56:51.040 --> 00:56:51.920
primitive.
870
00:56:52.400 --> 00:56:56.240
We are going to start looking for a lot of automated tooling
871
00:56:56.400 --> 00:56:57.280
around
872
00:56:59.440 --> 00:57:00.960
red teaming models,
873
00:57:01.040 --> 00:57:02.000
finding
874
00:57:02.865 --> 00:57:03.665
gaps
875
00:57:03.985 --> 00:57:04.785
in
876
00:57:04.865 --> 00:57:06.545
how models are running
877
00:57:06.785 --> 00:57:09.345
on data and systems and data streams,
878
00:57:09.745 --> 00:57:10.305
and
879
00:57:11.105 --> 00:57:20.460
monitoring these systems at extremely high scale. Watch every single conversation that's going on just like you would monitor human interactions
880
00:57:20.460 --> 00:57:22.700
at a very, very high level of scale.
881
00:57:23.020 --> 00:57:23.660
And
882
00:57:23.980 --> 00:57:27.980
right now, a lot of the tooling for observability and monitoring
883
00:57:28.780 --> 00:57:29.660
rely on
884
00:57:30.215 --> 00:57:33.735
looking at the model's own intermediate outputs.
885
00:57:33.975 --> 00:57:38.775
Like if you have a voice bot that's doing speech to text to speech, you monitor the text.
886
00:57:39.015 --> 00:57:42.375
But the problem with that is that you're monitoring a part of the system
887
00:57:42.775 --> 00:57:43.415
that
888
00:57:43.630 --> 00:57:45.150
you're trying to
889
00:57:45.390 --> 00:57:47.790
observe the behavior of the whole system.
890
00:57:48.109 --> 00:57:50.350
And so if a different part,
891
00:57:50.590 --> 00:57:56.990
like the text to speech piece of the bot, is going wrong, if you're just monitoring the text part,
892
00:57:57.230 --> 00:58:01.325
you're not going to know what's happening. So being able to observe
893
00:58:01.325 --> 00:58:03.405
the behaviors of these models
894
00:58:03.565 --> 00:58:04.605
and interacting
895
00:58:04.605 --> 00:58:06.445
with each other and with people
896
00:58:06.685 --> 00:58:08.685
in the same way that you would observe
897
00:58:09.005 --> 00:58:13.965
human beings interacting with each other and being able to derive insights and debug
898
00:58:14.310 --> 00:58:15.350
those things
899
00:58:15.990 --> 00:58:21.190
is I think a really big gap and where a lot of these problems and failure modes and
900
00:58:21.430 --> 00:58:23.510
newsworthy headlines are coming from.
901
00:58:24.070 --> 00:58:24.950
Absolutely.
902
00:58:25.110 --> 00:58:33.035
Well, thank you very much for taking the time today to join me and share the work that you and your team have put into developing this Ensemble architecture
903
00:58:33.035 --> 00:58:40.395
and some of the interesting use cases for voice AI as differentiated from the textual interfaces
904
00:58:40.395 --> 00:58:42.475
that have exemplified
905
00:58:42.475 --> 00:58:53.670
the past couple of years. So I appreciate all the time and energy that you and your team are putting into that, and I hope you enjoy the rest of your day. Well, thank you so much for having me. It was wonderful to chat, and I really enjoyed it. I hope you enjoy your day too.
906
00:58:55.785 --> 00:59:03.865
Thank you for listening. Don't forget to check out our other shows. The data engineering podcast covers the latest on modern data management, and podcast.init
907
00:59:03.865 --> 00:59:18.828
covers the Python language, its community, and the innovative ways it is being used. Visit the site to subscribe to the show, sign up for the mailing list, read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email hosts@aiengineeringpodcast.com
908
00:59:18.828 --> 00:59:19.548
with your story.