Full Transcript for Taming Voice Complexity with Dynamic Ensembles at Modulate

00:04:27.125 --> 00:04:28.005 with the 50 00:04:28.325 --> 00:04:45.820 caveat that it's only trying to convert that to a textual representation to increase the amount of corpora available for training of these language models. And I'm wondering if you can add a bit more nuance to some of the detail of what you're talking about with those elements of inflection and tone 51 00:04:45.900 --> 00:04:51.580 and how that's differentiated from purely transcription based or being able to even do 52 00:04:52.045 --> 00:05:00.045 speech to speech where I just wanna be able to take that voice signal, convert it to text, feed it to an LLM, generate some output, and then convert it back to voice. 53 00:05:00.685 --> 00:05:01.565 Absolutely. 54 00:05:01.565 --> 00:05:02.445 Yeah. So 55 00:05:02.765 --> 00:05:08.680 doing that flow speech to text to speech works reasonably well when 56 00:05:08.760 --> 00:05:09.960 the emotional 57 00:05:09.960 --> 00:05:14.280 content sentiment or kind of long term context of a conversation 58 00:05:14.520 --> 00:05:16.360 is relatively unimportant. 59 00:05:16.360 --> 00:05:23.175 So you can get kind of reasonable responses and reasonable back and forth on short time scales in that situation. 60 00:05:23.414 --> 00:05:28.615 But voice contains a ton more signal than just the, like, textual content 61 00:05:28.935 --> 00:05:43.510 of of what you're saying. How you say it is super, super important. So, like, for example, you know, I I say, like, hey. I'm super excited to be here. I'm thrilled. Or I can say, like, yeah. I'm super excited to be here. I'm thrilled. 62 00:05:43.750 --> 00:05:55.105 The exact same words. And if you transcribe it and feed it to an LLM, it will do exactly the same thing with those two signals. But you and I know that you would react to that statement extremely 63 00:05:55.105 --> 00:05:55.985 differently. 64 00:05:56.465 --> 00:06:02.865 And that's just one of the aspects. And there's a ton of, like, social relationship and context packed into 65 00:06:03.720 --> 00:06:09.320 why am I expressing that emotion in that way. So that was what makes it difficult, 66 00:06:09.560 --> 00:06:18.680 and that's kind of what makes it very important to be able to capture those other signals because you have to react almost in opposite ways depending on the tone and emotion of the voice. 67 00:06:19.775 --> 00:06:21.695 And now digging into 68 00:06:21.855 --> 00:06:25.775 some of the announcement of your ensemble listening approach, 69 00:06:25.935 --> 00:06:27.695 what are some of the 70 00:06:28.575 --> 00:06:36.090 challenges or constraints that are specific to voice AI that led you to putting in the effort to actually 71 00:06:36.890 --> 00:06:49.955 researching and developing this approach to how to actually manage some of that input processing and some of the general expectations and requirements around how it's supposed to operate that led you down that path? 72 00:06:50.595 --> 00:06:51.395 Absolutely. 73

00:06:51.395 --> 00:06:53.955 So I would say there's two main constraints 74 00:06:53.955 --> 00:06:58.275 that factored into our decision to pursue a totally new architecture. 75 00:06:58.595 --> 00:07:03.030 Constraint number one is cost and processing power required. 76 00:07:03.509 --> 00:07:05.030 Just transcribing, 77 00:07:05.110 --> 00:07:06.790 analyzing a waveform, 78 00:07:07.110 --> 00:07:09.750 even doing simple relatively simple operations 79 00:07:09.750 --> 00:07:19.065 like detecting accent or detecting whether there's music in the background or what kind of background noise there is, those kinds of operations on an audio stream, 80 00:07:19.865 --> 00:07:21.225 you can get 81 00:07:21.385 --> 00:07:28.110 a little bit of the way there with relatively simple and cheap models. But to do a very good job, 82 00:07:28.430 --> 00:07:33.950 you have to deploy much more powerful and expensive models. Like, there are transcription 83 00:07:33.950 --> 00:07:40.465 services out there that cost, you know, anywhere from 10¢ to a dollar to process an hour of audio. 84 00:07:40.705 --> 00:07:42.705 If you're a large social platform, 85 00:07:43.025 --> 00:07:44.865 you could have 50,000,000, 86 00:07:44.865 --> 00:07:46.065 a 100,000,000, 87 00:07:46.305 --> 00:07:51.185 three billion hours of voice content on your platform each month. 88 00:07:51.665 --> 00:07:52.385 And 89 00:07:52.810 --> 00:07:54.810 you're not really gonna pay $3,000,000,000 90 00:07:54.810 --> 00:08:03.530 a month to analyze that audio if you wanna do something useful with it. So on the one hand, you have to be really, really cost effective and efficient 91 00:08:03.610 --> 00:08:08.090 at processing those audio signals if you're gonna do anything useful at scale. 92 00:08:08.705 --> 00:08:16.065 So that's problem statement number one. Problem statement number two is that audio and especially voice conversations 93 00:08:16.065 --> 00:08:18.545 in particular are very kind of like a 94 00:08:18.865 --> 00:08:21.345 yeah. I I call it like a lumpy multimodal 95 00:08:21.345 --> 00:08:22.385 distribution. 96 00:08:22.625 --> 00:08:23.665 Right? Like, you have 97 00:08:24.210 --> 00:08:25.810 different pockets 98 00:08:26.130 --> 00:08:26.770 of 99 00:08:27.010 --> 00:08:28.370 kinds of audio, 100 00:08:28.610 --> 00:08:32.050 but within those pockets, there's a ton of shared similarities. 101 00:08:32.290 --> 00:08:32.770 So 102 00:08:33.330 --> 00:08:35.810 we're doing, like, a podcast conversation. 103 00:08:35.810 --> 00:08:40.714 Right? We're in relatively high quality audio environment, relatively low background noise. 104 00:08:40.795 --> 00:08:42.154 We're enunciating 105 00:08:42.315 --> 00:08:49.195 relatively clearly or at least you are. I hope I am. And we have kind of these properties that are shared 106 00:08:49.435 --> 00:08:50.235 among 107 00:08:50.635 --> 00:08:53.274 other very similar audio environments. 108 00:08:53.620 --> 00:09:01.860 And then contrast

that with calling into a customer support center, and I'm on my phone. It's over telephony, so it's, like, eight kilohertz audio, 109 00:09:01.940 --> 00:09:13.495 and I'm on a bus, and there are people talking in the background. And there are a ton of also calls like that, but that's a very different distribution. You take a model like Whisper and you apply it to that distribution, 110 00:09:13.735 --> 00:09:16.215 it's gonna produce absolute junk. 111 00:09:16.375 --> 00:09:17.015 Right? 112 00:09:17.335 --> 00:09:20.055 So you have these different pockets 113 00:09:20.310 --> 00:09:29.590 where you have a lot of voice conversations that are very similar to each other, but the pockets are very, very different and require very different modeling approaches. 114 00:09:29.750 --> 00:09:31.110 And if you wanna build 115 00:09:31.510 --> 00:09:37.555 a big model that can handle all of those different pockets, all of the different modes of that distribution, 116 00:09:37.954 --> 00:09:39.714 it has to be extremely 117 00:09:39.714 --> 00:09:41.235 flexible and powerful 118 00:09:41.315 --> 00:09:43.955 to handle those different data distributions 119 00:09:44.035 --> 00:09:48.620 equally well. If you build a model that's extremely powerful, 120 00:09:48.699 --> 00:09:51.980 it's gonna be super cost effective, so you violate the first constraint. 121 00:09:52.060 --> 00:09:54.060 So it's a really, really tricky situation, 122 00:09:54.060 --> 00:09:55.660 which is why it required new tech. 123 00:09:56.139 --> 00:09:58.220 And so digging into 124 00:09:58.300 --> 00:10:01.339 the specifics of this Ensemble architecture, 125 00:10:01.420 --> 00:10:03.394 what are some of the 126 00:10:04.035 --> 00:10:10.435 high level details about how it actually addresses some of those complexities around the 127 00:10:10.675 --> 00:10:17.555 high cost as well as high latency issues that come up when you're dealing with these large multipurpose 128 00:10:17.555 --> 00:10:18.274 models 129 00:10:18.590 --> 00:10:29.150 and then also tries to address some of the accuracy issues that you run into when you go to a smaller or more quantized model to address some of those latency considerations? 130 00:10:30.030 --> 00:10:30.990 Absolutely. 131 00:10:30.990 --> 00:10:34.350 Great question. So there are really two main properties 132 00:10:34.565 --> 00:10:38.565 that led us to the Ensemble model architecture. 133 00:10:38.885 --> 00:10:43.125 Property number one is that when we're talking about those multimodal 134 00:10:43.125 --> 00:10:44.245 distributions, 135 00:10:44.725 --> 00:10:50.940 because the similarities are so close to each other within a single kind of mode of that distribution, 136 00:10:51.980 --> 00:10:54.779 all of the data is relatively 137 00:10:54.779 --> 00:10:55.580 similar, 138 00:10:55.980 --> 00:11:01.180 and so you don't need a ton of capacity. A small model could get you very far. 139 00:11:01.675 --> 00:11:05.035 Small specialized models can

perform with high accuracy 140 00:11:05.115 --> 00:11:06.955 in certain circumstances, 141 00:11:07.435 --> 00:11:10.155 but you lose a lot in terms of generalizability 142 00:11:10.155 --> 00:11:13.435 if you are focused on a small model. 143 00:11:14.339 --> 00:11:15.060 However, 144 00:11:16.019 --> 00:11:17.620 the other insight 145 00:11:17.620 --> 00:11:20.740 is that these distributions 146 00:11:21.060 --> 00:11:24.660 continue on over the course of the conversation. 147 00:11:24.899 --> 00:11:29.140 So you and I are having this high audio quality back and forth conversation. 148 00:11:29.705 --> 00:11:36.105 We're very unlikely to switch to something like a noisy bus, low audio quality environment 149 00:11:36.345 --> 00:11:40.265 halfway through the conversation. Or even more importantly, we're unlikely to be swapping 150 00:11:40.505 --> 00:11:45.950 back and forth all the time. And you might have different distributions for different participants, 151 00:11:46.110 --> 00:11:48.350 but they remain relatively 152 00:11:48.350 --> 00:11:51.230 static over the course of a conversation. 153 00:11:51.709 --> 00:11:54.750 So the way to bring these two things together 154 00:11:54.990 --> 00:11:57.149 is that if you can isolate 155 00:11:57.149 --> 00:11:58.670 quickly and efficiently 156 00:11:59.245 --> 00:12:00.925 which distribution 157 00:12:01.165 --> 00:12:06.045 you're in or which distribution each participant of the conversation is in, 158 00:12:06.525 --> 00:12:09.005 then if you also have 159 00:12:09.325 --> 00:12:15.630 a small model or a set of small models that are tuned specifically for that distribution, 160 00:12:16.269 --> 00:12:20.190 you can pick the right model to apply to that distribution, 161 00:12:20.670 --> 00:12:25.230 trust it will mostly do a good job over the whole course of the conversation, 162 00:12:25.550 --> 00:12:29.470 and then course correct if something does change as it as it occasionally might. 163 00:12:30.185 --> 00:12:33.225 So this is sort of like how a really big model 164 00:12:33.945 --> 00:12:39.945 will find paths through the weights in the large network that are appropriate for different distributions. 165 00:12:39.945 --> 00:12:46.290 But instead of having to run all the weights in the network or at least a large subset of the weights in the network, 166 00:12:46.690 --> 00:12:47.570 you are 167 00:12:47.890 --> 00:13:00.545 pre isolating down to a very, very small subset of the overall Ensemble model that you know ahead of time or know based on the first analysis of the data in the conversation 168 00:13:00.625 --> 00:13:02.865 is going to be applicable to 169 00:13:03.025 --> 00:13:06.385 likely the whole rest of the conversation distribution. 170 00:13:07.345 --> 00:13:08.465 Architecturally, 171 00:13:08.465 --> 00:13:09.185 having 172 00:13:09.425 --> 00:13:12.529 more models increases the complexity 173 00:13:12.529 --> 00:13:21.890 and potential for failure as well as the variety of

failure modes as is the case with any complex distributed system. And so that definitely requires 174 00:13:21.890 --> 00:13:22.610 having 175 00:13:22.930 --> 00:13:25.649 a high degree of confidence 176 00:13:25.649 --> 00:13:26.930 in the 177 00:13:27.215 --> 00:13:28.575 orchestration, 178 00:13:28.655 --> 00:13:30.255 the failure recovery, 179 00:13:30.495 --> 00:13:44.769 and I imagine also some measure of fallback capability for the situation where the ensemble just catastrophically fails. And I'm just wondering if you can talk to some of the ways that you thought through and proved out this overall 180 00:13:45.009 --> 00:13:47.410 engineering effort and what your 181 00:13:47.889 --> 00:13:55.170 calculus was as far as which pieces you were able to pull off the shelf and what are the pieces that you had to engineer from whole cloth. 182 00:13:56.765 --> 00:13:57.645 Great question. 183 00:13:58.125 --> 00:13:58.685 So 184 00:14:00.125 --> 00:14:04.685 one of the things that surprised us most when we started deploying 185 00:14:04.845 --> 00:14:07.005 these Ensemble models 186 00:14:07.565 --> 00:14:08.925 was really that 187 00:14:09.580 --> 00:14:11.100 as they were 188 00:14:11.260 --> 00:14:12.700 running inference, 189 00:14:13.899 --> 00:14:17.580 we found it very hard to predict actually 190 00:14:17.899 --> 00:14:20.140 which models were important 191 00:14:20.140 --> 00:14:21.100 to the results 192 00:14:21.455 --> 00:14:33.135 and which models were relatively unimportant to the results and when a model was doing a good job or when a model was doing a bad job. So the actually, the initial 193 00:14:34.360 --> 00:14:36.600 idea, the initial architecture, 194 00:14:36.600 --> 00:14:38.279 sort of the v one 195 00:14:38.519 --> 00:14:41.240 of these ELMs in our particular 196 00:14:41.560 --> 00:14:43.560 model Velma, sort of v one, 197 00:14:43.879 --> 00:14:45.480 is much more of a static 198 00:14:46.345 --> 00:14:47.385 ensemble 199 00:14:47.465 --> 00:14:49.625 that is preselected 200 00:14:50.105 --> 00:14:52.505 at conversation initialization. 201 00:14:52.505 --> 00:14:56.665 So you would have, like, okay. We have a big ensemble of a bunch of different models. 202 00:14:56.905 --> 00:14:58.025 As you mentioned, 203 00:14:58.265 --> 00:15:07.090 a ton of models leads to a complex system, so you have to select down to a very sparse set of these models to actually deploy at runtime 204 00:15:07.170 --> 00:15:09.570 other for a given conversation. 205 00:15:10.290 --> 00:15:10.770 Otherwise, 206 00:15:11.595 --> 00:15:23.035 complexity goes up, which means the cost and difficulty of orchestration goes up, which also means that the cost in general of running the system goes up. And we talked about cost being one of the big constraints earlier on. 207 00:15:23.435 --> 00:15:37.250 So the initial idea was to statically select, like, okay. We know for a given language and a given, like, social chat environment or interview call environment or other things like that, 208 00:15:37.490 --> 00:15:38.930 we can kind of test 209 00:15:39.195 --> 00:15:45.515 based on some ground truth labeled data sets which subsets of models work best on these environments 210 00:15:45.755 --> 00:15:51.275 and then just kind of do a lookup table and and statically look up which models to run and run those for the whole conversation. 211 00:15:51.920 --> 00:15:53.440 And what we found 212 00:15:53.680 --> 00:15:55.519 was that when that fails, 213 00:15:55.680 --> 00:15:57.920 it fails very, very, very poorly 214 00:15:58.000 --> 00:15:59.760 because these models are small. 215 00:16:00.000 --> 00:16:10.685 And, individually, they don't generalize well. The whole ensemble generalizes well, but, individually, the models don't generalize well. If you pick the wrong small model, you're going to get very bad results. 216 00:16:11.165 --> 00:16:15.805 So this led us to doing a ton of research into how do you cost effectively 217 00:16:16.204 --> 00:16:17.005 monitor 218 00:16:17.805 --> 00:16:19.805 the results of the models, 219 00:16:20.259 --> 00:16:26.180 How do you spot check whether or not the models are performing well? And how do you course correct 220 00:16:26.420 --> 00:16:36.795 and select different subsets of models if you know these aren't performing well? And this is the idea of the dynamic ensemble block, which is one of the cool pieces 221 00:16:37.035 --> 00:16:39.675 of the ELM architecture that we invented, 222 00:16:40.155 --> 00:16:43.275 where we realized that this is actually an optimization problem. 223 00:16:43.515 --> 00:16:46.555 Each model has a cost to run-in 224 00:16:46.555 --> 00:16:53.510 a given ensemble, like our ensemble of transcription models or an ensemble of emotion detection models, 225 00:16:54.070 --> 00:16:56.310 each has an expected 226 00:16:56.310 --> 00:16:57.269 accuracy 227 00:16:57.670 --> 00:16:58.709 given 228 00:16:58.870 --> 00:17:02.550 the distribution of audio that you're currently looking at. 229 00:17:03.110 --> 00:17:03.750 And 230 00:17:04.525 --> 00:17:07.804 they each have a degree of confidence 231 00:17:07.804 --> 00:17:10.524 that you can have about their outputs. 232 00:17:11.165 --> 00:17:15.404 And the optimization problem is that as you're running data through, 233 00:17:15.885 --> 00:17:21.360 you're getting results from the models that you selected in the Ensemble. 234 00:17:21.680 --> 00:17:26.559 You compare those results to each other to the rest of the context of the conversation 235 00:17:26.720 --> 00:17:29.280 to figure out how much you trust those results. 236 00:17:29.914 --> 00:17:35.994 And then the optimization problem is around the next part of the conversation, say, the next minute that you're analyzing. 237 00:17:36.235 --> 00:17:37.674 For that next minute, 238 00:17:37.914 --> 00:17:41.674 how do you optimize for accuracy while minimizing cost? 239 00:17:41.914 --> 00:17:44.840 And that transforms that whole

problem 240 00:17:45.000 --> 00:17:55.000 down into a very well known, like, multi armed bandit exploration versus exploitation problem so you can deploy a ton of well known machinery to solving that optimally. 241 00:17:55.355 --> 00:18:00.155 That was the really, really cool thing once we found out this surprise that 242 00:18:00.635 --> 00:18:03.034 when the models go off the rails, they go really bad. 243 00:18:03.515 --> 00:18:05.595 Yeah. As you're describing 244 00:18:05.595 --> 00:18:06.955 the architecture, 245 00:18:06.955 --> 00:18:09.995 it brings to mind two, I guess, complementary 246 00:18:11.059 --> 00:18:32.614 aspects of prior art where you're talking about the optimization question. It brings to mind a lot of the cost based optimizers that go into SQL query planners, particularly for these large scale out distributed database engines or things like the Trino engine that allows you to be able to do predicate pushdowns based on what is the actual underlying storage. 247 00:18:33.335 --> 00:18:37.654 And then in terms of being able to determine which particular 248 00:18:38.560 --> 00:18:53.440 model or module you want to actually route to, it brings up a lot of the conversation that's happening right now about how to actually implement these multi agent systems and how to manage communication and routing between those different agents to be able to achieve the desired outcome. 249 00:18:55.125 --> 00:18:59.765 Exactly. And I think that's what makes it so exciting and so 250 00:19:00.885 --> 00:19:02.565 I don't wanna say possible, 251 00:19:02.565 --> 00:19:04.565 you know, but at least tractable 252 00:19:04.805 --> 00:19:05.205 to 253 00:19:06.320 --> 00:19:07.120 develop 254 00:19:07.120 --> 00:19:09.760 this Ensemble listening model architecture 255 00:19:09.760 --> 00:19:14.000 as a, you know, what, 45 person start up, is that 256 00:19:14.960 --> 00:19:16.480 as you described, 257 00:19:16.640 --> 00:19:19.120 cost optimization problems are everywhere. 258 00:19:19.725 --> 00:19:27.325 And we're really, really like, as a species, we're really good at solving cost optimization problems. So as soon as we found out, oh, 259 00:19:27.725 --> 00:19:31.565 this really tricky part of analyzing conversations 260 00:19:31.565 --> 00:19:33.165 cost effectively and accurately 261 00:19:33.540 --> 00:19:37.220 can be transformed into a cost optimization problem. 262 00:19:37.460 --> 00:19:57.464 That was the eureka where we were like, okay. We have the tools to actually deal with this. And you can deal with that in terms of, like, you know, what is your overall cost budget, but how sensitive are you to that? You can add in things like how many GPUs or clusters of GPUs we have available. Because one of the exciting things about cost optimization around 263 00:19:57.784 --> 00:19:59.304 our small models 264 00:19:59.465 --> 00:20:08.809 based architecture here is that you don't need the biggest and best GPUs. You can run on we're running

on, you know, five, six year old GPUs sometimes because they're the most cost effective. 265 00:20:09.050 --> 00:20:12.650 So you can make the optimization problem even more complicated, 266 00:20:12.890 --> 00:20:18.234 but we have the optimizers to do that. And we've had them for decades, and they're really, really, really good. 267 00:20:19.355 --> 00:20:24.075 The the routing question, I think, is a really, really fun research problem. 268 00:20:24.315 --> 00:20:28.955 And the thing that makes multi agent systems writing so hard, in my opinion, 269 00:20:29.195 --> 00:20:32.390 is that agents are so flexible. 270 00:20:33.590 --> 00:20:37.990 Any agent could, in theory, do a wide variety of things, 271 00:20:38.470 --> 00:20:42.070 take in a wide variety of inputs, produce a wide variety of outputs. 272 00:20:42.544 --> 00:20:51.104 It's very flexible subcomponents of the system. So figuring out how to route to which one is optimal when many of them can do many of the tasks 273 00:20:51.265 --> 00:20:52.705 can be very tricky. 274 00:20:52.865 --> 00:20:54.385 This is an advantage to 275 00:20:54.940 --> 00:21:05.660 the ensemble of small models approach, at least when trying to solve the routing problem, is that we know exactly what all of these small models do. We know exactly what their cost is. They don't have a thinking budget. 276 00:21:06.300 --> 00:21:10.414 They don't really have a very flexible compute graph. 277 00:21:10.495 --> 00:21:12.654 Each of these individual models 278 00:21:12.815 --> 00:21:16.334 are relatively small and self contained and predictable. 279 00:21:16.335 --> 00:21:22.414 And we also, because they're small, have a very good understanding of their output distribution 280 00:21:22.850 --> 00:21:29.650 across different kinds of inputs that they might be given and what kind of accuracy ranges we can expect. 281 00:21:30.050 --> 00:21:33.010 So in terms of routing between the different models, 282 00:21:34.130 --> 00:21:37.330 making each of the individual nodes super predictable, 283 00:21:37.765 --> 00:21:42.004 as we have turns that routing problem into 284 00:21:42.405 --> 00:21:44.085 a pretty deterministic 285 00:21:44.085 --> 00:21:54.220 kind of thing as opposed to something with agents where it's very easy for the agents to, like, get themselves into a loop or something like that. Or we're a very much a, like, directed graph 286 00:21:54.300 --> 00:21:56.299 with some feedback loops, 287 00:21:56.380 --> 00:21:57.340 asynchronous, 288 00:21:57.340 --> 00:21:58.379 low priority 289 00:21:58.460 --> 00:22:04.780 to do things like optimize selecting which models. But that doesn't get in the way of the feed forward processing of the data. 290 00:22:05.965 --> 00:22:09.964 It also reminds me of at least my understanding 291 00:22:09.965 --> 00:22:12.364 of the GPT five 292 00:22:12.365 --> 00:22:13.245 category 293 00:22:13.245 --> 00:22:18.445 of model deployments where my understanding is that it's not actually one monolithic 294 00:22:18.445 --> 00:22:19.164 model. 295 00:22:19.404 --> 00:22:20.524 It's actually several 296 00:22:20.929 --> 00:22:26.370 sub models that are specialized for different cases, and they abstract that underneath 297 00:22:26.370 --> 00:22:41.475 a routing layer that determines which model to actually use under the covers. And I'm wondering what are some of the aspects of prior art that you were able to pull from either anything that they've published or any other similar product launches that have happened over the past several months. 298 00:22:42.275 --> 00:22:43.315 Absolutely. 299 00:22:43.955 --> 00:22:46.595 We've been working with ensembles 300 00:22:46.595 --> 00:22:48.115 of models for years. 301 00:22:48.610 --> 00:22:51.489 So the prior art that we pull from 302 00:22:51.809 --> 00:22:54.049 is mostly more from 303 00:22:54.370 --> 00:22:57.249 traditional mixture of experts models 304 00:22:57.570 --> 00:22:59.249 and traditional 305 00:22:59.570 --> 00:23:01.169 hybrid systems 306 00:23:01.250 --> 00:23:02.049 between 307 00:23:02.395 --> 00:23:04.235 rules based and 308 00:23:04.635 --> 00:23:07.514 neural network or black box kind of architectures 309 00:23:07.674 --> 00:23:09.674 and combining those things together. 310 00:23:09.915 --> 00:23:10.794 I also 311 00:23:11.275 --> 00:23:15.674 it was really exciting. Got to pull on some of my work from JPL 312 00:23:15.960 --> 00:23:16.759 where 313 00:23:17.640 --> 00:23:20.760 we were doing a lot of investigations 314 00:23:20.760 --> 00:23:25.320 into how can you deploy AI systems safely on spacecraft. 315 00:23:25.960 --> 00:23:26.600 And 316 00:23:26.919 --> 00:23:28.039 on a spacecraft, 317 00:23:29.205 --> 00:23:30.725 doing something unpredictable 318 00:23:30.725 --> 00:23:40.085 is often quite fatal. Right? If you accidentally turn your solar panels away from the sun, it's hard to recover from that if your battery goes dead. 319 00:23:40.565 --> 00:23:42.885 And if you accidentally turn off your radio, 320 00:23:43.549 --> 00:23:46.190 unless you have a fallback system, which they do, 321 00:23:47.230 --> 00:23:50.429 it's hard to tell the spacecraft to turn it back on again. 322 00:23:51.070 --> 00:23:55.070 So when you're deploying an AI system onto a spacecraft, 323 00:23:55.150 --> 00:23:57.870 the number one lesson is bounding 324 00:23:58.264 --> 00:23:59.464 the effects 325 00:23:59.465 --> 00:24:01.624 that that one model can have 326 00:24:01.865 --> 00:24:13.919 on the rest of your system. So no matter what the network says, like, hey. We need to go look at this rock next. It's the most important rock I've ever seen in my entire life. Ultimately, 327 00:24:13.919 --> 00:24:17.919 the outputs of that model can be perfect or they can be garbage, 328 00:24:18.159 --> 00:24:22.559 but they're all fed into more of a rules based, well understood, 329 00:24:23.280 --> 00:24:24.559 constrained 330 00:24:24.720 --> 00:24:25.520 optimization 331

00:24:25.520 --> 00:24:26.480 algorithm 332 00:24:26.785 --> 00:24:31.184 such that even if the AI encounters something completely unpredictable, 333 00:24:31.905 --> 00:24:34.145 you've already got the predetermined 334 00:24:34.145 --> 00:24:36.144 bounds to keep it from 335 00:24:36.545 --> 00:24:40.785 completely crashing everything else. So this actually related back to 336 00:24:41.539 --> 00:24:42.980 how we orchestrated 337 00:24:42.980 --> 00:24:48.739 some of our models because we're using those input distribution features 338 00:24:48.980 --> 00:24:52.099 and the results from some of our sub ensembles 339 00:24:52.100 --> 00:24:56.100 to figure out, are we doing a good job at understanding this conversation? 340 00:24:56.764 --> 00:25:03.244 But you might see the problem with that of we're using our models to figure out if our models are doing a good job. 341 00:25:03.725 --> 00:25:04.364 And 342 00:25:04.845 --> 00:25:07.084 if they're confused but confident, 343 00:25:07.164 --> 00:25:09.484 you can mess up the entire conversation. 344 00:25:09.485 --> 00:25:12.860 And some of the stuff we're doing is actually pretty, 345 00:25:13.260 --> 00:25:19.660 like, high risk, high high reward kind of stuff. You know? We're deployed in social game stopping harassment. 346 00:25:19.900 --> 00:25:29.315 You accidentally banned the victim instead of the aggressor in our harassment situation. That can cause a lot of real harm to a lot of real people. We look for extremism, 347 00:25:29.315 --> 00:25:46.190 child safety incidents, other things like that. It's like, it can be relatively high cost if you get the stuff wrong. Of course, you have other safeguards on it, but it's still an important problem to get right. So if your models are just confident in their own abilities and you don't have a check on them, you can run into serious problems. 348 00:25:46.670 --> 00:25:55.615 So that's why you were asking a little bit about off the shelf models earlier. There's actually a place for the large, highly general 349 00:25:56.415 --> 00:25:57.215 models 350 00:25:57.375 --> 00:26:05.455 in an ELM framework as kind of a supervisor or a check. So we actually have a suite of models in our different ensembles 351 00:26:05.720 --> 00:26:08.760 that are very high capability, 352 00:26:08.760 --> 00:26:10.600 high cost generalists. 353 00:26:10.920 --> 00:26:14.600 And we will occasionally and this is, again, an optimization problem. 354 00:26:15.080 --> 00:26:17.800 We will occasionally run them to check 355 00:26:18.465 --> 00:26:19.665 the outputs 356 00:26:19.665 --> 00:26:22.225 of our smaller, more specialized models 357 00:26:22.385 --> 00:26:28.065 and make sure that they agree to a reasonable degree. And you can't run it too often or it gets expensive, 358 00:26:28.145 --> 00:26:34.860 but you can't run that not at all. Otherwise, you fall into this confidence problem. So you have this kind of checking feedback loop, 359 00:26:35.100 --> 00:26:39.179

and that's how you constrain your overall system 360 00:26:39.340 --> 00:26:49.085 to do a pretty good job. But, again, a lot of that priority is coming from stuff that people were working on one, two decades ago. Some of these ideas, they're not super, super new. 361 00:26:50.045 --> 00:26:51.005 And then 362 00:26:51.245 --> 00:26:56.125 generalizing beyond the voice use case, what are some of the 363 00:26:56.980 --> 00:26:58.980 environments or situations 364 00:26:58.980 --> 00:27:03.379 that you would advocate for teams to explore, 365 00:27:03.779 --> 00:27:05.299 either building their own 366 00:27:05.700 --> 00:27:14.735 Ensemble approach or if there are any other off the shelf frameworks that allow for building this style of Ensemble model capability, 367 00:27:14.735 --> 00:27:15.455 again, 368 00:27:16.095 --> 00:27:17.534 not including the 369 00:27:17.775 --> 00:27:21.775 multi agent use case because of the inherent flexibility involved. 370 00:27:24.370 --> 00:27:25.330 Great question. 371 00:27:26.450 --> 00:27:27.169 The 372 00:27:27.570 --> 00:27:28.450 applicability 373 00:27:28.450 --> 00:27:30.050 of Ensemble model 374 00:27:30.530 --> 00:27:32.130 well, first, I would say 375 00:27:33.090 --> 00:27:35.169 anybody working in a 376 00:27:35.410 --> 00:27:36.370 regime 377 00:27:36.450 --> 00:27:37.970 where you are going to do 378 00:27:38.605 --> 00:27:39.884 a similar 379 00:27:40.044 --> 00:27:41.004 task 380 00:27:41.085 --> 00:27:42.924 many, many times over, 381 00:27:43.804 --> 00:27:45.804 I would recommend looking into 382 00:27:46.044 --> 00:27:48.044 small models in general 383 00:27:48.205 --> 00:27:49.485 and ensembles 384 00:27:49.485 --> 00:27:50.684 of small models 385 00:27:50.870 --> 00:27:54.470 in the cases where you need a lot of general flexibility, 386 00:27:55.110 --> 00:27:57.830 but all around a similar kind of task. 387 00:27:58.070 --> 00:28:00.470 So the reason why ELMs were so 388 00:28:01.270 --> 00:28:03.750 directly applicable to voice conversations 389 00:28:03.830 --> 00:28:05.430 is because voice conversations 390 00:28:05.985 --> 00:28:07.904 have a ton of shared properties. 391 00:28:08.145 --> 00:28:20.705 You're always wanna gonna wanna get a transcript. You're always gonna wanna figure out who are these people that are talking. You're always gonna wanna figure out what emotion are they using, what accent are they using. Give me some basic demographics so I can understand the context. 392 00:28:21.260 --> 00:28:24.300 Always gonna wanna understand the topics of discussion, 393 00:28:24.540 --> 00:28:31.420 the intents of the people involved, like what are these people trying to achieve when they're discussing in order to understand that conversation. 394 00:28:31.660 --> 00:28:38.504 And there are tens of billions of conversations that happen on this planet every single day, but they all fall into that structure. 395 00:28:38.825 --> 00:28:40.744 And if you contrast with something like 396 00:28:41.145 --> 00:28:42.904 just a large language model, 397 00:28:43.304 --> 00:28:51.959 there's none of that structure of how must I analyze this data built into it. Even though the space of conversations, 398 00:28:52.039 --> 00:28:54.599 all voice conversations is very, very general, 399 00:28:55.559 --> 00:28:58.599 it's still a much, much more constrained 400 00:28:59.240 --> 00:29:13.475 space of what kinds of things do you want to do to analyze this data than just say an arbitrary stream of text or an arbitrary stream of data, which is what the big foundation models are applicable to. So if you can identify that structure 401 00:29:13.635 --> 00:29:14.674 and say, 402 00:29:15.635 --> 00:29:22.710 I'm gonna be tackling a very diverse set of data, but they all need to be analyzed in similar ways, 403 00:29:23.669 --> 00:29:24.309 then 404 00:29:25.110 --> 00:29:30.789 deploying small models lets you take advantage of the cost efficiencies of those small models. 405 00:29:30.870 --> 00:29:37.404 You can break the problem down into several pieces, like let's analyze emotion. Let's analyze transcription. 406 00:29:37.485 --> 00:29:42.684 Let's do these other things. And if you still want to maintain the flexibility 407 00:29:43.005 --> 00:29:44.285 and generality 408 00:29:44.285 --> 00:29:47.245 of applying to a bunch of different distributions, 409 00:29:47.405 --> 00:29:48.845 like different kinds of conversations, 410 00:29:49.600 --> 00:29:52.399 then that's where you'll want ensembles of those models. 411 00:29:52.720 --> 00:30:00.479 So that's that's where I would suggest looking into using ensemble models instead of just going straight for a big foundation model. 412 00:30:00.960 --> 00:30:01.919 And then 413 00:30:02.295 --> 00:30:03.015 the 414 00:30:03.415 --> 00:30:13.735 large open question that I see a lot of people trying to deal with right now as a broader set of people are entering this ML and AI ecosystem 415 00:30:13.895 --> 00:30:21.549 is how do I actually verify that anything that I changed didn't just completely destroy everything that I've been working on for the past six months? 416 00:30:21.790 --> 00:30:27.390 And so one of the approaches that has gained a lot of popularity is the idea of evaluations. 417 00:30:28.110 --> 00:30:31.470 There are also the observability aspects 418 00:30:31.470 --> 00:30:32.270 of doing 419 00:30:32.794 --> 00:30:38.634 real time analysis of model performance and outputs to determine if that is actually 420 00:30:39.034 --> 00:30:40.634 producing the desired 421 00:30:41.115 --> 00:30:41.754 result. 422 00:30:42.235 --> 00:30:43.355 And as 423 00:30:43.355 --> 00:30:45.434 you add this Ensemble 424 00:30:45.434 --> 00:30:47.275 aspect to the overall 425 00:30:48.450 --> 00:30:49.889 request path, 426 00:30:50.049 --> 00:31:02.130 what are some of the ways that that complicates that question of how do I actually validate the overall functionality of the system as well as being

able to do more isolated validation of those sub models? 427 00:31:03.395 --> 00:31:04.514 It gives you 428 00:31:04.755 --> 00:31:06.994 added complexity you need to handle, 429 00:31:07.395 --> 00:31:11.634 but it also gives you a lot of power and tools to solve that complexity. 430 00:31:12.115 --> 00:31:13.874 So the added complexity, 431 00:31:14.115 --> 00:31:24.660 as you might guess, is the extraordinary number of different routes through the system that your data can take. So if I have some small models that are specialized to 432 00:31:24.900 --> 00:31:26.340 high quality audio 433 00:31:26.660 --> 00:31:27.380 and 434 00:31:27.685 --> 00:31:32.565 my ensemble picks a model specialized to low quality audio instead, 435 00:31:33.045 --> 00:31:34.565 the rest of the system, 436 00:31:34.805 --> 00:31:37.684 all bets are off, anything downstream from that. 437 00:31:37.925 --> 00:31:42.005 And because you're doing this dynamic optimization that I was talking about earlier, 438 00:31:42.325 --> 00:31:44.480 where the results of models 439 00:31:44.480 --> 00:31:47.519 running earlier in the conversation feed back 440 00:31:48.080 --> 00:31:52.719 into the system to help it pick which models to run-in the future. 441 00:31:53.759 --> 00:31:55.199 If you've got garbage 442 00:31:55.200 --> 00:31:58.000 coming out of your models early in the conversation, 443 00:31:58.505 --> 00:32:04.184 you can poison that decision making in those choices for later on in the conversation. 444 00:32:04.985 --> 00:32:08.985 So it makes it really, really hard because of the sheer number of paths 445 00:32:08.985 --> 00:32:11.065 the data can take through your Ensemble. 446 00:32:11.750 --> 00:32:16.309 And the flip side is the Ensemble approach also gives you a lot of tools. 447 00:32:16.629 --> 00:32:17.269 So 448 00:32:17.750 --> 00:32:21.669 part one is the structure that I was talking about earlier. 449 00:32:21.830 --> 00:32:23.750 So we know that a conversation 450 00:32:24.705 --> 00:32:28.864 has these different pieces of structure that we're looking for. 451 00:32:29.105 --> 00:32:30.465 And all of the 452 00:32:30.705 --> 00:32:32.144 structured outputs, 453 00:32:32.145 --> 00:32:36.465 like for example, the text content and emotion 454 00:32:36.705 --> 00:32:40.470 content of what I'm saying are related. 455 00:32:40.870 --> 00:32:42.230 They're not identical. 456 00:32:42.310 --> 00:32:47.669 One's just not a strict function of the other. Otherwise, you wouldn't need the multiple models, but they're related. 457 00:32:48.070 --> 00:32:51.590 And so for observability and debugging purposes, 458 00:32:51.830 --> 00:32:54.764 one thing you can do, which we in fact do, 459 00:32:55.005 --> 00:32:57.885 is check sentiment extracted 460 00:32:57.885 --> 00:32:59.085 from text 461 00:32:59.245 --> 00:33:01.325 and check emotional tone 462 00:33:01.325 --> 00:33:07.485 and see if they're matching up relatively often. It's a statistical approach, not a deterministic 463 00:33:07.485 --> 00:33:08.605 debugging tactic, 464 00:33:09.000 --> 00:33:10.760 But you expect 465 00:33:11.400 --> 00:33:15.399 different outputs from different pieces of your ensemble 466 00:33:15.400 --> 00:33:17.480 to correlate in certain ways. 467 00:33:17.800 --> 00:33:21.800 And so if I identify that for certain kinds of conversations, 468 00:33:22.345 --> 00:33:24.825 like, let's say, low quality audio calls, 469 00:33:25.545 --> 00:33:30.985 no matter what text is being output, my emotion model always thinks people are angry, 470 00:33:31.545 --> 00:33:35.065 then I know that something's likely to be wrong. 471 00:33:35.550 --> 00:33:37.710 And in certain situations, 472 00:33:37.710 --> 00:33:38.350 again, 473 00:33:38.910 --> 00:33:43.390 the emotion and the text might not match. But if they're completely uncorrelated 474 00:33:43.390 --> 00:33:46.190 over a long a large amount of data, 475 00:33:46.510 --> 00:33:48.830 then that gives you a pointer for what to look into. 476 00:33:50.485 --> 00:33:56.004 And digging now into some of the specifics of these voice models, 477 00:33:56.165 --> 00:34:03.445 I have a generalized understanding of the ways that large language models work in terms of these transformer architectures, 478 00:34:03.679 --> 00:34:11.920 the attention mechanisms that are used, the variance there, the idea of these mixture of experts models for the case where you want to be able to incorporate more reasoning. 479 00:34:12.160 --> 00:34:24.795 And I'm just curious how much of that is analogous in this case of audio and particularly voice focused models, and how much of the actual underlying architecture needs to shift because of the modality 480 00:34:24.795 --> 00:34:27.115 and the specifics of speech? 481 00:34:28.075 --> 00:34:30.155 It's actually very complementary. 482 00:34:30.474 --> 00:34:31.115 So 483 00:34:31.515 --> 00:34:32.635 when you're tackling 484 00:34:33.430 --> 00:34:34.710 an audio 485 00:34:34.710 --> 00:34:35.510 analysis 486 00:34:35.510 --> 00:34:36.390 problem, 487 00:34:36.950 --> 00:34:38.230 you're looking at 488 00:34:38.790 --> 00:34:41.430 both how do I get the right 489 00:34:41.910 --> 00:34:42.710 results 490 00:34:42.710 --> 00:34:45.030 out of the underlying 491 00:34:45.030 --> 00:34:45.510 data? 492 00:34:45.965 --> 00:34:52.045 And then how do I interpret those to make broader sense of the conversation and do something useful with it? 493 00:34:52.685 --> 00:34:55.485 The techniques that you're talking about around attention 494 00:34:55.885 --> 00:34:59.165 and all sorts of other capabilities 495 00:34:59.165 --> 00:35:00.365 for these models, 496 00:35:01.390 --> 00:35:08.670 audio is a very long time scale kind of data, right? Like, you've got a 48 kilohertz audio signal. 497 00:35:09.070 --> 00:35:09.950 That's 498 00:35:09.950 --> 00:35:30.800 a ton of different data points for just a second where you might utter, like, you know, what, one or a couple

of different phones make a couple of different sounds. So you're you're mapping this very, very, very high dimensional audio signal down to, even if you're doing emotion and all the nuance and everything, a much, much, much lower dimensionality 499 00:35:30.800 --> 00:35:31.600 space. 500 00:35:32.800 --> 00:35:34.880 Technologies like transformers 501 00:35:34.880 --> 00:35:44.175 and even older school stuff from the twenty tens, you know, earlier twenty tens, like dilated convolutions and wave nets and things like that, All of these different strategies 502 00:35:44.815 --> 00:35:46.415 and architectures 503 00:35:46.815 --> 00:35:47.775 perform 504 00:35:47.934 --> 00:35:57.295 very, very well on these longer sets of data. And a lot of the research that's been going into these very large models has been increasing their context windows 505 00:35:57.510 --> 00:36:00.550 and how to efficiently handle and select 506 00:36:00.950 --> 00:36:04.390 in extremely large data contexts 507 00:36:04.550 --> 00:36:07.270 what parts of the data to pay attention to. 508 00:36:07.510 --> 00:36:09.750 All of that is applicable in extracting 509 00:36:10.425 --> 00:36:14.825 different pieces of information accurately from these audio signals. 510 00:36:15.385 --> 00:36:16.025 But 511 00:36:16.345 --> 00:36:21.225 the next layer up is really where a lot of the Ensemble 512 00:36:21.225 --> 00:36:22.025 technology 513 00:36:22.025 --> 00:36:24.185 that I've been talking about comes in. 514 00:36:24.425 --> 00:36:24.985 So 515 00:36:25.450 --> 00:36:28.090 when you're building an ensemble model, 516 00:36:28.970 --> 00:36:30.730 if you can slot in 517 00:36:31.130 --> 00:36:33.770 a better emotion detection model, 518 00:36:33.850 --> 00:36:42.535 the whole ensemble's gonna work better. And the better emotion detection model might be replacing, say, a standard dilated convolution network 519 00:36:42.535 --> 00:36:45.975 with a network that employs an attention mechanism 520 00:36:46.135 --> 00:36:46.935 or 521 00:36:47.174 --> 00:36:48.695 some other thing like that. 522 00:36:51.174 --> 00:36:58.260 And that you have to change in terms of the Ensemble model is your priors on the accuracy of that model, assuming it gets more accurate, 523 00:36:58.420 --> 00:37:07.859 and your cost parameters of the model, assuming it might cost more to run. And then once you've changed that, the rest of the Ensemble just works as it normally did. So they're very complementary. 524 00:37:09.444 --> 00:37:13.125 Speaking to that longer horizon and the longer 525 00:37:13.684 --> 00:37:15.845 window of attention that's necessary, 526 00:37:15.924 --> 00:37:28.200 it also brings up the question of another trend that's becoming very popular in agentic contexts is the idea of agentic memory and being able to store and retrieve certain aspects of information 527 00:37:28.440 --> 00:37:29.400 for later 528 00:37:29.880 --> 00:37:30.680 usage. 529 00:37:30.760 --> 00:37:33.080 And particularly if you're dealing with 530 00:37:33.480 --> 00:37:34.280 a 531 00:37:34.359 --> 00:37:37.125 minutes to hours long conversation 532 00:37:37.125 --> 00:37:38.885 or even monologue. 533 00:37:39.125 --> 00:37:54.820 Obviously, there are cases where you're going to overflow the available context window for even an ensemble model, and I'm just curious how you take advantage of some of these storage and retrieval questions for being able to do that 534 00:37:55.060 --> 00:37:55.700 more 535 00:37:55.940 --> 00:37:57.140 long horizon 536 00:37:57.140 --> 00:37:58.020 analysis 537 00:37:58.020 --> 00:37:59.860 of this high 538 00:38:00.100 --> 00:38:01.620 information density 539 00:38:01.860 --> 00:38:02.420 medium. 540 00:38:03.795 --> 00:38:06.355 This is where knowing the structure 541 00:38:06.435 --> 00:38:26.950 of your data is so important. It's like what's what's old is new again. Right? It's always been in machine learning, like, understand what data your models are going to operate on. Bring in domain expertise as well as do backprop through the neural net. Right? For something like a conversation, you could have a five hour long conversation, 542 00:38:27.109 --> 00:38:32.595 but you have priors, very good priors on what data is actually important 543 00:38:32.915 --> 00:38:34.515 to that conversation. 544 00:38:34.675 --> 00:38:37.795 Right? You wanna know who are these people, 545 00:38:37.955 --> 00:38:43.155 what are their intents right now, what have their intents and goals been through the course of the conversation, 546 00:38:43.670 --> 00:38:53.270 How are they acting towards each other? What's the distribution of emotions? Like, for example, I sound pretty excited all the time. So when I deviate from that norm, 547 00:38:53.510 --> 00:38:55.590 then that's information that's important. 548 00:38:56.505 --> 00:39:00.345 By digging into, like, how do humans understand conversations, 549 00:39:00.345 --> 00:39:03.145 like, what kinds of tasks are you trying to achieve 550 00:39:03.385 --> 00:39:05.385 by analyzing these conversations, 551 00:39:05.464 --> 00:39:06.825 you can actually pre 552 00:39:07.145 --> 00:39:09.065 kind of preregister, 553 00:39:09.065 --> 00:39:10.185 precreate, 554 00:39:10.505 --> 00:39:11.464 preallocate 555 00:39:11.980 --> 00:39:16.940 the the the pieces of information that are going to be broadly, 556 00:39:17.740 --> 00:39:19.020 generally relevant 557 00:39:19.100 --> 00:39:19.900 to 558 00:39:19.900 --> 00:39:39.570 the conversation. If I'm gonna make if I'm if I'm gonna make a super, super simplified example, it's like, you know, pinning the header or pinning the rows in an Excel spreadsheet or something like that. It's like, yeah, you can kinda scroll back and look at a sentence that happened, like, three hours ago or a tab that's, you know or like a cell that's, like, you know, 200 559 00:39:39.570 --> 00:39:50.930 rows over. But you always know that some of this data is going to be

important and should be front and center and visible to your models at all times. So instead of for our use case, so instead of 560 00:39:51.250 --> 00:39:53.090 trying to have some extremely 561 00:39:53.170 --> 00:39:54.050 very flexible 562 00:39:54.725 --> 00:39:55.605 dynamic 563 00:39:56.005 --> 00:40:01.845 storage with absolutely no priors or structure on it, we have a ton of preallocated 564 00:40:01.845 --> 00:40:04.965 structure around what's a running summary of the conversation. 565 00:40:05.045 --> 00:40:06.405 Who are the participants? 566 00:40:06.485 --> 00:40:11.280 What do they sound like? What's their general distribution of emotions 567 00:40:11.440 --> 00:40:13.200 across the conversation? 568 00:40:13.200 --> 00:40:15.760 All of these different things. And those are preregistered 569 00:40:15.760 --> 00:40:16.880 as kind of a 570 00:40:17.920 --> 00:40:22.480 fixed memory block that serves as the starting point for context 571 00:40:22.875 --> 00:40:26.155 for all of the individual models in the ensemble. 572 00:40:26.555 --> 00:40:31.595 And of course, some of the models can add more context and go in dynamically. 573 00:40:31.755 --> 00:40:43.020 And of course, those things like the summary and the participant roles and the behaviors they're exhibiting, they change over time. But we always know that those pieces of information are going to be relevant. So it massively, 574 00:40:43.020 --> 00:40:45.420 massively, massively simplifies 575 00:40:45.500 --> 00:40:47.260 the memory problem 576 00:40:47.500 --> 00:40:48.700 for this system, 577 00:40:48.860 --> 00:40:49.980 and it also 578 00:40:49.980 --> 00:40:50.860 reduces 579 00:40:50.860 --> 00:40:51.260 the 580 00:40:51.785 --> 00:40:54.265 ability to make weird errors. 581 00:40:54.265 --> 00:40:56.985 Right? If you're thinking about a system with memory, 582 00:40:57.065 --> 00:40:59.785 the system might be doing something really, really stupid. 583 00:41:00.025 --> 00:41:13.190 And you're like, why is it doing this? And it's because, oh, it didn't have the right information in memory. Why didn't it have the right information in memory? Now you're going down a very long debugging path that is going to be very complicated. 584 00:41:13.589 --> 00:41:14.150 And 585 00:41:14.550 --> 00:41:15.190 by 586 00:41:16.069 --> 00:41:21.785 changing how the system stores things in memory, you could change a ton of other behaviors without knowing it. 587 00:41:22.184 --> 00:41:23.145 Preregistering 588 00:41:23.145 --> 00:41:24.265 the structure 589 00:41:24.505 --> 00:41:31.944 of the memory or even just a subset of the memory ahead of time means that your models will always have the relevant information 590 00:41:32.400 --> 00:41:38.160 and reduces a ton of the errors that these very, very, very dynamics and flexible systems can have. 591 00:41:39.359 --> 00:41:40.880 As you have been 592 00:41:41.920 --> 00:41:44.000 exploring this architecture, 593 00:41:44.079 --> 00:41:46.160 developing your own implementation 594 00:41:46.160 --> 00:41:47.119 of it, 595 00:41:47.440 --> 00:41:47.599 and 596 00:41:48.615 --> 00:41:51.575 keeping an eye on just the overall ecosystem 597 00:41:51.575 --> 00:41:53.415 of how people are 598 00:41:53.575 --> 00:41:54.695 dealing with 599 00:41:55.015 --> 00:41:56.775 these model deployments, 600 00:41:56.855 --> 00:41:58.215 model optimization. 601 00:41:58.615 --> 00:42:00.215 What are some of the complexities 602 00:42:00.215 --> 00:42:03.895 that you as an engineering team have had to 603 00:42:04.250 --> 00:42:10.170 come to terms with and address to be able to achieve this objective and some of the 604 00:42:10.569 --> 00:42:12.570 useful learnings that you 605 00:42:13.609 --> 00:42:17.690 developed in the process of actually getting to delivering this as a feature set? 606 00:42:19.705 --> 00:42:22.585 Probably the biggest and most obvious 607 00:42:22.905 --> 00:42:25.705 complexity that we've had to deal with and overcome 608 00:42:26.025 --> 00:42:29.224 has been the fact that this Ensemble 609 00:42:29.224 --> 00:42:31.945 listening model architecture by necessity 610 00:42:32.025 --> 00:42:33.785 is a very distributed system. 611 00:42:34.329 --> 00:42:42.810 In general, if you're going to run a neural network on a lot of data, you want to load the weights into the GPU and keep them there. You don't want to be swapping models in and out. 612 00:42:43.290 --> 00:42:45.050 If you're going to do that for 613 00:42:45.690 --> 00:42:51.005 a possibility of over one one hundred different models that you might run on any given piece of data, 614 00:42:51.484 --> 00:42:54.845 some models can be shared in a single GPU. 615 00:42:54.925 --> 00:43:00.285 But you're not going to fit all 100 models into one GPU or even a couple GPUs. 616 00:43:00.525 --> 00:43:01.885 So you're starting to look at, 617 00:43:02.810 --> 00:43:04.730 in order to run inference 618 00:43:04.810 --> 00:43:06.570 on any individual 619 00:43:06.570 --> 00:43:07.530 data point, 620 00:43:08.010 --> 00:43:12.810 I'm going to need to route that data to a small subset 621 00:43:12.810 --> 00:43:13.450 of 622 00:43:13.610 --> 00:43:15.610 a bunch of different machines 623 00:43:16.025 --> 00:43:18.905 with a bunch of different models loaded onto them. 624 00:43:19.305 --> 00:43:21.385 So it very, very quickly becomes 625 00:43:21.865 --> 00:43:24.105 a distributed computing problem. 626 00:43:24.905 --> 00:43:27.065 When tackling a distributed problem, 627 00:43:27.305 --> 00:43:33.300 the distributed problem becomes a lot harder if you need strong consistency guarantees. 628 00:43:33.859 --> 00:43:37.380 One of the really nice things about modern machine learning 629 00:43:37.460 --> 00:43:40.740 is that a lot of models can be very forgiving 630 00:43:40.819 --> 00:43:47.525 when given errors or gaps in data. And we exploited that very, very heavily in the ELM architecture. 631 00:43:47.845 -->

00:43:49.125 So instead of 632 00:43:49.445 --> 00:43:50.725 asserting that 633 00:43:51.045 --> 00:43:55.125 all of the different models in the ensemble must have their data available 634 00:43:55.770 --> 00:44:00.890 by some sort of time or worse, locking until every single data point has come in, 635 00:44:02.250 --> 00:44:05.210 we make sure that our data flow 636 00:44:05.610 --> 00:44:06.650 goes from 637 00:44:07.450 --> 00:44:10.970 less flexible models to more flexible models 638 00:44:11.155 --> 00:44:13.395 in a pretty strict hierarchy, 639 00:44:13.395 --> 00:44:15.715 at least for that feedforward processing. 640 00:44:16.355 --> 00:44:16.995 So 641 00:44:17.315 --> 00:44:19.315 if our less sophisticated 642 00:44:19.635 --> 00:44:21.635 models, smaller models, 643 00:44:22.355 --> 00:44:24.115 don't all return results 644 00:44:24.115 --> 00:44:26.435 in a required amount of time, 645 00:44:27.279 --> 00:44:31.040 The good news is that data is flowing into a more flexible 646 00:44:31.200 --> 00:44:32.480 and more forgiving, 647 00:44:32.480 --> 00:44:34.480 in some ways, model. 648 00:44:34.799 --> 00:44:38.160 And we have a much greater ability 649 00:44:38.400 --> 00:44:46.735 to account for and deal with missing data than we would if we were feeding into another super tiny, super specialized 650 00:44:47.535 --> 00:44:49.695 model in the Ensemble. 651 00:44:50.175 --> 00:44:55.615 So that's been a very, very strong rule of thumb for us. Always flow data from 652 00:44:56.070 --> 00:44:57.750 less sophisticated, 653 00:44:57.830 --> 00:44:59.030 smaller models 654 00:44:59.350 --> 00:45:00.150 into 655 00:45:00.150 --> 00:45:02.710 more flexible, more general models. 656 00:45:03.270 --> 00:45:07.590 And that helps us account for these distributed computing errors 657 00:45:07.590 --> 00:45:10.150 and still do a good job generally, 658 00:45:10.470 --> 00:45:13.305 even in times of high model dropout. 659 00:45:13.945 --> 00:45:14.585 And 660 00:45:15.385 --> 00:45:17.065 as far as the 661 00:45:17.305 --> 00:45:21.305 cost question of being able to deliver this functionality 662 00:45:21.465 --> 00:45:23.145 more economically, 663 00:45:23.465 --> 00:45:26.345 what are some of the ways that you're tracking the 664 00:45:26.510 --> 00:45:39.869 real world impact of that, particularly given the distributed nature of the actual cost as well as the increased operational overhead and potentially headcount needed to be able to keep the system running? 665 00:45:42.585 --> 00:45:43.465 Good question. 666 00:45:44.105 --> 00:45:46.985 In terms of the additional complexity, 667 00:45:47.785 --> 00:45:49.145 I would say that 668 00:45:49.625 --> 00:45:53.785 most engineering systems have complexity in a variety of different places. 669 00:45:54.345 --> 00:45:54.985 And 670 00:45:55.460 --> 00:45:56.180 by 671 00:45:56.339 --> 00:45:57.380 specifically 672 00:45:57.380 --> 00:46:01.780 focusing on orchestrating a bunch of different models

together, 673 00:46:02.420 --> 00:46:03.460 we're solving 674 00:46:04.500 --> 00:46:07.780 a ton of these different orchestration problems, 675 00:46:07.859 --> 00:46:10.180 but they're all very similar problems. 676 00:46:11.165 --> 00:46:11.885 So 677 00:46:13.165 --> 00:46:18.125 it actually pays off for us to have a small number of really, really 678 00:46:19.165 --> 00:46:20.925 deep knowledge experts 679 00:46:21.165 --> 00:46:21.885 in 680 00:46:22.285 --> 00:46:24.045 orchestrating systems of models 681 00:46:24.300 --> 00:46:27.260 and then building reusable tools 682 00:46:27.580 --> 00:46:28.780 to manage 683 00:46:28.940 --> 00:46:29.660 those 684 00:46:29.740 --> 00:46:31.500 different ensembles. 685 00:46:31.820 --> 00:46:33.500 Given that we can do that, 686 00:46:33.900 --> 00:46:39.660 we end up not having a huge amount of what I would call additional overhead 687 00:46:39.325 --> 00:46:40.765 Because the alternative 688 00:46:41.085 --> 00:46:46.125 is if you're working with a very, very, very big foundational model, 689 00:46:47.165 --> 00:46:49.485 now you're trying to shard 690 00:46:49.805 --> 00:46:52.605 that model across a bunch of different GPUs. 691 00:46:53.319 --> 00:46:56.359 And if you're trying to do data communication 692 00:46:56.440 --> 00:46:59.720 between different parts of, say, one neural network, 693 00:47:00.280 --> 00:47:01.560 that's a situation 694 00:47:01.880 --> 00:47:02.520 where 695 00:47:03.319 --> 00:47:08.599 changes in data or dropout or distributed system failures can be very catastrophic. 696 00:47:09.215 --> 00:47:15.535 It's very, very unpredictable what a neural network will do if you just cut out a little chunk of its weights. 697 00:47:16.335 --> 00:47:23.135 And so you have to solve those distributed problems if you're building big, like very big foundation models 698 00:47:23.589 --> 00:47:26.230 in strong consistency ways. 699 00:47:26.470 --> 00:47:30.230 And you have to deploy a ton of really, really capable hardware 700 00:47:30.390 --> 00:47:32.710 and network them in extremely 701 00:47:32.710 --> 00:47:34.630 low latency setups 702 00:47:34.869 --> 00:47:37.270 in order to do that kind of inference 703 00:47:37.385 --> 00:47:40.585 in a way that results in a good customer experience. 704 00:47:41.305 --> 00:47:46.665 By chunking down all of our models to be able to run on single machines or single GPUs, 705 00:47:47.224 --> 00:47:49.865 we actually buy a ton of flexibility. 706 00:47:50.025 --> 00:47:51.705 And so I wouldn't say it's 707 00:47:51.944 --> 00:47:52.744 more complex. 708 00:47:53.360 --> 00:47:55.600 It's a different kind of complexity, 709 00:47:55.840 --> 00:47:58.400 and in some ways, actually more forgiving 710 00:47:58.400 --> 00:48:03.840 than trying to be a model provider that just holds a really big foundational model. 711 00:48:04.560 --> 00:48:06.720 And then 712 00:48:06.065 --> 00:48:08.065 in terms of the 713 00:48:08.945 --> 00:48:11.665 evolution of

this style of architecture, 714 00:48:12.065 --> 00:48:14.305 what are some of the impacts 715 00:48:14.305 --> 00:48:15.825 that you foresee 716 00:48:16.465 --> 00:48:19.665 these capabilities having on the broader industry 717 00:48:19.825 --> 00:48:25.340 and the challenges that we're already running into as far as just brute force scaling and 718 00:48:25.500 --> 00:48:32.780 increasing the number of parameters versus these more small specialized models, and then also how that factors into 719 00:48:32.940 --> 00:48:45.215 the push to the edge of having this be a more naturally distributed system where you can have some of these very small models on the edge to determine whether to even propagate that request downstream. 720 00:48:46.575 --> 00:48:48.655 I think that Ensemble 721 00:48:49.135 --> 00:48:49.935 models 722 00:48:50.410 --> 00:48:52.010 are going to 723 00:48:52.250 --> 00:48:53.290 replace 724 00:48:53.370 --> 00:48:56.730 a lot of existing foundation model use cases 725 00:48:56.970 --> 00:48:58.410 and extend 726 00:48:58.730 --> 00:49:02.010 a lot of other foundation model capabilities. 727 00:49:02.490 --> 00:49:05.050 So in the replacing existing use cases, 728 00:49:07.665 --> 00:49:09.505 I see Ensemble models 729 00:49:09.585 --> 00:49:11.425 as being the right 730 00:49:11.745 --> 00:49:12.545 tool 731 00:49:12.865 --> 00:49:13.985 to solve 732 00:49:14.225 --> 00:49:15.185 repeated 733 00:49:15.185 --> 00:49:16.705 structured problems. 734 00:49:17.105 --> 00:49:19.985 Again, even as general as conversation analysis, 735 00:49:20.640 --> 00:49:25.119 that is a very general problem statement but much, much narrower than 736 00:49:25.359 --> 00:49:26.160 arbitrary 737 00:49:26.160 --> 00:49:28.160 operations on streams of data. 738 00:49:28.640 --> 00:49:32.160 So if you're going to do a lot of conversation analysis, 739 00:49:32.775 --> 00:49:35.974 it can pay off in terms of accuracy, 740 00:49:35.974 --> 00:49:37.494 determinism, and cost 741 00:49:37.734 --> 00:49:38.535 to 742 00:49:38.855 --> 00:49:42.455 build out an ensemble to take advantage of that structure. 743 00:49:43.174 --> 00:49:45.175 I think of the parallel as 744 00:49:45.815 --> 00:49:47.575 if you have a novel task, 745 00:49:48.359 --> 00:49:53.320 say, this is the first time I ever wanna just write down a transcript of this podcast. 746 00:49:53.720 --> 00:49:55.800 You just go ask somebody to do it. 747 00:49:56.279 --> 00:49:59.079 But if you need to transcribe 10,000 podcasts, 748 00:49:59.400 --> 00:50:02.519 you don't go ask a human to transcribe 10,000 podcasts. 749 00:50:02.825 --> 00:50:05.545 You ask a specially constructed 750 00:50:06.745 --> 00:50:07.545 system 751 00:50:07.705 --> 00:50:09.945 to do that transcription 752 00:50:09.945 --> 00:50:13.465 because it's going to do that same work deterministically 753 00:50:13.705 --> 00:50:16.740 10,000 times over. In a similar way, 754 00:50:17.140 --> 00:50:19.300 these large model companies, 755 00:50:19.460 --> 00:50:21.300 like OpenAI and Anthropic, 756 00:50:21.460 --> 00:50:22.260 are building 757 00:50:22.500 --> 00:50:24.340 basically artificial people. 758 00:50:25.060 --> 00:50:34.575 And those systems are trying to be extremely, extremely general. But just as you wouldn't ask a real person to do the same repetitive task 10,000 times, 759 00:50:35.055 --> 00:50:40.095 I think you also wouldn't ask an artificial person to do the same repetitive task 760 00:50:40.175 --> 00:50:48.090 10,000 times. You're just wasting a ton of capacity in either case, and you would much rather prefer a deterministic, 761 00:50:48.330 --> 00:50:50.490 accurate, and more cost effective 762 00:50:50.810 --> 00:50:53.610 solution that takes advantage of the structure 763 00:50:53.930 --> 00:50:54.570 and 764 00:50:54.730 --> 00:50:55.610 repeatability 765 00:50:55.610 --> 00:50:57.450 of the problem you're trying to tackle. 766 00:50:58.035 --> 00:50:59.555 So a lot of 767 00:50:59.875 --> 00:51:01.875 businesses and individuals 768 00:51:01.954 --> 00:51:05.475 are using big foundation models to tackle 769 00:51:05.714 --> 00:51:06.755 repetitive, 770 00:51:06.994 --> 00:51:10.194 highly structured tasks just because they're so powerful. 771 00:51:10.960 --> 00:51:17.680 But I think that is going to turn out to be very, very far from the most cost effective and scalable solution. 772 00:51:18.080 --> 00:51:20.480 When you have these repeated situations, 773 00:51:20.720 --> 00:51:26.315 you can deploy systems that are much better adapted to them. So I think that's 774 00:51:26.395 --> 00:51:28.795 how Ensemble models will 775 00:51:29.355 --> 00:51:30.155 replace 776 00:51:30.155 --> 00:51:39.434 some of the applications of the larger foundation models. I think the flip side is they'll actually work in harmony to augment the intelligence and capabilities. 777 00:51:39.690 --> 00:51:45.450 Right? So especially with things like like like chain of thought systems and memory systems, 778 00:51:45.609 --> 00:51:47.130 having more compute 779 00:51:47.450 --> 00:51:50.730 gives you access to more intelligence capabilities. 780 00:51:51.655 --> 00:51:55.015 But if you're trying to do something like media understanding 781 00:51:55.015 --> 00:51:57.335 or voice understanding or conversation 782 00:51:57.335 --> 00:51:58.295 understanding, 783 00:51:58.695 --> 00:52:01.495 you're spending a lot of your compute and your capacity 784 00:52:01.895 --> 00:52:03.335 to process 785 00:52:03.335 --> 00:52:07.460 that stream of data and get to a useful representation 786 00:52:07.460 --> 00:52:11.380 of that data that the rest of the system can work with. 787 00:52:12.020 --> 00:52:13.700 I see Ensemble models 788 00:52:13.780 --> 00:52:14.500 as 789 00:52:14.580 --> 00:52:17.380 basically becoming a suite of extremely 790 00:52:17.380 --> 00:52:18.260 powerful 791 00:52:18.660 --> 00:52:21.380 data processing tools 792 00:52:20.744 --> 00:52:23.545

available to these more flexible agents 793 00:52:23.704 --> 00:52:27.785 so that if you have an incoming data stream or a media stream, 794 00:52:28.025 --> 00:52:31.464 the agent can deploy a tool like an ELM 795 00:52:31.545 --> 00:52:34.480 to understand that stream, save, 796 00:52:34.720 --> 00:52:37.200 again, 100000x 797 00:52:37.280 --> 00:52:41.680 on the compute it would have by itself needed to understand that data stream, 798 00:52:41.920 --> 00:52:46.640 and then take the representation of that data produced by the Ensemble model 799 00:52:47.075 --> 00:52:54.275 and process it with all of that leftover compute that was available to it because it didn't have to understand the data itself. 800 00:52:55.075 --> 00:53:04.839 And the real question I wanna know is when is this Voice AI going to eliminate the need to manually edit my podcasts? 801 00:53:05.799 --> 00:53:10.760 It's coming. It's coming very, very soon. I think 2026 802 00:53:11.000 --> 00:53:12.440 is already 803 00:53:12.805 --> 00:53:15.205 starting off as an extraordinary 804 00:53:15.205 --> 00:53:17.045 year for voice AI models. 805 00:53:17.125 --> 00:53:18.085 You're seeing 806 00:53:18.405 --> 00:53:19.445 increases 807 00:53:19.445 --> 00:53:20.165 from 808 00:53:20.405 --> 00:53:23.125 Ensemble models and other kinds of technologies 809 00:53:23.205 --> 00:53:26.565 of 10, a 100 x in terms of cost, 810 00:53:26.805 --> 00:53:29.040 in terms of scalability. 811 00:53:29.120 --> 00:53:31.200 You're seeing improvements in accuracy, 812 00:53:31.200 --> 00:53:35.200 not just on the normal standard clean data sets, but on tricky 813 00:53:35.440 --> 00:53:40.160 conversation data sets. You're seeing voice AIs that are starting to be able to take instruction 814 00:53:40.815 --> 00:53:44.335 and follow instructions and follow nuances 815 00:53:44.415 --> 00:53:46.575 in order to be much, much more capable. 816 00:53:46.895 --> 00:53:50.575 And you're starting to get to the level of accuracy and capability 817 00:53:50.655 --> 00:53:52.335 that human beings 818 00:53:52.415 --> 00:53:55.840 doing these kinds of tasks also exhibit. 819 00:53:56.080 --> 00:54:11.185 So I think we're going to see a lot of really, really cool tech around voice AI coming out within the next twelve months. 2026 is gonna be a very, very exciting year for voice AI tech. Are there any other aspects of the work that you're doing at Modulate, 820 00:54:11.185 --> 00:54:12.945 this overall Ensemble 821 00:54:12.945 --> 00:54:13.905 architecture, 822 00:54:14.224 --> 00:54:15.585 or the applications 823 00:54:15.585 --> 00:54:19.825 that you're applying it to that we didn't discuss yet that you'd like to cover before we close out the show? 824 00:54:20.145 --> 00:54:28.390 I think we've hit on the main points. I think for a different audience, it could be interesting to go into the determinism 825 00:54:28.390 --> 00:54:29.590 and kind of 826 00:54:30.230 --> 00:54:31.430 ability to 827 00:54:32.150

--> 00:54:33.030 avoid 828 00:54:33.990 --> 00:54:37.670 things like hallucinations better by using these 829 00:54:37.670 --> 00:54:40.155 ensembles of small models. 830 00:54:40.395 --> 00:54:47.275 But I think that's actually more applicable to, like, maybe almost like a a compliance or safety audience, 831 00:54:47.675 --> 00:54:55.850 which is a lot of aspects of the things we're working in. So I think we actually hit on a a ton of the main topics I really wanted to cover. I'm feeling pretty good about this. 832 00:54:56.810 --> 00:55:13.655 Alright. Well, for anybody who wants to get in touch with you and follow along with the work that you and your team are doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gaps in the tooling technology or human training that's available for AI systems today. 833 00:55:14.615 --> 00:55:15.415 The 834 00:55:15.415 --> 00:55:18.775 biggest problem with AI tooling and systems 835 00:55:18.935 --> 00:55:19.415 is 836 00:55:19.750 --> 00:55:20.790 understanding 837 00:55:21.030 --> 00:55:24.710 where they fail and what kinds of failures they can exhibit. 838 00:55:25.190 --> 00:55:26.550 Think of things like 839 00:55:26.710 --> 00:55:30.950 customer support bots and voice bots that will just offer refunds 840 00:55:31.110 --> 00:55:37.105 or even when they're out of policy or reset people's passwords even when they haven't authenticated, 841 00:55:37.345 --> 00:55:37.905 or 842 00:55:38.385 --> 00:55:42.545 will flirt with somebody because they started flirting with it. 843 00:55:43.505 --> 00:55:46.065 These systems can go off the rails 844 00:55:46.145 --> 00:55:47.585 very, very easily 845 00:55:47.825 --> 00:55:49.265 and in unpredictable 846 00:55:49.265 --> 00:55:49.664 ways. 847 00:55:50.349 --> 00:55:51.869 And there is not 848 00:55:52.829 --> 00:55:56.670 These large models are the epitome of a black box algorithm. 849 00:55:57.470 --> 00:55:59.470 Even things like chain of thought, 850 00:55:59.950 --> 00:56:02.590 it gives the illusion that you can understand 851 00:56:03.070 --> 00:56:04.589 how the model operates. 852 00:56:05.505 --> 00:56:06.224 But 853 00:56:06.625 --> 00:56:09.185 there's actually a ton of 854 00:56:11.585 --> 00:56:15.664 different kinds of failure modes and different kinds of behaviors 855 00:56:15.664 --> 00:56:17.744 that these models can unpredictably 856 00:56:17.744 --> 00:56:18.464 exhibit 857 00:56:18.710 --> 00:56:19.430 from 858 00:56:19.670 --> 00:56:27.670 the right random combinations of input data. And it's very, very hard to know or guarantee or even put bounds on 859 00:56:28.150 --> 00:56:31.910 when you will experience the normal happy path operation 860 00:56:32.150 --> 00:56:33.750 and when you will experience 861 00:56:34.464 --> 00:56:35.585 a negative, 862 00:56:35.585 --> 00:56:36.625 unexpected, 863 00:56:36.625 --> 00:56:38.625 or out of distribution operation. 864 00:56:39.425 --> 00:56:40.865 And I think 865 00:56:41.105 --> 00:56:43.265 that in addition to 866 00:56:43.744 --> 00:56:44.545 some 867 00:56:44.865 --> 00:56:47.825 tools for testing and observability, 868 00:56:48.145 --> 00:56:51.040 which are, I think, still relatively 869 00:56:51.040 --> 00:56:51.920 primitive. 870 00:56:52.400 --> 00:56:56.240 We are going to start looking for a lot of automated tooling 871 00:56:56.400 --> 00:56:57.280 around 872 00:56:59.440 --> 00:57:00.960 red teaming models, 873 00:57:01.040 --> 00:57:02.000 finding 874 00:57:02.865 --> 00:57:03.665 gaps 875 00:57:03.985 --> 00:57:04.785 in 876 00:57:04.865 --> 00:57:06.545 how models are running 877 00:57:06.785 --> 00:57:09.345 on data and systems and data streams, 878 00:57:09.745 --> 00:57:10.305 and 879 00:57:11.105 --> 00:57:20.460 monitoring these systems at extremely high scale. Watch every single conversation that's going on just like you would monitor human interactions 880 00:57:20.460 --> 00:57:22.700 at a very, very high level of scale. 881 00:57:23.020 --> 00:57:23.660 And 882 00:57:23.980 --> 00:57:27.980 right now, a lot of the tooling for observability and monitoring 883 00:57:28.780 --> 00:57:29.660 rely on 884 00:57:30.215 --> 00:57:33.735 looking at the model's own intermediate outputs. 885 00:57:33.975 --> 00:57:38.775 Like if you have a voice bot that's doing speech to text to speech, you monitor the text. 886 00:57:39.015 --> 00:57:42.375 But the problem with that is that you're monitoring a part of the system 887 00:57:42.775 --> 00:57:43.415 that 888 00:57:43.630 --> 00:57:45.150 you're trying to 889 00:57:45.390 --> 00:57:47.790 observe the behavior of the whole system. 890 00:57:48.109 --> 00:57:50.350 And so if a different part, 891 00:57:50.590 --> 00:57:56.990 like the text to speech piece of the bot, is going wrong, if you're just monitoring the text part, 892 00:57:57.230 --> 00:58:01.325 you're not going to know what's happening. So being able to observe 893 00:58:01.325 --> 00:58:03.405 the behaviors of these models 894 00:58:03.565 --> 00:58:04.605 and interacting 895 00:58:04.605 --> 00:58:06.445 with each other and with people 896 00:58:06.685 --> 00:58:08.685 in the same way that you would observe 897 00:58:09.005 --> 00:58:13.965 human beings interacting with each other and being able to derive insights and debug 898 00:58:14.310 --> 00:58:15.350 those things 899 00:58:15.990 --> 00:58:21.190 is I think a really big gap and where a lot of these problems and failure modes and 900 00:58:21.430 --> 00:58:23.510 newsworthy headlines are coming from. 901 00:58:24.070 --> 00:58:24.950 Absolutely. 902 00:58:25.110 --> 00:58:33.035 Well, thank you very much for taking the time today to join me and share the work that you and your team have put into developing this Ensemble architecture 903 00:58:33.035 --> 00:58:40.395 and some of the interesting use cases for voice AI as differentiated from the textual interfaces 904 00:58:40.395 --> 00:58:42.475 that have exemplified 905 00:58:42.475 --> 00:58:53.670 the past couple of years. So I appreciate all the time and energy that you and your team are putting into that, and I hope you enjoy the rest of your day. Well, thank you so much for having me. It was wonderful to chat, and I really enjoyed it. I hope you enjoy your day too. 906 00:58:55.785 --> 00:59:03.865 Thank you for listening. Don't forget to check out our other shows. The data engineering podcast covers the latest on modern data management, and podcast.init 907 00:59:03.865 --> 00:59:18.828 covers the Python language, its community, and the innovative ways it is being used. Visit the site to subscribe to the show, sign up for the mailing list, read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email hosts@aiengineeringpodcast.com 908 00:59:18.828 --> 00:59:19.548 with your story.