TRANSCRIPT 🔗
WEBVTT
00:00:09.519 --> 00:00:13.599
So what I'm here to do is talk about voice.
00:00:13.759 --> 00:00:23.600
So voice is a very natural interface, it's how we uh work with each other, uh, and why can't this be something that we use to talk with our uh products and our machines?
00:00:23.839 --> 00:00:31.359
Um there may be questions about this later, but there's some real challenges to how to make this a viable business and how to scale this.
00:00:31.440 --> 00:00:38.320
So this is really where uh we come in as FEMTOAI and going to be sharing a few of our tips and tricks to help make this work.
00:00:40.640 --> 00:00:42.079
Preaching to the choir here.
00:00:42.159 --> 00:00:52.000
So we all know this and uh are aware that these voice uh-based interfaces to these large models are becoming a little bit more and more popular.
00:00:52.159 --> 00:00:59.039
You know, we're finding that people actually like talking to their chat GPTs, their Geminis, um, you know, their series even.
00:00:59.280 --> 00:01:02.000
Um of course, this all comes with some compromises.
00:01:02.399 --> 00:01:08.159
When you're talking to these systems, uh those providers, of course, have to pay for that, or you have to pay for that.
00:01:08.239 --> 00:01:11.680
Uh, you know, you notice the latency, it's not like we're just talking to any other person.
00:01:11.760 --> 00:01:19.760
Uh, and then of course, you are definitely sacrificing something on privacy, and of course, you this doesn't work unless you have that internet connection.
00:01:21.359 --> 00:01:25.200
So, of course, the prescription is maybe we should push all out onto the edge.
00:01:25.359 --> 00:01:32.480
Now, nothing is uh there's trade-offs to everything, and uh just like that, when you move to the edge, as we all know, there are some real challenges.
00:01:32.560 --> 00:01:39.840
Uh, the edge uh does not have infinite resources relative to a big data center and all of the nuclear power plants that power it.
00:01:40.000 --> 00:01:50.879
Um, the edge also has uh very limited power constraints, and I think maybe one thing that does not get emphasized enough to make this a viable product, it has to still perform well.
00:01:51.040 --> 00:01:56.719
So even if you satisfy all the efficiency constraints, you know, if it doesn't perform well, you know, it's all for naught.
00:01:58.159 --> 00:02:07.280
So I know we're all team edge AI here, and uh we'd love to win over cloud AI, uh, but the real world is uh actually you know somewhere in between, right?
00:02:07.599 --> 00:02:11.520
We could do the everything on the cloud, which is a lot of what is happening today.
00:02:11.680 --> 00:02:17.599
We could imagine a world where um you know maybe everything is red on the edge, uh, but there's a lot of gray area between there.
00:02:17.759 --> 00:02:24.400
And if you are a product developer building the systems, uh what makes sense for you and your product is very dependent on what you're trying to build.
00:02:24.560 --> 00:02:31.919
What we're here to do is enable more and more of those capabilities that might be locked up in the cloud to be pushed out to the edge.
00:02:32.159 --> 00:02:44.560
However, of course, each step towards the edge, uh you have more constraints on latency, uh, you have more uh constraints on the power budgets, and so you correspondingly have to become much more efficient and uh better at it.
00:02:46.159 --> 00:02:48.080
So we'll go through a few uh techniques.
00:02:48.159 --> 00:02:51.759
Some of these are tried and true that I'm guessing uh many people are here are aware of.
00:02:51.840 --> 00:03:03.039
Uh, some of these may be a little bit newer, uh, but these are just a few of the things that us as FEMTOAI have found to be very helpful to help increase the capability of voice AI at the edge.
00:03:04.719 --> 00:03:08.560
So the first one uh that's a little bit underappreciated is uh sparsity.
00:03:08.639 --> 00:03:23.599
So this is really our core technology as FEMTOAI, is to really say there are some good ideas from maybe biology that we can take into electronics and chip design uh and deploy onto products to enable these voice interfaces.
00:03:23.759 --> 00:03:37.039
So sparsity is basically saying, you know, you could take a big AI model that is very flexible and can do a lot of things, uh, and then you can basically remove maybe 90% of it because you have a product and it does certain things, and then you can deploy that.
00:03:37.199 --> 00:03:42.879
And R Silicon is uniquely able to help uh exploit this fact to say that you don't need to do everything.
00:03:43.120 --> 00:03:52.719
You maybe only need to do a subset, and when you run your big model on R chip, you still retain the capabilities that matter, and then you can deploy it onto something as small as uh we're seeing here.
00:03:52.800 --> 00:03:54.639
So it just fits in a little tweezer.
00:03:54.800 --> 00:04:01.520
So this is through, this basically enables you to increase your efficiency by saying, uh, what do you actually need to do?
00:04:01.680 --> 00:04:04.639
Only do that, you know, please don't do anything else.
00:04:06.879 --> 00:04:08.960
Of course, it's just a claim without some data.
00:04:09.039 --> 00:04:12.800
So whenever we engage with customers, they always ask, like, okay, how real is this?
00:04:12.960 --> 00:04:14.400
How far can I actually push this?
00:04:14.479 --> 00:04:18.240
So this is a real study uh that we did in partnership with one of our customers.
00:04:18.319 --> 00:04:20.879
Uh for anybody who can read Korean, you know what this is about.
00:04:20.959 --> 00:04:38.560
Uh, but it's basically proving the point, is to say that if you want to uh take a model, which maybe started at 500 kilobytes of memory, uh you can compress it down to 60 kilobytes, so almost 10x compression, while still maintaining production level customer sort of acceptance level performance.
00:04:38.720 --> 00:04:45.120
So 95% true positive
Load full transcript
rate and you know, maybe only 0.25 uh false activation rate per day.
00:04:45.199 --> 00:04:56.879
So this is something that is the threshold for their production that they have found with their customer studies is actually acceptable, and that we can retain or we can preserve these performance while taking that footprint down uh almost 10x.
00:04:59.600 --> 00:04:59.920
Okay.
00:05:00.560 --> 00:05:01.600
Data augmentation.
00:05:01.759 --> 00:05:05.680
One of the challenges, of course, in the edge is that there's many, many things you could want to do.
00:05:05.839 --> 00:05:07.360
And how do you get data off for all that?
00:05:07.439 --> 00:05:20.720
So tried and true technique is basically taking uh the data that you do have, tweaking it a little bit to account for the variability that you will encounter in the world, whether that's say different transfer functions for different rooms or different people and how they sound.
00:05:20.800 --> 00:05:26.079
Uh, these are all different ways to kind of parameterize the variability of the world and account for that in your AI model.
00:05:26.160 --> 00:05:32.160
Because you definitely don't want to go out and collect data from uh millions and millions of people, that's just not a scalable technique.
00:05:32.240 --> 00:05:34.399
So definitely something that we recommend.
00:05:36.720 --> 00:05:50.639
One thing when we start to get more flexible AI models, so if we start to move beyond, hey Alexa, hey Siri, hey Google, right, those simple wake words, and we start getting into commands, you start getting the possibility for confusion of the AI model, right?
00:05:50.879 --> 00:05:59.519
So, for example, say you wanted to play music on your speaker, um, but there's a lot of things that sound like maybe play music, right?
00:05:59.680 --> 00:06:04.079
So you could say play, you could say plain, you could say pay, you could say okay, right?
00:06:04.160 --> 00:06:05.519
All these kind of sound similar.
00:06:05.680 --> 00:06:13.439
And you of course have to make sure as you take this model and make it smaller, uh, that you are preserving this ability to discriminate between things that sound similar.
00:06:13.600 --> 00:06:15.519
So there's kind of a space of what you want to hit.
00:06:15.680 --> 00:06:22.399
You have to do that, but you also have to put kind of a fence around things that are nearby so you don't get distracted by things that sound similar.
00:06:22.560 --> 00:06:31.519
So this has been very key in, I guess, some of the previous slides that you saw where it came through the data on uh not hitting those false activations while still maintaining the performance.
00:06:33.120 --> 00:06:43.839
Okay, and then again on the data side, I'm I'm talking just as much about sort of underlying processing technology as how to handle data here, and that's probably an insight into how what it takes to make these products.
00:06:43.920 --> 00:06:51.040
Um, but yeah, synthetic data generation, there are
a lot of very powerful cloud AI models, and this doesn't have to be a you know us versus them thing.
00:06:51.279 --> 00:07:08.399
We can leverage those big AI models to generate very realistic data that can help us address those challenges with uh similar sounding distractors, uh different accents, different languages, and to basically help the big AI models become smaller while still being robust to the real world.
00:07:11.360 --> 00:07:18.160
And then finally, here's another one that I think does not get talked about as much in sort of AI-first communities, right?
00:07:18.480 --> 00:07:24.879
So I think probably some people here will be coming more from the DSP uh land and kind of moving into AI.
00:07:25.040 --> 00:07:31.360
Other people will be uh like us kind of be AI first and trying to say, well, I want to add AI value to the world.
00:07:31.519 --> 00:07:36.240
Um, however, of course, this these AI systems they don't live in isolation.
00:07:36.399 --> 00:07:41.680
You know, they live in an ecosystem themselves of other processors, other functionality.
00:07:41.920 --> 00:07:49.839
And what we're seeing is that you know, the AI processors, while they may be targeted towards AI, they're kind of also good at doing some of these other functions.
00:07:50.079 --> 00:07:57.439
So for DSP uh developers, what we're seeing is that a lot of the DSP capabilities are kind of getting sucked in to the AI processor.
00:07:57.519 --> 00:08:00.720
So it's not really true that it's just an edge AI processor anymore.
00:08:00.879 --> 00:08:05.519
It is sort of an edge processor to handle uh the whole signal processing chain.
00:08:05.680 --> 00:08:20.160
And this is, of course, you know, one of the angles that we're taking as a you know, as a venture-backed company to say like, okay, we're adding more and more value, and this is not just the AI kind of niche market, but this is really you know the whole market of general signal processing.
00:08:20.240 --> 00:08:20.399
Right?
00:08:20.560 --> 00:08:26.879
That's a much more uh established space, and it is definitely something that the customers are pulling us towards.
00:08:27.199 --> 00:08:37.919
So by combining these AI models with these DSB functions, uh, we can of course add more value, and we can also play better with the uh cloud systems that are behind it.
00:08:39.600 --> 00:08:41.600
So again, a little bit of data here.
00:08:41.840 --> 00:08:52.559
Um again, our customer asks us not just to run the AI model, but to also run some of the DSP processing that they already do to help support their cloud model.
00:08:52.720 --> 00:08:54.799
So there's a big cloud model in the back end.
00:08:55.039 --> 00:08:57.120
If it gets things wrong, that's a problem.
00:08:57.279 --> 00:09:02.159
It's not that robust to noise because you know they can't account for all of the noise that could happen in the world.
00:09:02.240 --> 00:09:14.639
Um, but with a little bit of processing on the edge, we reduce the
amount of bandwidth that they have to use to send data back and forth, which is cost to them, and we improve the performance of those large AI models running on the back end.
00:09:14.799 --> 00:09:20.080
So again, it's not so much a us versus them cloud or edge, you only pick one.
00:09:20.159 --> 00:09:29.759
It's really how does this all kind of work together in a coherent system and running this on these kind of new edge AI processors that are emerging, like RZF M2 AI.
00:09:29.919 --> 00:09:50.480
So in this case, we basically took a model like Whisper, so speech to text, uh, in a noisy environment, it might have had a 20% uh error rate for the characters, uh, with just a little bit of pre-processing that's done on our AI processor, but done in a conventional way, we can basically reduce that error uh by more than 2x, right?
00:09:50.639 --> 00:09:56.080
And so that's of course better customer experience and is really was really our requirement for working with them.
00:09:56.240 --> 00:09:59.120
So we don't have to do just AI, although that's very important.
00:09:59.279 --> 00:10:05.919
We have to do AI plus DSP plus some like light management plus a lot of things, all kind of in one platform.
00:10:06.080 --> 00:10:28.559
And so I think maybe as we go forward uh as a community, we'll start seeing more and more of these capabilities integrated into these systems because you know ultimately customers want nice unified solutions and a platform that is flexible to run not just this cool, cool AI stuff, but also uh a little more ground grounded uh established workloads as before.
00:10:30.399 --> 00:10:38.480
Okay, so putting it all together, uh, this has enabled us to get you know real production um great systems that have very low latency.
00:10:38.559 --> 00:10:48.879
That's kind of that that was one of the selling points for edge AI, is that we don't want to have to say, you know, ask something, wait for the internet connection, wait for it to come back, you just want the result right there.
00:10:49.039 --> 00:10:54.960
But we can do it in a form factor and footprint in terms of power uh that is acceptable to these products.
00:10:55.120 --> 00:11:15.600
So you're talking about you know less than half a milliwatt of power, we're talking about you know 100 kilobyte sort of footprint, so plenty of capacity to add other workloads, uh, and this is these are all kind of the threshold of what is necessary to enable always on voice uh for every product, not just you know, the smart speakers or the limited places where it is right now.
00:11:18.559 --> 00:11:20.320
So, where are we going from here?
00:11:20.480 --> 00:11:24.720
Um, you know, we're talking about edge AI as often these sort of small systems.
00:11:24.879 --> 00:11:29.600
Um there's always the question of well, what happened if I take these techniques and I scale them up?
00:11:29.919 --> 00:11:36.639
So one thing that's nice about our core technology of sparsity is that it scales with the workload.
00:11:36.879 --> 00:11:42.240
If you want to run a if you want to run a bigger workload, um you can sparsify it more.
00:11:42.480 --> 00:11:46.000
So these are uh results that you know we haven't made public anywhere else before.
00:11:46.080 --> 00:11:56.480
So it's part of a sneak preview of some things that may be to come, of taking a full speech to text model, Whisper in this case, and pruning it kind of as far as it goes.
00:11:56.639 --> 00:12:07.519
And you can see from the data is basically on the bottom is the size of the model, how many non-zero parameters ultimately do these systems have, and the uh performance on the y-axis, right?
00:12:07.679 --> 00:12:21.440
And this basically pruning provides you a very natural way to sweep out uh a pre-to-optimal curve of saying, like, well, whatever your acceptable uh level of performances, you can find the most uh compact model uh to support it.
00:12:21.600 --> 00:12:26.960
So this is definitely something that's going to be coming down the line in uh next generation products.
00:12:27.120 --> 00:12:35.200
It's again, it's part of that sort of trend to take the cloud, uh very powerful cloud AI models, and push them onto the edge.
00:12:36.639 --> 00:12:40.000
So, with that, um thank you for your time.
00:12:40.080 --> 00:12:44.080
Uh, we are very excited to continue to support our developers and our customers.
00:12:44.240 --> 00:12:46.960
Uh please visit our website as femto.a.
00:12:47.360 --> 00:13:00.000
And uh, if you are a builder or a developer, uh, we very much welcome you to our uh developer portal where you can access all of those uh models that we were running, and of course, all the data that you may need to have confidence that you can build edge AI.