TRANSKRIP 🔗
WEBVTT
00:00:00.080 --> 00:00:03.439
It's clear to me now more than ever that like not all words are created equal, right?
00:00:03.680 --> 00:00:09.599
In context, you know, missing a yes or no could be everything or it could be nothing.
00:00:09.759 --> 00:00:16.480
Now that we're have we're starting to deal with audio models that like take prompting in and situational context in addition to the audio itself.
00:00:16.640 --> 00:00:20.640
Like just going on like raw, how many words did we get right?
00:00:20.719 --> 00:00:25.039
Like is a pretty bad proxy for all the stuff that we actually care about.
00:00:35.200 --> 00:00:39.759
Hello everyone and welcome to another episode of Deep Learning with PolyAI.
00:00:40.000 --> 00:00:42.640
Today I've got Oliver back on the show with me.
00:00:42.880 --> 00:00:43.679
Good to see you again.
00:00:44.000 --> 00:00:44.479
Great to be here.
00:00:44.640 --> 00:00:44.880
Yeah.
00:00:44.960 --> 00:00:56.320
Oliver is one of our elite Asian designers and a real kind of like force behind a lot of work, whether it's, you know, pioneering Open Claw or just, you know, bringing the good spirit of linguistics in.
00:00:56.560 --> 00:01:09.120
And today, after we spoke about some very interesting data that he showed me, I thought it would be really interesting to just kind of like have a whole episode that really looked into the magical notion of a word error rate, right?
00:01:09.280 --> 00:01:18.959
So we build voice agents, and how well these systems work feeds downstream from whether you're able to recognize what someone has said or not, right?
00:01:19.280 --> 00:01:30.319
And I think most companies, especially those that focus predominantly on that component, really just talk about improving the word error rate and getting to a better and better one in their public benchmarks.
00:01:30.400 --> 00:01:38.560
And it's kind of treated as a holy grail by a lot of people buying software, deciding what component to use, deciding whether it's good or not.
00:01:38.799 --> 00:01:42.400
And we'll talk a lot about whether it's good or not.
00:01:42.719 --> 00:01:53.040
I think like one anecdote from my PhD supervisor was really good where I think it was the early days of deep learning, and we were all we just thought it would all be done in two years.
00:01:53.200 --> 00:01:55.840
I think that's the prevalent attitude that's continued ever since.
00:01:56.000 --> 00:02:03.040
And I remember Steve telling me, like, whoever thinks that speech recognition will be solved in two years has worked on the problem for less than two years.
00:02:03.200 --> 00:02:06.319
And, you know, I've now spent more than two years working on the problem.
00:02:06.560 --> 00:02:14.639
I feel like it was very but Oliver, like maybe over to you to talk a bit about like the word error rates and the findings that you've had over the past kind of few months.
00:02:14.879 --> 00:02:15.439
Yeah, totally.
00:02:15.520 --> 00:02:24.960
I so I mean, I
Load full transcript
have much less experience like working on this like word error rate in an academic context than you, but you know, what I can tell you is that But more than two years.
00:02:25.199 --> 00:02:27.280
More than two, uh yeah, I guess so.
00:02:27.520 --> 00:02:37.120
So I first of all, I know that ASR is not solved, and second of all, I know now that word error rate is a pretty terrible proxy for conversation quality.
00:02:37.280 --> 00:02:55.360
Like if you're using that as a way to evaluate both how well your ASR is performing in context and just as a way to understand like how well are my customers being understood by the virtual agent, like it is just utterly confounded with noise.
00:02:55.599 --> 00:03:02.240
And you know, the thing I've been saying as I've been going through a lot of this data is like it's clear to me now more than ever that like not all words are created equal, right?
00:03:02.319 --> 00:03:09.120
Like in context, you know, missing a yes or no could be everything or it could be nothing.
00:03:09.360 --> 00:03:16.879
And depending on the turn length, and that could be 50% word error rate or it could be 1% word error rate.
00:03:17.199 --> 00:03:41.199
And when now that we have these models that are able to reason around ASR mistranscriptions and reason in context, and especially now that we're have we're starting to deal with audio models that like take prompting in and situational context in addition to the audio itself, like just going on like raw how many words did we get right, like is is a pretty bad proxy for all the stuff that we actually care about.
00:03:41.599 --> 00:03:41.919
100%.
00:03:42.319 --> 00:03:47.840
I think that I remember, you know, and data sets evolve and change and gradually become more difficult.
00:03:48.080 --> 00:03:59.199
But you know, we went from kind of like talking about a 5%, you know, state-of-the-art word error rate to now, you know, recently I was speaking with a CIO of a company and he's like, and you were like a 2% like everyone else.
00:03:59.280 --> 00:04:05.039
And I'm like, no one is a two percent for anything that really objectively matters in a complex conversation.
00:04:05.280 --> 00:04:17.839
And I think kind of like you know, my my favorite kind of like peasant logic thing to break this is like to anyone who has Alexa or Google Assistant, it's like, does it work 98% of the time?
00:04:18.000 --> 00:04:20.639
And it's like, I think it's really far from that.
00:04:20.879 --> 00:04:27.920
But it somehow managed to permeate the consciousness of people where it's like, oh yeah, it's all but done, voice is solved, and I don't know.
00:04:28.000 --> 00:04:31.279
I guess we'll dig into your data and take a look at why it's not, but yeah.
00:04:31.600 --> 00:04:32.000
Totally.
00:04:32.160 --> 00:04:36.160
Okay, well, so I'm happy to pull up just some basic stats of like what we what we've done here.
00:04:36.399 --> 00:04:46.480
You know, I ran this analysis over the course of about a m a month on on some of our really high volume production calls, analyzed over 100,000 individual user turns.
00:04:46.639 --> 00:05:04.959
What we used kind of as the gold standard for word error rate here was a consensus between two different speech recognition vendors, uh, um, or I should say audio, audio LLM vendors, um, which were able to take the entire conversation and all of the context and that definitely perform in a vacuum with the best word error rate compared to any ASR.
00:05:05.120 --> 00:05:10.800
They're of course a bit slower and they don't produce like continuous transcription, so they're a bit harder to use in production.
00:05:10.879 --> 00:05:12.560
Um, but they they perform really well.
00:05:12.639 --> 00:05:23.920
And so when when we retro retroactively do analysis like this, we can use that kind of consensus between these two vendors as like a really good, confident gold standard to compare against.
00:05:24.240 --> 00:05:27.279
And so this was over the course of around 25,000 calls.
00:05:27.519 --> 00:05:42.319
And what we did was when we found turns that were that differed in production from the gold standard that the consensus produced, we had an LLM judge basically classify those mistranscriptions, you might call them, as either benign.
00:05:42.480 --> 00:05:48.160
So these are like little formatting errors, kind of filler words, stuff that doesn't ultimately change the meaning.
00:05:48.480 --> 00:05:57.040
Semantic, which is changes that alter the meaning of the user input, but that could be reasoned around in context or that you could ask clarification about.
00:05:57.360 --> 00:05:58.399
And then critical.
00:05:58.560 --> 00:06:01.439
And these are specifically things like what we would call entity flips.
00:06:01.519 --> 00:06:08.480
So things that change the value of a slot that the user is providing in such a way that obfuscates or changes the value in a critical way.
00:06:08.639 --> 00:06:15.519
So this is something like if you're collecting a phone number and you mishear a digit, that's a critical error because you need to get that 100% right.
00:06:15.680 --> 00:06:19.439
And it's not necessarily obvious in context that you misheard something.
00:06:19.680 --> 00:06:26.000
The the agent has no reason to believe it misheard something if it heard, you know, for some reason, three instead of eight.
00:06:26.160 --> 00:06:27.759
But so that would be something critical.
00:06:27.839 --> 00:06:35.199
And I can pull up for you just like a little to show you kind of the proportions that we're dealing with of these errors of the of the missed turns.
00:06:35.439 --> 00:06:42.959
So up to 42%, so so around 28% of mistranscriptions were deemed benign.
00:06:43.279 --> 00:06:46.879
Up to potentially 42, some of them weren't judged for a variety of reasons.
00:06:47.360 --> 00:06:52.079
We have a relatively small proportion of just of these kind of meaning-changing semantic errors.
00:06:52.160 -->
00:07:00.319
So this is when like content words get flipped, but in context, they can often be reasoned around, or it's clear the what the clarification question should be.
00:07:00.480 --> 00:07:04.560
And then about half of the errors we saw are these entity flipping errors.
00:07:04.879 --> 00:07:16.000
And what this tells me actually is that the the proportion that are benign is a huge amount of obfuscating and confounding noise, basically, in the word error rate signal.
00:07:16.160 --> 00:07:20.800
I'll stop there for a second to like hear, like, I'm sure you have thoughts about this, but like that's my first takeaway from this.
00:07:21.040 --> 00:07:29.360
No, no, I mean it's like I think it's like the empty calories and this half where, you know, like anything people need to understand it's not like a word is missed.
00:07:29.439 --> 00:07:42.560
Like, yeah, not all words are equal, but in particular, you know, articles, like whether you pick up and or the or whether someone might have said an or not, and whether that's in or not is completely material.
00:07:42.800 --> 00:07:44.480
It's like semantically empty.
00:07:44.720 --> 00:07:54.160
I think, you know, beyond that, like plural or singular might matter for the meaning, but again, like an L-based system is probably gonna make it through about just fine, right?
00:07:54.399 --> 00:08:02.240
Similarly, you know, like a not omitted can be really, really devastating and creating like an antonym and opposite meaning of other things.
00:08:02.399 --> 00:08:08.160
So between all that, like it it can be really hard to like decipher like you know which ones really matter.
00:08:08.319 --> 00:08:13.920
And it's very easy to have even an you know something with a much higher word error rate performing much worse.
00:08:14.000 --> 00:08:25.839
And it's plagued evals on dialogue systems forever in that you improve something, you measure on this like intermediate task, you see a higher number, you roll it into production, and god forbid you made changes downstream as well.
00:08:26.000 --> 00:08:29.519
At that point, you're completely lost for like where did you make progress?
00:08:29.680 --> 00:08:31.199
Where did you degrade?
00:08:31.279 --> 00:08:32.559
And it's it's tough, right?
00:08:32.639 --> 00:08:36.240
And the more we integrate these components, the the harder it gets still, right?
00:08:36.399 --> 00:08:57.120
Yeah, and it's not even like we can identify like particular classes of words that are semantically like like in, you know, we have this basic like distinction in linguistics between content words and function words, where you have like your words that sort of mark syntactic information, like which are your function words, so things like articles and and modal verbs and stuff like that.
00:08:57.200 --> 00:09:01.919
And then you have your content words, which actually sort of introduce the mental content into the sentence.
00:09:02.240 --> 00:09:09.519
But it's not even like
you can necessarily just classify sort of an inventory of content and function words and know which ones are matter and which ones aren't.
00:09:09.600 --> 00:09:11.200
Like it's obviously all contextual.
00:09:11.360 --> 00:09:21.360
So if the question is, you know, are you calling to check the status of an order and the user just says yes and you mishear the yes, like that's a disastrous mishearing.
00:09:21.440 --> 00:09:23.679
Like you you missed the whole content of what they said.
00:09:23.840 --> 00:09:30.720
But if they say, Yes, I am, and you miss the yes, but you hear the I am, that's basically completely benign.
00:09:30.799 --> 00:09:35.840
Like the like you'll proceed and and interpret that exactly the same as if they had just said yes.
00:09:36.720 --> 00:09:37.600
So it's like Yeah.
00:09:37.840 --> 00:09:43.279
And I mean, you know, rabbit hole, but like in different languages, it also like behaves very, very differently, right?
00:09:43.360 --> 00:09:49.519
I mean, like the morphology of Slavic languages in the cases, for instance, means that word error rates tend to be quite a bit higher.
00:09:49.759 --> 00:09:52.480
For all intents and purposes and understanding, it doesn't really matter.
00:09:52.720 --> 00:10:00.720
In fact, there are entire groups of say, I don't know, Serbian population that get cases wrong or use fewer than like what language does.
00:10:00.879 --> 00:10:03.120
And you know, like it doesn't really matter.
00:10:03.279 --> 00:10:04.240
Like, not for this.
00:10:04.480 --> 00:10:11.759
That's so interesting because like the stem is it gets like the stem right, but it like it's yeah, it's like inflected wrong, and so that's counted as a word error.
00:10:11.840 --> 00:10:12.000
Yeah.
00:10:12.080 --> 00:10:12.799
Oh, that's so interesting.
00:10:13.120 --> 00:10:22.720
And you know, I think there's almost like you would then look at character level error rates, but I feel like it would probably have the exact same empty calories, and maybe more actually, than the word error rate.
00:10:22.879 --> 00:10:23.360
Yeah.
00:10:23.679 --> 00:10:35.440
You know, one of the things that we were looking to sort of correlate or not correlate word error rate with is our proprietary evaluation, which is what we call our polyscore, which gets run on all of our production calls.
00:10:35.759 --> 00:10:40.799
And so I thought I'd take a second to just sort of talk a little bit about polyscore and what it consists of.
00:10:41.039 --> 00:10:49.279
So actually, maybe I feel like you're maybe better suited to speak to like a little of the history of polyscore and sort of what how that came about and how we're I'm happy to.
00:10:49.759 --> 00:10:57.679
I mean, like look, I mean, I think that stuff like predict predictive, like MPS scores and C sets and just like a measure of like, is it a good call or not?
00:10:57.919 --> 00:10:59.279
Has been something that's always needed.
00:10:59.360 --> 00:11:04.559
And you know, whether
you trust the vendor and showing you whether this is good or not is one piece.
00:11:04.799 --> 00:11:11.919
The other piece where it's just objectively very useful, 21 launching a system is you want to see the great calls and you want to see the terrible calls, right?
00:11:12.240 --> 00:11:19.200
So I think that, you know, as a measure of that, it's something that's been honed for years and you know, continues to be honed.
00:11:19.279 --> 00:11:23.440
It really composes it's composed of a measure of the quality of the conversation.
00:11:23.600 --> 00:11:28.559
Like, you know, is the fluidity is the UX is it a natural human-sounding conversation?
00:11:28.720 --> 00:11:30.159
Are we interrupting each other?
00:11:30.320 --> 00:11:32.639
Or if they're interrupting, are we stopping?
00:11:32.799 --> 00:11:41.840
And very, very many things that have to do with everything from you know the quality of like the audio piece to kind of just like the fluidity of the back and forth.
00:11:42.000 --> 00:11:44.480
And then maybe most importantly, like task success.
00:11:44.639 --> 00:11:52.480
Like, did the AI agent actually do what the agent is supposed to do in that particular uh conversation?
00:11:52.639 --> 00:11:54.879
And you know, sometimes the caller is not happy with that.
00:11:54.960 --> 00:12:04.879
If they want a refund and you're not able to give them a refund, you know, like the score may end up being high because the system behaved and did the right thing, but yeah, it's pretty nuanced.
00:12:04.960 --> 00:12:11.440
And, you know, again, kind of like the LLM judges from the other one, it's it's not perfect, but it's pretty good.
00:12:11.679 --> 00:12:12.000
Yeah.
00:12:12.159 --> 00:12:26.879
And it's really nice that it it returns this kind of granular, like constituent broken down score because we actually get to see some like interesting ways in which the number of meaning-changing errors that occur over the course of the call affect conversation quality.
00:12:27.039 --> 00:12:36.320
So obviously, like the need for clarification and repeating of the same questions, but actually don't affect task success nearly as much as you might expect, which is kind of interesting.
00:12:36.480 --> 00:12:56.000
So, you know, the customer tends to pay in friction, but in terms of like the actual effect it has on getting where they're trying to go or even call outcomes like escalations and handoffs, you're actually it's kind of amazing how effective the model is at like uh retaining that containment, even at the expense of a little bit of friction of re-asking those questions.
00:12:56.159 --> 00:13:01.840
Um, and just to sort of hammer in something I was saying before about this empty calories, I like that metaphor you're using.
00:13:02.080 --> 00:13:08.639
You know, benign only word error rate, like as you might expect, varies widely and like predicts nothing on the polyscore side.
00:13:08.720 --> 00:13:30.559
Like we can we can account for maybe up to 0.5% of variance in polyscore via the benign word error rate, which first of all goes to kind of validate the the LLM judge that was determining the severity of errors, but also tells you exactly what you'd expect, which is that like word error rate is confounded by all of this noise.
00:13:30.799 --> 00:13:35.440
And if you isolate the benign stuff, like it's just it's literally just noise.
00:13:35.679 --> 00:13:35.919
Yeah.
00:13:36.159 --> 00:13:36.399
Yeah.
00:13:36.480 --> 00:13:41.759
I mean, like the the zero correlation is almost like it looks it almost looks rigged, Oliver.
00:13:42.960 --> 00:13:46.639
Yeah, the task success zero correlation, it does, it does almost look rigged.
00:13:46.799 --> 00:13:48.000
I was kind of amazed by that.
00:13:48.080 --> 00:13:50.080
And I you can actually see that even more.
00:13:50.320 --> 00:13:58.559
It's amazing how how the task success stays super flat, even as we see the compounding effects of meaning changing errors.
00:13:58.639 --> 00:14:07.519
And this is sort of the flip side to word error rate, which is where we see these actually, I think, really valuable correlations, which is how many semantic or critical errors happen over the course of the call.
00:14:07.600 --> 00:14:27.759
And you see this really clear monotonic relationship between the number of these meaning-changing errors and conversation quality on the polyscore, for instance, which is this left graph over here, or handoff rate, which you can see in this right graph, which crawls up with each meaning-changing error that happens over the course of the call.
00:14:28.000 --> 00:14:32.639
That little dotted line you see on the top of that left graph over there is the task success.
00:14:32.720 --> 00:14:35.679
So again, like amazingly flat the entire time.
00:14:35.919 --> 00:14:40.639
And really, it's just the user is paying in additional turns and clarification.
00:14:40.720 --> 00:14:51.039
But I like my sort of rose-colored glasses take on that is like that's an amazing vindication of the capacity of LLMs to recover from these mishearings.
00:14:51.279 --> 00:14:59.759
Because even if it requires a question to be asked more than once or a follow-up question to be asked, the user is still getting the task accomplished that they want to.
00:14:59.840 --> 00:15:01.440
They're just paying in a little bit of friction.
00:15:01.519 --> 00:15:07.600
And I think like that's something that, you know, back in the deterministic days, that like was just a complete non-starter.
00:15:07.679 --> 00:15:14.080
You could not ask targeted follow-ups, you could not repeat questions in a way to actually glean the clarification that you wanted.
00:15:14.399 --> 00:15:14.960
No, totally.
00:15:15.120 --> 00:15:29.440
I think like, you know, maybe just to give like some contextual examples, but you know, if you're being asked for, I don't know, your British car license plates, and you said LO23 EWN, and then like it repeated it and you
know, got the last character wrong.
00:15:29.519 --> 00:15:30.960
And then you would be like, no, no, no.
00:15:31.039 --> 00:15:33.120
It's like and for November, right?
00:15:33.360 --> 00:15:36.639
At that point, you're like, okay, like that last one, yeah, good.
00:15:36.799 --> 00:15:41.360
Or like repeat the last three system repeats, it's that, not that poof, right?
00:15:41.440 --> 00:15:49.919
And that's like very much how humans would talk to each other, and it's like that's exactly it, and it's just really you're right, like how much, and again, this has to be implemented, right?
00:15:50.000 --> 00:15:51.679
It doesn't always work by default, right?
00:15:51.759 --> 00:15:55.840
It can still be frustrating, and it is frustrating, we see it from the coal quality score.
00:15:55.919 --> 00:16:02.159
But I think like it's interesting in that graph how you have like the steepness of the coal quality falling is higher, right?
00:16:02.240 --> 00:16:12.879
Because I guess the quality is decreasing, but the functional capability of the system is there as long as the human stays in the coal, which again is something that I guess we would have to kind of like derive from the data, right?
00:16:13.120 --> 00:16:24.480
Yeah, though, I mean, I I guess another thing that I was sort of pleasantly surprised by is to see how while we do have absolutely that slope downward over those first two errors, like it's not as steep as you might think it would be.
00:16:24.559 --> 00:16:34.320
Like you might think that people, when they get one critical or semantic error and it's clear that they've been misunderstood, they're like, screw this, let me talk to a human.
00:16:34.480 --> 00:17:04.640
And so, and on on neither round, on neither graph, do you see this incredibly steep either degradation in call quality, meaning that like we're not able to recover from those errors, or an extremely steep increase in handoffs, which you might expect to see, which also points to this capacity for the model to retain engagement, even after, you know, polyscore, which detracts for repeated or clarified questions, like is claiming that the call quality has degraded.
00:17:04.799 --> 00:17:19.839
And so only until up until that like third semantic or critical error, which is where we see that real cliff in conversation quality, you know, the first couple are not free, but like not as disastrous as you might expect them to be.
00:17:20.000 --> 00:17:20.160
Yeah.
00:17:20.240 --> 00:17:23.920
So so wait, like the three plus on the handoffs going to 68.
00:17:24.000 --> 00:17:25.119
What is that one about?
00:17:25.440 --> 00:17:30.400
That one's, you know, our Wilson band there, our 95% confidence interval is quite wide.
00:17:30.480 --> 00:17:32.160
So I don't want to make any claims about that.
00:17:32.400 --> 00:17:36.720
You can see those little whiskers, but I think we just don't have enough data there.
00:17:36.880 --> 00:17:52.000
Because, and and again, this is also
like I was happy to see our cohort of three plus critical or semantic errors in in in our entire database of over 100,000 turns is in the like tens of calls.
00:17:52.240 --> 00:17:57.680
So, you know, we have very few calls have three plus meaning-changing turns.
00:17:57.839 --> 00:18:16.799
And the other thing to emphasize here, again, is like, you know, I wouldn't be I wouldn't be very scientific if I didn't like whip out the correlation is not causation, because other thing like you need to keep in mind is that things that affect conversation quality and handoff rate also affect things like word error rate and like critical error.
00:18:16.960 --> 00:18:23.279
So like if like the user is in a very noisy environment, like that's this external factor that's going to affect all of this stuff.
00:18:23.519 --> 00:18:30.799
And so you can't necessarily say, oh, the ASR was bad, so it led to this compounding effect of errors.
00:18:30.960 --> 00:18:38.960
You can say, you know, the user is driving 70 miles an hour down the highway blasting music, and it's not going to lead to a good experience.
00:18:39.119 --> 00:18:39.359
Yeah, yeah.
00:18:39.440 --> 00:18:42.079
And I mean, also like it's it's a very complex if, right?
00:18:42.160 --> 00:18:54.400
Because like to those conversations that get to have, I guess like the caveat, just knowing what I know about us, would be like we don't tend to proactively not hand off often if we get three like really bad turns, right?
00:18:54.559 --> 00:19:01.200
So in places where they get to accumulate, especially if there's very few of them, those are going to be those very, very long calls.
00:19:01.440 --> 00:19:15.759
And I think just anecdotally, you know, you kind of like the longer you stay in gameplay, maybe if you start the game with three lives, you know, like it refills like for every minute that you've stayed in the conversation, the sun cost of rage quitting and demanding to be handed off.
00:19:15.839 --> 00:19:18.960
You don't know if the human can seamlessly continue the conversation.
00:19:19.200 --> 00:19:30.480
We have many clients where the flow has been implemented to kind of like persist and fill out to CRM partially, and it's great when it does, but for the most part, and industry-wide, it's not usually implemented well.
00:19:30.559 --> 00:19:34.799
So you're kind of used to just like you get out of that process, and then it's like cool.
00:19:34.880 --> 00:19:35.680
So what was your name?
00:19:35.839 --> 00:19:37.200
What was your like social security number?
00:19:37.279 --> 00:19:39.039
And you're like, oh my god, right?
00:19:39.119 --> 00:19:42.400
Like that's that that's that C set is zero, right?
00:19:42.799 --> 00:19:48.079
So I think like the yeah, like if you if you manage to last that long, then people are pretty committed.
00:19:48.160 --> 00:19:58.400
So it makes sense that the handoff would would fall, because you know, you're about to achieve something, you will persist
even if your last name is Merkshic and you really have to, you know, somehow convey that to an automated thing.
00:19:58.640 --> 00:20:01.359
What was the word error rate in that aggregate analysis?
00:20:01.680 --> 00:20:05.039
Um, our headline word error rate for English was 6.4.
00:20:05.359 --> 00:20:05.680
Yeah.
00:20:05.839 --> 00:20:09.200
And for Spanish, which we had a smaller cohort of, was 8.8.
00:20:09.359 --> 00:20:09.519
Yeah.
00:20:09.839 --> 00:20:12.319
Which gives us a total of 6.7 weighted.
00:20:12.799 --> 00:20:13.039
Yeah.
00:20:13.279 --> 00:20:21.759
I mean, you know, when you look at like how people benchmark against, you know, like data sets, they're talking about 2%, 3%.
00:20:22.240 --> 00:20:34.319
And I think you know, it completely amidst the fact that like there's a different low pass filter on the phone, and would avoid that, you know, you might get several of them hitting the data and then just like lossy packets and stuff.
00:20:34.480 --> 00:20:38.240
So I really almost think that we should start producing a data set of those.
00:20:38.640 --> 00:20:39.759
And there are some that are.
00:20:40.000 --> 00:20:42.319
I remember like evaluating.
00:20:42.480 --> 00:20:56.079
We we used to collaborate with uh a bunch of kind of like car makers for their voice agents back back at Cambridge, and those in car word error rates would literally go around Cambridge in a car and record samples there.
00:20:56.240 --> 00:20:58.400
It would be between 20 and 40 percent.
00:20:58.640 --> 00:21:03.200
Now that's like car music, everything, and like an era of different models.
00:21:03.599 --> 00:21:05.839
But that's where people are calling customer service from.
00:21:05.920 --> 00:21:06.799
So, you know.
00:21:07.279 --> 00:21:07.680
Sure.
00:21:08.319 --> 00:21:10.400
It's very, very it's not a yeah.
00:21:10.799 --> 00:21:14.559
Anything else that you kind of like think in this one is worth kind of maybe sharing with the audience?
00:21:14.799 --> 00:21:28.559
Yeah, I mean, I guess the the last thing I wanted to share, which again I think is kind of intuitive if you think about it, is the difference between like in terms of handoffs versus quality, whether like the different effects that semantic errors have versus critical errors.
00:21:28.799 --> 00:21:35.759
And, you know, what's kind of interesting is that like we punish the semantic errors on quality a lot more, but they don't move the handoff needle very much.
00:21:36.000 --> 00:21:41.759
And that makes a lot of sense when you consider that like semantic errors are easy to sort of disambiguate around.
00:21:41.920 --> 00:21:51.599
And so if we're punishing them on the quality score because we're re-asking a question or we're asking a clarifying question, but they're not leading, but they have basically no effect on handoff rate.
00:21:51.839 --> 00:21:57.440
Whereas interestingly, you know, the critical errors have half of the quality degradation that semantic errors
do.
00:21:57.519 --> 00:21:59.839
And that's because critical errors are getting escalated quickly.
00:22:00.319 --> 00:22:01.519
Which is what you'd want to see.
00:22:01.680 --> 00:22:13.759
Like when the model like misses an entity and is and fails to look up an account or fails to retrieve an account or validate the user or something, and something has gone really wrong to like obstruct the path of the flow.
00:22:13.839 --> 00:22:19.519
Like we want to escalate to an agent quickly and in a way that does not degrade the quality.
00:22:19.680 --> 00:22:27.119
So you can see this like minus 0.27 effect on quality that critical errors have versus the minus 0.55 that semantic errors have.
00:22:27.279 --> 00:22:33.920
And again, that has a lot to do with how Polyscore scores quality and how it punishes clarification and disambiguation.
00:22:34.000 --> 00:22:42.160
But like again, I was happy to see that there's not this huge quality degradation around critical errors, which shows that we are escalating appropriately at the right times.
00:22:42.400 --> 00:22:43.200
Yeah, yeah, yeah, yeah.
00:22:43.519 --> 00:22:44.799
This is a bit of a chicken and egg.
00:22:44.960 --> 00:22:49.359
Like, I'm not sure if we can claim good science, but it's it's really interesting.
00:22:49.519 --> 00:23:00.400
I mean, it's also like the the whole like you know, confirming stuff and like, you know, how you attribute that to quality if you, you know, just kind of like the best call is what's your name?
00:23:00.480 --> 00:23:01.279
What's your date of birth?
00:23:02.319 --> 00:23:10.400
Ideally, I'm an AI oracle, I get everything right, and we don't go through the awkward like S-H-O-U-L-S-O-N, right?
00:23:10.720 --> 00:23:11.359
Whatever.
00:23:11.759 --> 00:23:16.240
I mean, that's just like really not there, not there yet in terms of good design.
00:23:16.319 --> 00:23:20.720
But if you correct that, you know, you're doing it with a human just the same as you are with a machine.
00:23:20.960 --> 00:23:21.200
Right.
00:23:21.279 --> 00:23:37.039
I mean, that's why that's why I kept trying to emphasize that like a lot of this, not an artifact, but a result of the way that Polyscore works, which is that it tends to punish that kind of like sort of the opposite of what you're saying, which is where we can't, we don't just like asked an answer to every single question.
00:23:37.359 --> 00:23:50.160
So, you know, we should interpret conversation quality through that lens and understand that a lot of these cases where we're seeing reduced quality, they're still navigating the conversation in a very natural and human-like way.
00:23:50.400 --> 00:23:53.039
And so that's why it's helpful to know sort of what that score consists of.
00:23:53.200 --> 00:23:53.279
Yeah.
00:23:53.440 --> 00:23:58.720
Okay, so maybe like if we break out a bit from just, you know, like the analysis itself.
00:23:58.960 --> 00:24:09.519
You know, if you could have like one wish of speech recognition, like one thing that it could do better to make dialogue systems a whole lot better, like what is the thing that is like most annoying in practice?
00:24:09.839 --> 00:24:20.079
I mean, I think that one thing that we encounter a ton in practice is the difference between in performance, between models that perform really well on short turns and models that perform well on longer turns.
00:24:20.400 --> 00:24:24.079
And because that's so unpredictable in the conversation, like you don't know necessarily.
00:24:24.319 --> 00:24:31.519
Like you can you can sort of predict if you're asking a yes or no question, you can imagine that the utterance is the response is probably gonna be short, but you don't know.
00:24:31.839 --> 00:24:43.599
And it's frustrating to have to configure different models on every single turn based on whether you're expecting a shorter or longer response, whether you're expecting a yes-no versus like kind of numeric or more value-driven response.
00:24:43.839 --> 00:24:45.759
So that's super annoying in practice.
00:24:46.160 --> 00:24:59.279
And then I think also just like the reason that these audio models perform so well and we can treat them as the gold standard probably has a lot to do with the way that they are being prompted in addition to just returning sort of blind transcriptions.
00:24:59.440 --> 00:25:08.559
Um, like they get to see the conversation context, they get text context, and they evaluate and they understand the audio tokens in the context of all of that text.
00:25:08.720 --> 00:25:12.319
And so I think like that's really the direction that this has to go, probably.
00:25:12.799 --> 00:25:13.119
100%.
00:25:13.519 --> 00:25:17.519
I mean, maybe like to take a quick digression that is very like topical.
00:25:17.680 --> 00:25:19.839
Have you seen the new like GPT Live?
00:25:20.079 --> 00:25:20.240
No.
00:25:20.400 --> 00:25:22.880
Oh, oh, I saw I saw their like little promo video for it.
00:25:22.960 --> 00:25:23.119
Yes.
00:25:23.599 --> 00:25:23.759
Yeah.
00:25:24.079 --> 00:25:28.559
I find it really interesting because you know, I think that talking about short terms, the quick yeses and no's.
00:25:28.720 --> 00:25:37.119
I think you know, it's not as much our world and that we're really focused on like real conversations that drive like business and and and customer needs.
00:25:37.359 --> 00:25:43.200
But they have this one where like let's say that you're talking and I'm going, yes, yes, yes, no.
00:25:45.279 --> 00:25:57.519
It's clear that someone has just like fixated on the problem and really wanted to show that like full duplex, like I'm speaking while listening, and kind of like inputs are coming in and out, which we as humans do, and when we do, we just tend to be annoying, right?
00:25:57.759 --> 00:26:15.519
I think I was curious about how you felt about that piece because I think like you know, like Raven Omni and our you know internal voracious appetite for it, amazement at
like everything that it can do, both in terms of understanding using all that context to just you know fix problems that we never could fix before is one thing.
00:26:15.680 --> 00:26:17.599
But how do you feel about the full duplex piece?
00:26:17.759 --> 00:26:20.480
Do you think it's even good as a feature for a conversational system?
00:26:20.799 --> 00:26:21.680
Yeah, I do.
00:26:21.839 --> 00:26:31.519
Like, I think I can't count the number of times I've seen like a delay in a barge in taking effect such that we only hear the tail end of what someone said and then totally misinterpret it.
00:26:31.599 --> 00:26:44.079
Like I think, I think just in terms of at least like responsiveness to barge in and interruption, it's necessary to have like maybe that's not full duplex, like we just need to know whether they're speaking or not, not necessarily what they're saying.
00:26:44.400 --> 00:26:46.799
But yeah, yeah, I I don't know.
00:26:46.880 --> 00:26:53.039
I would want to think about it more, but I feel I have to feel like it's necessary because it's such a key part, as you said, of like the way humans navigate conversations.
00:26:53.200 --> 00:27:09.279
Yeah, I mean, I feel like you know, they've done one UX thing, you know, a grave, grave, grave, grave crime there, which is because it's built on Chat GPT, which is used to, you know, sociopathically uh inducing more more more interaction from you, right?
00:27:10.079 --> 00:27:17.680
It is built and designed for you to interrupt it rather than to have a natural flow of back and forth conversation, right?
00:27:18.640 --> 00:27:25.680
Because of that, I kind of feel like they've taken for granted that that should be the interaction mode between two parties, right?
00:27:25.759 --> 00:27:30.400
That I just speak and that you will interrupt me, and that's why we have the whole fascination with bargin and stuff.
00:27:30.480 --> 00:27:31.519
Although bargeing matters, right?
00:27:31.599 --> 00:27:35.519
If I'm asking a question, you go, yep, and I don't hear you, like that's obviously bad, right?
00:27:35.680 --> 00:27:44.799
But I feel like now it's almost like turning into a separate, you know, like academic discipline of just like, and look, like this model is like taking inputs, producing outputs.
00:27:44.880 --> 00:27:52.880
Like, we as humans, if we're talking, and if you try to like slot something in, like I don't really, for the most part, one want you to do it, right?
00:27:53.039 --> 00:27:56.240
Like, sure, if it's a critical piece of information, you should interrupt me.
00:27:56.559 --> 00:28:05.519
But I think like for the most part, we see people react very weirdly when they get interrupted by AI because they just feel like they're not being heard, right?
00:28:05.839 --> 00:28:24.000
Yeah, yeah, no, and I I I see what you're saying about how it feels like it feels like they're excusing themselves for the way that the model is just going to monologue at you and like not actually design information delivery in a way that like makes it makes you not want to interrupt the model as much.
00:28:24.160 --> 00:28:28.640
And just saying basically, like, we're not concerned with that problem, like we'll just let you interrupt.
00:28:28.799 --> 00:28:31.440
Yeah, and as a conversation designer, that does hurt.
00:28:32.160 --> 00:28:32.480
Fair.
00:28:32.640 --> 00:28:35.119
Well, on that note, I think we I think we're out of time for today.
00:28:35.279 --> 00:28:36.880
But Oliver, always a pleasure.
00:28:36.960 --> 00:28:38.400
And always a pleasure.
00:28:38.640 --> 00:28:42.400
Please, you know, like, share, subscribe, and we'll see you on the next one.