Full Transcript for AI Gets Eyes and Ears | LiveKit’s Russ d’Sa

its portfolio. 00:04:23.415 --> 00:04:30.886 And this is, causing a lot of large tech companies, people adjacent to search and AI to swarm the scene. 00:04:31.125 --> 00:04:33.016 Literally, somewhat like Vultures 00:04:33.238 --> 00:04:34.228 Yeah, like vultures. 00:04:34.406 --> 00:04:36.776 Immediately buy, the Chrome browser. 00:04:36.776 --> 00:04:38.216 Now, why is this important? 00:04:38.216 --> 00:04:45.005 Obviously, we all know Google is a large company, very large reaching tech portfolio powers, a lot of the modern world that we live in. 00:04:45.334 --> 00:04:45.624 Yeah. 00:04:45.966 --> 00:04:50.495 when you talk about a fundamental tool like Google Chrome, which is almost ubiquitous now. 00:04:50.511 --> 00:04:53.600 With accessing the internet, there's a lot at stake. 00:04:53.661 --> 00:04:55.581 It's a large user base. 00:04:55.740 --> 00:05:04.440 and this is happening at a critical time when we're completely re-evaluating what it even means to go online and use an application or search for information. 00:05:04.680 --> 00:05:07.081 And the ways in which people are doing this are. 00:05:07.456 --> 00:05:08.565 Kind of flipping on its head. 00:05:08.805 --> 00:05:11.233 You have a AI that's kind of coming into search. 00:05:11.293 --> 00:05:13.353 You know, we've been covering this a lot on the pod. 00:05:13.353 --> 00:05:23.028 We've been talking about how, there's these encroaching wars between going to chat GPT to search a query and going to Google and getting your AI generated responses. 00:05:23.309 --> 00:05:25.678 they're both playing in each other's ball yard. 00:05:25.694 --> 00:05:37.341 So all of that is to say, is that you've got companies like Yahoo, perplexity, DuckDuckGo, all lining up, OpenAI of course, wanting to obtain, some of this for themselves. 00:05:37.401 --> 00:05:39.401 So Ben, what do you think of, this circumstance? 00:05:39.668 --> 00:05:39.809 Yeah. 00:05:39.809 --> 00:05:47.531 Well first of all, I, I lo as someone who used to work for Yahoo, I love that they're trying to find a way to make sure they remain relevant, uh, in the AI era. 00:05:47.930 --> 00:05:50.151 and also I just wanna say, I just want to have it out there. 00:05:50.151 --> 00:05:52.161 I would also like to buy Chrome. 00:05:52.281 --> 00:05:55.490 I might end up being like DuckDuckGo and not actually being able to afford it. 00:05:55.490 --> 00:05:56.511 But that's besides the point. 00:05:56.511 --> 00:06:00.110 I would like to buy it if that's possible, but personally. 00:06:00.223 --> 00:06:01.343 have to talk with them and ask. 00:06:01.776 --> 00:06:01.956 Yeah. 00:06:02.735 --> 00:06:10.865 But personally, you know, I would love to see it hosted like under a foundation similar to Mozilla, but it's probably like an unreasonable thing to expect. 00:06:11.146 --> 00:06:20.505 and maybe they'll become like an independent company, but it really does not surprise me at all that we have all these vultures circling to scoop it up

once Google is forced to sell it. 00:06:20.536 --> 00:06:25.466 I mean, I think everyone would like to own Chrome, so it's kind of, it's like, yeah. 00:06:25.466 --> 00:06:26.995 You and everyone else, thank you. 00:06:27.206 --> 00:06:36.855 I think too is that the key thing here is that the tool we're gonna use to search or go online and go to websites, I think the tool we're gonna be using in a few years from now probably hasn't even been invented yet. 00:06:36.855 --> 00:06:37.185 Yeah. 00:06:37.353 --> 00:06:39.713 and the way we access it, completely reinvents itself. 00:06:39.713 --> 00:06:45.398 I. frequently the browser wars like that used to be a thing, and 00:06:45.425 --> 00:06:45.846 Yeah. 00:06:45.997 --> 00:06:47.648 which was going to emerge ahead. 00:06:47.648 --> 00:06:52.600 You got like, Mosaic, then you had Netscape, then you had Internet Explorer, then you had Firefox and you had Chrome. 00:06:52.600 --> 00:06:55.571 And then Chrome became huge and you know what's next? 00:06:55.571 --> 00:06:57.550 I don't know, but I'm sure it'll be something different. 00:06:58.059 --> 00:07:02.829 Yeah, and I would definitely love to hear more about Open AI's vision about an AI first browser. 00:07:02.858 --> 00:07:05.199 'cause that was something they mentioned as a part of this. 00:07:05.199 --> 00:07:07.959 So like it, it's probably gonna be a cool experience. 00:07:07.959 --> 00:07:09.278 I would really love to see that happen. 00:07:09.329 --> 00:07:12.418 But, and this wasn't the only thing that we saw in the a AI wars, right? 00:07:12.418 --> 00:07:15.598 It wasn't the, we have another story involving Cursor and Microsoft, 00:07:15.649 --> 00:07:16.459 oh, yes we do. 00:07:16.459 --> 00:07:20.026 So, there's something that happened very recently on the VS. 00:07:20.026 --> 00:07:22.665 Code extension for c and c plus plus. 00:07:22.915 --> 00:07:26.096 this is an extension that powers a lot of the entire, like the tooling within VS. 00:07:26.096 --> 00:07:27.355 Code to work within that. 00:07:27.360 --> 00:07:30.901 That language as a very popular part of the ecosystem, right? 00:07:31.201 --> 00:07:35.060 But recently, Microsoft, released agent mode for GitHub copilot. 00:07:35.151 --> 00:07:41.637 And so this is GitHub copilot starting to move into things like cursors territory, going back to our turf wars. 00:07:41.817 --> 00:07:43.266 We just covered with search. 00:07:43.266 --> 00:07:55.737 and, and with ai you have this happening as well with, code assistance and so by, blocking the VS code extension from working in things like cursor, which is entirely within like their realm and their right. 00:07:55.766 --> 00:07:58.107 'cause this is an extension they have developed and if they 00:07:58.250 --> 00:07:58.540 Yeah. 00:07:58.617 --> 00:07:59.966 for their own applications. 00:08:00.201 --> 00:08:11.031 They can, they haven't stopped cursor from using or adopting any, a different type of tool within the ecosystem because there's lots of open source alternatives and they have since done so. 00:08:11.091 --> 00:08:11.541 Right. 00:08:11.541 --> 00:08:18.482 So it really shows that, you know, Microsoft might try to close a door to make less of an opportunistic space for their competitor. 00:08:18.482 --> 00:08:25.442 but because they also control so much of the ecosystem, they get to make moves like this that are quite fascinating that other companies can't quite do. 00:08:26.160 --> 00:08:32.879 Yeah, I think the AI space is gonna get really cutthroat and we're just gonna see this competition heat up, especially with how much money. 00:08:33.179 --> 00:08:33.990 Is on the line. 00:08:34.269 --> 00:08:38.289 also given that nobody's established a defensible moat in this space yet. 00:08:38.289 --> 00:08:41.799 So everyone is prone to being disrupted that operates here. 00:08:42.279 --> 00:08:51.950 And I doubt it's a coincidence that Microsoft Times this blocking with, or blocking the VS code extensions with the release of their agentic mode for copilot. 00:08:52.460 --> 00:08:58.639 And I think, on one hand it's a lesson that all startups should learn, like it's cursor's fault for relying on a competitor's product. 00:08:58.970 --> 00:09:03.000 To build upon, I also understand why they made that decision it's a tiny company. 00:09:03.000 --> 00:09:04.590 They don't have very many employees. 00:09:04.590 --> 00:09:07.195 They bootstrap their way to, to where they are. 00:09:07.404 --> 00:09:10.615 So it's a natural that they're gonna encounter growing pains like this. 00:09:10.664 --> 00:09:17.565 and it also makes sense to me why they would adopt open source as the solution, you know, because it's, a lot lower barrier to entry. 00:09:18.076 --> 00:09:19.246 Yeah, completely. 00:09:19.784 --> 00:09:19.995 Yeah. 00:09:19.995 --> 00:09:24.284 So this week in Vibe coding, we have yet another story on vibe coding. 00:09:24.284 --> 00:09:25.634 So what, what do we have, Andrew? 00:09:26.356 --> 00:09:28.937 Okay, so this is a, a really great one that I read. 00:09:28.937 --> 00:09:30.376 I also shared it on LinkedIn. 00:09:30.447 --> 00:09:31.947 it's from, comes from Pete Hodgson. 00:09:32.006 --> 00:09:39.907 and it talks about a prompting method within an agentic coding tool like Cursor or GitHub, copilot agent mode, called chain of. 00:09:40.001 --> 00:09:40.572 Vibes. 00:09:41.190 --> 00:09:41.480 Nice. 00:09:41.591 --> 00:09:45.101 thesis here is that AI isn't ready for unsupervised coding. 00:09:45.201 --> 00:09:55.642 And Pete found that a workflow called Chain of Vibes lets him maximize how much he can lean on the AI for coding while still having a really good rigid process with a human in the loop. 00:09:56.062 --> 00:09:59.542 the approach consists of driving the AI through a series of. 00:09:59.591 --> 00:10:12.221 Very separate and very distinct, but fully autonomous vibe, coding changes that build upon each other, and between each of them, there's a

thorough human review to make sure that things haven't gone off the rails. 00:10:12.221 --> 00:10:19.652 I, myself, have had a lot of luck, with this kind of technique, and, and definitely encourage anyone experimenting, with agentic coding to check that out. 00:10:19.711 --> 00:10:21.572 But Ben, have you, uh, tried anything like that? 00:10:21.909 --> 00:10:29.500 Yeah, I know you've been using Agentic coding for a few weeks now, and it's been fascinating to me just seeing secondhand, like learning from you what you've been doing with it. 00:10:29.830 --> 00:10:39.460 And as I mentioned earlier, like I just got into like Vibe coding or Agentic coding this week with Cursor, and it's frankly been an eye-opening experience. 00:10:39.490 --> 00:10:46.924 You know, I still haven't figured out really how to maximize the use of things like rules to facilitate better agentic coding. 00:10:46.924 --> 00:10:50.274 Like this is kind of what Pete is describing in this article, right? 00:10:50.274 --> 00:10:57.024 It's, it's like having more discreet chunks of work that have rules that guide it in a human overseeing the process. 00:10:57.291 --> 00:11:02.961 but it really does like mirror a lot of the approach I've taken with AI just even outside of coding. 00:11:03.278 --> 00:11:03.697 Mm-hmm. 00:11:03.725 --> 00:11:13.926 focus has always been on breaking down complex work into a series of smaller steps that can easily be handled by individually purpose-built gpt. 00:11:14.255 --> 00:11:33.489 And the reason I do this is 'cause it improves consistency, but it also makes it much easier for you to inject human judgment into the loop to make sure that the output from one GPT is good enough for the input of the next, and there's a couple of other tips that really stood out to me, that are things that I've already adopted, clear context frequently. 00:11:33.519 --> 00:11:37.599 Like personally, I treat almost every interaction with a GPT is ephemeral. 00:11:37.629 --> 00:11:42.999 Like the moment that I'm done with it, it, it is deleted from existence, or I never think of it again. 00:11:43.328 --> 00:11:49.798 Or if I feel like it's going off the rails, like I just start over from, from scratch with a fresh, GPT prompt. 00:11:50.624 --> 00:11:50.833 Yeah. 00:11:50.833 --> 00:11:53.563 And then he also mentioned using the right AI for each task. 00:11:53.563 --> 00:12:06.514 Like when you start breaking down these complex chains into discrete tasks, you might end up finding out that one model works for a certain task much better than another model, but then for a different task, the reverse is true. 00:12:06.514 --> 00:12:08.619 So, you know, I really like this approach. 00:12:08.619 --> 00:12:14.239 Really makes it easy to set yourself up for success, but also experiment with the tools that are out there. 00:12:14.931 --> 00:12:26.846 Yeah, this article does a really good job of kind of setting the perspective correctly of what you should be thinking when you

open a tool like Cursor or when you go to Amazon Q, in your command line. 00:12:26.910 --> 00:12:28.681 when you open an IDE now. 00:12:28.686 --> 00:12:30.365 That has these capabilities. 00:12:30.635 --> 00:12:33.666 You know, you're putting on more of a product manager hat than 00:12:33.719 --> 00:12:34.139 Mm-hmm. 00:12:34.235 --> 00:12:35.316 a developer hat. 00:12:35.615 --> 00:12:43.596 Instead of focusing on the exact code lines that you're going to write, you're instead focused on what is the purpose of what I'm building? 00:12:43.836 --> 00:12:47.735 What's the intended impact and what do I need in order to get. 00:12:47.735 --> 00:12:48.171 There. 00:12:48.201 --> 00:12:57.831 And then your, goal becomes really as that product manager to rally all of that information together and get everyone on board. 00:12:57.831 --> 00:13:03.171 And when I say everyone, I mean all of these ephemeral little chats that you're about to have, like 00:13:03.208 --> 00:13:03.499 Yeah. 00:13:03.561 --> 00:13:10.270 just described with the IDE, and, and another guide that, or two sets of guides actually that go really well in depth with. 00:13:10.291 --> 00:13:12.091 This, they're from Joffrey Hunter. 00:13:12.091 --> 00:13:12.870 They're on his website. 00:13:12.870 --> 00:13:15.571 We've actually linked them before in the download. 00:13:15.760 --> 00:13:19.307 so we'll be sure to link back to those as well if, if you haven't, uh, checked those out. 00:13:19.567 --> 00:13:24.697 it breaks it down into a really, repeatable method similar to what Pete Hodgson is kind of touching on here. 00:13:24.697 --> 00:13:32.897 and you start really by, understanding what it is that you're building and writing very clear documentation for yourself and for the agent to use. 00:13:33.197 --> 00:13:39.527 then you spend some time writing the bounds and the rules and the constraints of what you're building in cursor. 00:13:39.527 --> 00:13:42.167 This might even mean writing rules for the project you're doing. 00:13:42.496 --> 00:13:45.886 And then finally, you're gonna iterate across all of those ephemeral sessions. 00:13:46.067 --> 00:13:48.226 You're going to reference that well-written spec. 00:13:48.346 --> 00:13:51.407 You're gonna point to the, to those strong well-defined rules. 00:13:51.557 --> 00:13:53.567 And you're gonna iterate until you get the results you want. 00:13:53.626 --> 00:13:59.746 Reviewing every step of the way to make sure that you don't, of course, get slop squatted, like we mentioned earlier. 00:13:59.956 --> 00:14:11.667 So, really kind of changes your whole perspective about what you're doing when you go into the IDE and it, it really makes me wonder, from to, from to our listeners, you know, are you experimenting with a agentic coding right now too? 00:14:11.667 --> 00:14:15.897 I think a lot of people are trying it out for the first time, and we want to hear about your experiences. 00:14:16.136 --> 00:14:23.716 I'm personally posting on LinkedIn every week, and, and learning with, others in the open about how to use things like Cursor, Amazon Q. 00:14:23.746 --> 00:14:25.517 Windsurf, all these other tools. 00:14:25.697 --> 00:14:28.356 So come learn with us and, share your experiences too. 00:14:28.955 --> 00:14:40.014 Yeah, everyone's learning right now, and this is a good time to remind our audience that we have a Substack newsletter where we share a lot of these stories as well as a, a LinkedIn community where we're sharing this as well. 00:14:40.414 --> 00:14:48.245 so if you wanna be the first to find out about all these great guides and, and get these tips delivered straight to your inbox, make sure you're subscribed to our substack. 00:14:48.245 --> 00:14:49.745 It's the best way to get all this news. 00:14:50.735 --> 00:14:57.695 So we got a bonus story too, and I wanted to include this one because I like to walk and I use crosswalks frequently. 00:14:58.414 --> 00:14:59.345 What is the bonus story? 00:14:59.345 --> 00:15:00.304 This week, Andrew? 00:15:00.892 --> 00:15:02.912 Okay, so this is a really weird one. 00:15:02.989 --> 00:15:22.859 recently in Seattle, some of the crosswalks appeared to have been, been hacked, and people at first were not quite sure how this happened, but when you would go to the crosswalks on some intersections and press the button, instead of it telling you to wait or to cross, uh, instead you were greeted by a deep faked voice of Jeff Bezos or Elon Musk or other. 00:15:22.908 --> 00:15:26.989 Tech billionaires and they'd be talking to you about all sorts of stuff. 00:15:27.229 --> 00:15:34.548 And this is a neat assortment of weird technology all coming together to create what was ultimately a viral moment. 00:15:34.548 --> 00:15:39.229 Because when you boil it down, it's a very simple vulnerability actually, that they took advantage of. 00:15:39.509 --> 00:15:42.839 it turns out that a crosswalk is a lot like a router. 00:15:42.839 --> 00:15:47.879 It comes with factory settings, including a factory login over a technology like Bluetooth. 00:15:48.011 --> 00:15:56.241 so it really was probably just as simple as somebody who knew the model of the crosswalk and had their Bluetooth open and was able to connect with a default. 00:15:56.501 --> 00:15:58.692 Password and username to it. 00:15:58.692 --> 00:16:04.731 And then once they're in, they could upload, you know, an MP3 of what it should play, when the crosswalk button is pressed. 00:16:04.731 --> 00:16:14.922 But what makes this a very impressive and very interesting, usage of technology is ultimately They used, modern tools like AI to deep fake these voices. 00:16:15.072 --> 00:16:27.221 They used very modern tools and the things that we're all worried about in terms of like misinformation to rapidly create a high impact message that then hacked something more than the crosswalk. 00:16:27.402 --> 00:16:28.721 It hacked social media. 00:16:28.777 --> 00:16:30.726 Because it became a

viral story. 00:16:30.907 --> 00:16:36.787 Suddenly you had people going out to these crosswalks and recording it and sharing it on X and on Blue Sky and all these places. 00:16:36.787 --> 00:16:38.407 look what this crosswalk is saying. 00:16:38.527 --> 00:16:40.356 And then the whole world was talking about it. 00:16:40.356 --> 00:16:48.277 Now we're talking about it, and that's ultimately what this kind of hacking in the wild is all about, is making a message known using the tools available. 00:16:48.576 --> 00:16:50.287 So quite a fascinating one. 00:16:50.287 --> 00:16:51.037 What do you think Ben? 00:16:51.409 --> 00:16:51.649 Yeah. 00:16:51.649 --> 00:16:56.029 So first of all, I want to just say put this out there, like safety first. 00:16:56.029 --> 00:17:02.600 Like, if you're gonna do something like this, please don't screw up the crosswalk so that like a disabled person can't use it. 00:17:02.629 --> 00:17:06.749 Like, It's already dangerous enough being a pedestrian in many American cities. 00:17:07.209 --> 00:17:09.278 but you know, this is a tale as old as time. 00:17:09.278 --> 00:17:15.489 Like somebody's setting up a piece of digital equipment and just not bothering to change the default password. 00:17:15.489 --> 00:17:17.898 Like, come on, that's, it's like security 101. 00:17:17.919 --> 00:17:20.858 Like nobody should ever be doing that in this day and age. 00:17:21.608 --> 00:17:25.596 Um, but, the AI brings a, a, a very new twist to this and. 00:17:25.991 --> 00:17:43.201 You and I have been talking about this concept of code as art, as code gets more accessible to more audiences, through ai, I really get the sense that we're gonna see more emergences of people just using AI and using code and unique ways that we had never really considered before. 00:17:43.261 --> 00:17:48.342 Again, going back to like our lop squatting example, there's lots of unknown unknowns out there. 00:17:49.047 --> 00:17:51.626 AI's gonna keep opening up New attack vectors. 00:17:51.896 --> 00:17:53.156 Stay tuned to Dev Interrupted. 00:17:53.156 --> 00:17:56.118 We're gonna keep you on top of all the stories as they develop. 00:17:56.118 --> 00:17:57.884 So Andrew, who's our guest this week, 00:17:58.500 --> 00:17:59.010 Oh yes. 00:17:59.010 --> 00:18:01.050 This week's guest is a really interesting one. 00:18:01.260 --> 00:18:09.421 We have Russ D'sa, the CEO and Co-founder of Live Kit, and he's really pulling back the curtain on where the future of AI is heading. 00:18:09.421 --> 00:18:11.760 And I don't mean just in the chat bots that we, I. 00:18:11.989 --> 00:18:12.108 I. 00:18:12.526 --> 00:18:19.605 Interact with now, or even the agents that people are building and everyone wants to talk about, but instead interacting with us on our terms. 00:18:19.756 --> 00:18:28.336 Being able to see, hear, feel, and understand from the real world using multimodal inputs and the impact of this technology is profound. 00:18:28.695 --> 00:18:31.605 Talk about saving lives even on 9 1 1 calls. 00:18:31.695 --> 00:18:32.476 So stick around. 00:18:32.476 --> 00:18:33.496 You don't wanna miss this one. 00:18:34.755 --> 00:18:36.555 Are your code reviews slowing you down? 00:18:36.960 --> 00:18:45.000 With LinearB automations, you can transform the way your teams review code with automatic AI powered PR descriptions and code reviews. 00:18:45.029 --> 00:18:48.180 Your developers get instant feedback on every pull request. 00:18:48.750 --> 00:19:00.599 Combine that with smart AI orchestration and you can cut the noise and boost your productivity LinearB AI flags bugs suggests improvements and keeps your team focused on what really matters, building great software. 00:19:00.960 --> 00:19:04.890 Say goodbye to review fatigue and hello to faster, higher quality delivery. 00:19:05.384 --> 00:19:10.545 Head over to LinearB.io to learn more about incorporating AI into your code reviews. 00:19:12.996 --> 00:19:13.986 Hey, everyone. 00:19:14.016 --> 00:19:19.026 Joining us today is Russ d'Sa, the co-founder and CEO of LiveKit. 00:19:19.336 --> 00:19:29.185 Where they're building the nervous system for multimodal technology that interacts with the world through voice, vision, and real time understanding and what they're building at LiveKit. 00:19:29.205 --> 00:19:36.135 It's not just about its applications in ai, it's about how all modern systems are starting to sense, interpret, and act. 00:19:36.425 --> 00:19:41.675 In real time, and to give our listeners a sense of this technology's impact just right off the bat. 00:19:41.855 --> 00:19:45.935 You know, right now LiveKit delivers voice to millions of chat CPT users. 00:19:46.145 --> 00:19:50.826 It even saves lives on 9 1 1 emergency calls, and we're gonna talk a bit about that. 00:19:51.185 --> 00:19:55.026 It's also used in live streaming and robotic systems across the world. 00:19:55.415 --> 00:20:01.175 And in today's chat with Russ, we're diving into the major shift that's happening in software development that. 00:20:01.175 --> 00:20:07.865 It's currently underway and learning how your team can embrace those multimodal opportunities in your work. 00:20:08.046 --> 00:20:09.905 So Russ, welcome to the show. 00:20:10.739 --> 00:20:11.759 Thank you so much, Andrew. 00:20:11.818 --> 00:20:15.719 It's uh, lovely to be here and, hi to, to all of the listeners as well. 00:20:16.451 --> 00:20:17.320 It's great to have you. 00:20:17.320 --> 00:20:22.594 And you know, Russ, you and I had such a great, chat and preparation for this, you know, your take on technology. 00:20:22.653 --> 00:20:23.824 It was really refreshing. 00:20:23.824 --> 00:20:24.784 It stuck with me. 00:20:25.023 --> 00:20:33.413 I've been really excited to bring you onto the show, especially this idea that, you put in my head about how we're outgrowing a traditional request response model. 00:20:33.864 --> 00:20:36.653 And I wanna dive into that a bit in our conversation today. 00:20:36.653 -->

00:20:41.334 But first, I think to benefit our listeners, let's zoom out a little bit 'cause we've been talking about. 00:20:41.334 --> 00:20:44.743 The shift into a real time multimodal world. 00:20:45.434 --> 00:20:53.035 there is this huge shift happening with technology and not something that I foresaw when we started LiveKit. 00:20:53.055 --> 00:21:02.625 but then something that kind of ended up happening, for me, that was what we started to do with open AI and, building voice mode with them for chat GPT. 00:21:03.345 --> 00:21:07.335 And it was at that moment that I kind of like looked up a little bit and I was like, well, where does all this go? 00:21:07.335 --> 00:21:07.575 Right? 00:21:07.575 --> 00:21:08.654 Like, this is a really cool feature. 00:21:08.654 --> 00:21:13.565 I can talk to an ai, you know, using my voice, but where is all of this headed? 00:21:13.565 --> 00:21:20.075 And I think the thing that I realized, back then, this was like August, 2023. 00:21:20.075 --> 00:21:31.595 Is that the internet or let's say the web really was not built for multimodal real time audio video. 00:21:31.595 --> 00:21:31.924 Right. 00:21:31.924 --> 00:21:44.105 It When you type into the browser, http colon slash slash like what you're typing in is you're typing in a protocol there and that protocol HTTP stands for Hyper Text Transfer Protocol. 00:21:44.434 --> 00:21:50.055 What you're doing is you're taking text and you're transferring it, from one computer to another computer. 00:21:50.474 --> 00:21:57.944 It's not hyper voice transfer protocol or hyper video or vision transfer protocol. 00:21:57.944 --> 00:21:59.174 It's hyper text. 00:21:59.565 --> 00:22:07.934 And so, you know, just that word is telling like the way transferring high bandwidth data, like audio and video over a network. 00:22:08.605 --> 00:22:16.404 It fundamentally requires a different approach, a different paradigm than transferring text over the network. 00:22:16.734 --> 00:22:20.724 The way that we interact with computers today, for the most part, is we open a browser. 00:22:21.005 --> 00:22:22.265 not every single use case, right? 00:22:22.265 --> 00:22:25.144 Like we use Zoom sometimes and we use Discord and all of that stuff. 00:22:25.144 --> 00:22:30.184 So it's, we are definitely transferring audio and video for some of these applications that we use. 00:22:30.184 --> 00:22:35.855 But predominantly what you're doing is you're in a web browser, you're navigating to a webpage, you are. 00:22:36.500 --> 00:22:42.700 Clicking, uh, into a form, you're filling out some fields, you're clicking submit, or other buttons on that page. 00:22:43.150 --> 00:22:44.380 and that. 00:22:44.990 --> 00:22:50.630 kind of paradigm or interaction model is the stateless web application model, right? 00:22:50.630 --> 00:22:57.589 So you click a button, some text is transferred to a server, somewhere that server receives that text. 00:22:57.589 --> 00:23:01.029 And what you want to do, it looks up who you are in a database. 00:23:01.329 --> 00:23:05.259 It has some side effects that are generated, so it runs through some business logic. 00:23:05.259 --> 00:23:08.200 Say that you are, um, trying to get a reservation at a restaurant, right? 00:23:08.559 --> 00:23:14.349 And it's looking up like, are there spaces available at that restaurant for your party size? 00:23:14.529 --> 00:23:30.339 And then if there are, then it's going to, put a record in the database for that time slot for that restaurant saying that you have the reservation and then it's gonna send back some text, that gets rendered on that, uh, that website that you're on saying you have booked your table, show up at this time. 00:23:30.963 --> 00:23:36.273 That's not really a latency sensitive application, and most of the applications on the web today are not. 00:23:36.374 --> 00:23:39.374 The way that you're going to interact with computers in the future, right. 00:23:39.374 --> 00:23:40.304 At a high level. 00:23:40.693 --> 00:23:46.499 What we're trying to build, I. I guess, you know, in society now is AGI, right? 00:23:46.709 --> 00:23:51.719 if we're trying to build AGI like what is AGI, in my opinion, I. 00:23:52.963 --> 00:23:55.064 Humans are tool builders, right? 00:23:55.064 --> 00:24:04.673 We create these things like hammers and nails and screwdrivers and planks of wood and all this stuff so that we can construct things and solve problems. 00:24:04.723 --> 00:24:09.933 And what we're building now, is, we're kind of building the ultimate tool, which is a tool builder. 00:24:09.963 --> 00:24:13.443 We're building ourselves right, the mirror in some ways. 00:24:13.443 --> 00:24:16.203 And, it's a computer that can like. 00:24:17.598 --> 00:24:19.219 Behave like a human being. 00:24:19.219 --> 00:24:20.628 Talk like a human being. 00:24:20.628 --> 00:24:24.618 You know, maybe in its ultimate form is indistinguishable from a human being. 00:24:25.152 --> 00:24:28.122 not trying to like do any scare fearmongering or anything like that. 00:24:28.122 --> 00:24:32.832 It's just like, it, it's a computer that very much behaves like in mimics a human being. 00:24:33.521 --> 00:24:37.751 And when computers were kind of these. 00:24:38.592 --> 00:24:40.001 as intelligent machines. 00:24:40.692 --> 00:24:44.801 We had to create something like a keyboard and a mouse and adapt our own behavior. 00:24:44.801 --> 00:24:50.172 I know, you know, you're probably a bit younger than me, but I had to take in school, right? 00:24:50.172 --> 00:24:52.781 Where I didn't need to learn how to type on this layout of keys. 00:24:53.021 --> 00:25:03.132 Like we have to adapt our behavior and learn how to give the computer information so that we can use it as a tool to help us right the bicycle of the mind. 00:25:03.521 --> 00:25:03.741 Yes. 00:25:04.460 --> 00:25:06.589 To use Steve Jobs's kind of phrasing. 00:25:06.769 --> 00:25:08.660 Now we're almost creating the mind itself.

00:25:08.660 --> 00:25:17.059 And if like you are, if you are gonna build a computer that is as smart as a human right, longer have to adapt yourself. 00:25:17.109 --> 00:25:19.630 The computer actually will adapt itself to you. 00:25:20.049 --> 00:25:33.059 And the way that you give information and communicate with other human beings is the way that you're probably gonna communicate and behave, or give information to that computer that is very human-like. 00:25:33.539 --> 00:25:35.609 and so humans use eyes, ears, and mouths. 00:25:36.049 --> 00:25:42.470 the computer of the future is going to use cameras, microphones, and speakers. 00:25:42.529 --> 00:25:42.799 Those 00:25:42.799 --> 00:25:43.579 are the equivalent 00:25:43.579 --> 00:25:47.450 sensors to kind of the human's, eyes, ears, and mouths. 00:25:47.869 --> 00:25:56.265 If you're going to build a computer that takes in natural human input and output in the same way or a similar way, you have to almost change. 00:25:57.365 --> 00:26:02.555 The way that you build applications for that computer, right, that run on that computer. 00:26:02.825 --> 00:26:11.865 It's not, where I click a button and then I wait, and then like a database kind of looks up information and some actions are taken, and then a response gets generated some number of seconds later. 00:26:12.484 --> 00:26:13.315 This is. 00:26:13.690 --> 00:26:17.049 sim more similar to how you and I are communicating with one another right now. 00:26:17.049 --> 00:26:18.819 Like you are constantly listening to me. 00:26:18.819 --> 00:26:20.109 I have your attention, 00:26:20.237 --> 00:26:20.656 Mm-hmm. 00:26:21.250 --> 00:26:26.500 figuring out whether I'm done speaking or not, or whether you should interrupt with the next question. 00:26:26.799 --> 00:26:27.910 You are you. 00:26:27.994 --> 00:26:34.444 Keeping this rolling context, growing in your mind of everything that I'm saying, and you're committing some of it to memory. 00:26:34.444 --> 00:26:37.984 And you're gonna come back to maybe something that I said or focus in on it. 00:26:38.285 --> 00:26:53.105 It's this constant connection and the data is streaming to you, throughout this entire time that we're talking and your processing it in your mind and also at the same time listening to new information that's coming to you From my mouth and through my movement. 00:26:53.494 --> 00:26:57.724 And so if you're gonna build applications for that future, they're gonna look just very different. 00:26:58.559 --> 00:27:21.329 So when we're talking about shifting, how people are looking at, are we gonna be using technology in the future, it's really a fundamental shift from getting outta the box that we put ourselves in, of using text and converting everything in our world to text, to talk to a machine, which is the traditional model, and more about opening it up and bringing, elevating the model or, or rather the computer. 00:27:21.934 --> 00:27:46.339 Up to our level, allowing it to participate in our own senses, have that same kind of understanding of the world, and then it unlocks, new use cases, more ability for that technology to have impact on our world and everything that you're building so far, of course there's the trend of, of what you're describing, of ultimately, a very intelligent machine is going to need very intelligent senses to interact. 00:27:46.545 --> 00:27:47.625 With our world. 00:27:47.934 --> 00:28:00.684 but along the way there are so many use cases as well that evolve and unfold once you bring sensory input into the conversation and you're sitting right at like the helm of that. 00:28:00.684 --> 00:28:04.565 You're seeing it, being applied and used in a lot of different places. 00:28:04.565 --> 00:28:13.545 What are you think the strongest or most impactful use cases, that you like to tell people about that this technology is transforming today? 00:28:15.067 --> 00:28:25.048 So my favorite use case is actually the one that I think would tie in very closely or most closely with what you were just saying about how. 00:28:26.038 --> 00:28:37.958 this kind of new paradigm or way you interact with the computer and what, kind of abilities it is imbued with now that it has kind of human level senses of vision and, uh, and hearing. 00:28:38.377 --> 00:28:42.357 I think it's self-evident in, this 9 1 1 use case. 00:28:42.387 --> 00:28:43.157 So LiveKit. 00:28:44.518 --> 00:28:48.837 Today is used by about 25% of 9 1 1. 00:28:48.837 --> 00:28:50.248 So 9 1 1 calls, right? 00:28:50.605 --> 00:28:50.994 Yeah. 00:28:51.367 --> 00:28:54.228 there's a company out there, called prepared. 00:28:54.228 --> 00:28:59.038 And what they do is they deploy, LiveKit server, our media server in. 00:29:00.163 --> 00:29:05.053 Dispatch centers around the country, so about a quarter of them, or maybe close to a third of them. 00:29:05.482 --> 00:29:16.130 And, you know, it was started by, three folks that, were college kids, at Yale, in a different, a very different kind of generation for me. 00:29:16.130 --> 00:29:31.086 So when I had just graduated college, bowling for Columbine had come out and it really kind of brought to light, this, tragedy that we have in this country around school shootings and safety, at these institutions. 00:29:31.086 --> 00:29:33.156 And, you know, that was for me. 00:29:33.156 --> 00:29:43.636 And then like, I think maybe 15 years later the founders of the, of this company, were at Yale and they're going through school shooter training and there's a school shooting happening. 00:29:44.896 --> 00:29:54.737 I think three times a week or something like that around the us It's a really shocking, uh, kind of amount, of this happening. 00:29:55.030 --> 00:30:04.090 And so they were going through this school shooting training, uh, and one of the things that they thought about was like, well, what if I could actually broadcast what's happening? 00:30:04.340 --> 00:30:19.830 live, like any student could take

out their phone and start to broadcast and provide situational data to the police or, the authorities that are coming to deal with and handle this issue and, you know, restore safety to the campus and. 00:30:20.296 --> 00:30:30.915 They were talking to one of the officers that were there on campus during the training about this thing that they were working on, and that officer said, well, you should go talk to the dispatch center in New Haven, uh, for 9 1 1. 00:30:30.915 --> 00:30:37.455 Because they have been built on the telephones telephone system from like the seventies or eighties. 00:30:37.756 --> 00:30:39.226 The technology hasn't changed. 00:30:39.226 --> 00:30:44.806 Like you still have to get on the phone and like tell them where you are and explain the situation and they don't have any eyes. 00:30:45.540 --> 00:30:55.621 Um, you know, they don't get GPS data streaming to them about location or anything like that, and so they've been dreaming of having this richer situational data. 00:30:56.070 --> 00:30:58.560 In the dispatch center right, for these 9 1 1 calls. 00:30:58.560 --> 00:31:04.721 And so they started, they did a pilot in New Haven and I think they did another one in Nevada. 00:31:04.721 --> 00:31:16.306 And within the first week, someone had a heart attack and called 911 The dispatch agent sent a text to their phone with a mobile URL. 00:31:16.546 --> 00:31:26.135 They tapped on, the user, who was calling, tapped on that URL, and they were in a mobile browser streaming audio and video and GPS data to the dispatch, agent and. 00:31:26.398 --> 00:31:34.489 they had someone hold their phone, and the dispatch agent coached them on delivering CPR to this person who had had a heart attack and was unconscious. 00:31:34.618 --> 00:31:35.848 and they saved that person's life. 00:31:35.848 --> 00:31:47.729 And now, every single week, a very similar story to this, where someone has a heart attack, and is revived through, a video call with, with a dispatch agent, through LiveKit, of course. 00:31:47.969 --> 00:31:50.128 so it's very rewarding and impactful for me. 00:31:51.159 --> 00:31:52.878 it saves a person's life every single week. 00:31:52.878 --> 00:31:53.429 And so 00:31:53.459 --> 00:31:53.848 Wow. 00:31:54.028 --> 00:31:57.909 you know, it's, but it's something that just was not possible. 00:31:58.209 --> 00:32:08.472 before we had this ability to really like stream audio and video on demand in real time with low latency, and kind of like teleport that dispatch agent to the scene of the emergency. 00:32:08.682 --> 00:32:11.143 So just an incredible use case, 00:32:11.505 --> 00:32:11.795 Yeah. 00:32:12.119 --> 00:32:12.690 Wow. 00:32:13.002 --> 00:32:14.742 of this new world that we are entering into. 00:32:15.251 --> 00:32:17.021 what an amazing, impactful story. 00:32:17.179 --> 00:32:24.048 like an incredible first usage, or rather like a, it goes to show how quick the impact of that came. 00:32:24.259 --> 00:32:28.818 Like

they put that in place and then it was immediately useful and immediately saving lives. 00:32:28.818 --> 00:32:35.659 So it really speaks to like this really big need in our world and a disconnect between how we live in it. 00:32:35.798 --> 00:32:38.618 And how we try to use technology to influence it. 00:32:38.868 --> 00:32:44.759 so it, it is really kind of hinting at here how life kit is bridging, things that before were not bridged. 00:32:44.759 --> 00:32:58.739 And with that you get these huge gains, like in this case, saving someone's life or in other cases responding to real time, like scenarios like being able to use this technology to drive that kind of good, I'm sure is, has been really rewarding. 00:32:59.108 --> 00:33:05.509 the way that you describe it too, like I love your passion for how, people would use this technology and what it would mean. 00:33:05.778 --> 00:33:13.989 And I also like how you're talking about the evolution of, you know, we took our brains and we turned 'em into text in order to use computers, and now we need to, bring the computers to us. 00:33:13.989 --> 00:33:16.479 We need to help them understand the world that we live in. 00:33:16.538 --> 00:33:20.019 And it reminds me so much of like when mobile. 00:33:20.233 --> 00:33:24.314 Really came on the scene and about how everything tried to get shoved into mobile. 00:33:24.314 --> 00:33:24.763 Right. 00:33:24.794 --> 00:33:27.614 And we tried to figure out how do we use everything on mobile? 00:33:27.614 --> 00:33:29.334 And, that's a growing pain. 00:33:29.463 --> 00:33:33.153 we turned things that did and didn't work into mobile experiences. 00:33:33.153 --> 00:33:34.713 We learned not to just make. 00:33:34.713 --> 00:33:43.189 That standalone mobile website and we all figured out responsive design and that required so much effort and now it's like innate to our world. 00:33:43.189 --> 00:33:45.618 And our world is in fact almost built mobile first. 00:33:45.769 --> 00:33:54.288 So it really kind of hints at these possibilities of like being in a multimodal first world where by default you're interacting with it that way. 00:33:54.288 --> 00:33:55.969 Do you, do you see that in our future? 00:33:56.804 --> 00:33:57.939 I definitely do. 00:33:57.939 --> 00:34:01.979 I think that technology adoption kind of moves in these phases, right? 00:34:01.979 --> 00:34:12.521 Where, you have these early adopters and then you kind of have, this period of time where I. people are like the paradigm that's already at scale, I guess. 00:34:12.722 --> 00:34:15.298 and then there, there're these, these s-curves, right? 00:34:15.298 --> 00:34:36.188 And you're kind of taking the paradigm that's already at scale and you're adapting what's at scale to work in this new paradigm and adopting, the technology in as almost like a. You know, it's not meant to be like reductive, or diminishing the value of doing this because we all kind of do it at first, which is, you know, you kind of

bolt it on, right? 00:34:36.188 --> 00:34:37.148 Like, are you, you 00:34:37.351 --> 00:34:37.590 Right. 00:34:37.659 --> 00:34:41.409 to integrate that new technology into what exists already. 00:34:41.838 --> 00:34:48.998 but then after that wave, and the technology or the paradigm shift kind of gets popularized to a degree. 00:34:50.068 --> 00:34:55.208 And you're firmly in like the adoption kind of neck of the S-curve, um, or in that middle section. 00:34:55.599 --> 00:35:04.518 Then you start to have products and companies, get built, which are kind of native to that new paradigm. 00:35:04.518 --> 00:35:04.818 Right? 00:35:04.818 --> 00:35:05.449 And so 00:35:05.610 --> 00:35:05.670 Yeah. 00:35:05.838 --> 00:35:14.969 mobile it was like, it was Uber and it was Snap, it was Instagram and it was, it was these companies that were kind of like, mobile only, right? 00:35:14.969 --> 00:35:16.079 Or mobile native. 00:35:16.416 --> 00:35:16.766 Right. 00:35:17.139 --> 00:35:26.559 Thinking about what they were building, like as, as mobile being the only entry point into, those services and, and how the interaction should work and how the product, should work. 00:35:26.559 --> 00:35:31.659 And so I think we're definitely gonna see that with, with, with AI as well. 00:35:31.893 --> 00:35:35.974 I think it's on an accelerating timeframe though, which is, kind of fascinating. 00:35:36.050 --> 00:35:43.130 uh, like everything, like I think all technologies an accelerating timeframe, so any point along the curve is also gonna happen a lot faster. 00:35:43.389 --> 00:35:48.166 as these things, you know, kind of move along the, uh, progression that I see. 00:35:48.166 --> 00:35:51.003 I think that there's, there's kind of like three things. 00:35:51.054 --> 00:35:57.923 I think the first one is we're kind of in this like co-pilot moment right now I don't think co-pilots are actually gonna go away. 00:35:57.923 --> 00:36:05.784 Just to be clear, I don't think it's like the co-pilot is the bolt-on thing and then like it gets completely replaced by something else. 00:36:05.784 --> 00:36:06.384 I think that. 00:36:07.001 --> 00:36:16.635 It starts off with this co-pilot use case where your copilot is your virtual voice assistant in the ChapGPT world or it's cursor that's sitting in your code editor. 00:36:16.974 --> 00:36:35.034 or it's another experience where you have this assistant that is there to like, help you with stuff, and I think at first it will help you by, augmenting your abilities, like inserting itself in certain parts of your workflow and, uh, you know, generating content or information for you that you can leverage, right? 00:36:35.034 --> 00:36:41.364 Even just thinking as basic as going into notion and having it rewrite a paragraph that you wrote or something like that. 00:36:41.414 --> 00:36:41.684 Yeah. 00:36:41.693 --> 00:36:42.983 like the copilot use case. 00:36:43.032 --> 00:36:53.023 and I think if you take that all the way to its end,

I think what you end up with is something like Jarvis, uh, from Ironman where it's like, well, hey, like I'm envisioning something here. 00:36:53.172 --> 00:37:03.677 Instead of it, it being more engineering driven, now it's more design driven where it's like, Hey, I'm like the maestro and I'm telling you kind of what I want, and then you are kind of generating and doing a lot of that mundane work for me. 00:37:04.713 --> 00:37:07.052 Putting something in front of me and then I'm more of the editor. 00:37:07.322 --> 00:37:07.742 Right. 00:37:07.822 --> 00:37:23.932 I think that that's kind of where the copilot use case or implementation is taken to, its, its logical end, but that, what ends up happening is you end up going from copilot to coworker that's more of like this age agentic thing that's going on or that people talk about. 00:37:23.932 --> 00:37:24.773 That term is kind of. 00:37:25.860 --> 00:37:26.434 Yeah, it's. 00:37:26.492 --> 00:37:27.543 a product called agents too. 00:37:27.543 --> 00:37:41.157 It's like the term doesn't mean anything anymore, but uh, this agentic kind of workflow thing is you are instead giving like instructions or what you desire to this AI entity and it's going and it's. 00:37:41.420 --> 00:37:52.842 it's doing the whole thing itself and then coming back and you're having kind of a meeting with it and you're like, uh, making sure that you're in sync or you're helping give it some more information if it needs to refine, the work that it's done. 00:37:52.842 --> 00:37:56.682 And so I think that that's gonna be like the next stage of all of this. 00:37:56.682 --> 00:37:59.202 And then I think the third stage is. 00:37:59.422 --> 00:38:07.822 we now have a model or a set of models that are so smart they can go and do work themselves and all of that stuff and execute on pretty complex tasks. 00:38:08.242 --> 00:38:15.291 Well, okay, now I need to embody these things and put them inside a, A thing that's shaped like us and can move like us. 00:38:15.342 --> 00:38:30.155 and then it's like, it's literally like a, a coworker or a friend or a companion that can go and, and navigate the physical world and do things in the physical world, that help me and, you know, increase my productivity or maybe just increase my happiness. 00:38:30.541 --> 00:38:32.672 Yeah, I think it'd be a really interesting journey. 00:38:32.802 --> 00:38:35.202 I agree with you about the natural end of copilot. 00:38:35.561 --> 00:38:45.713 I think that even like the parallel that we just drew between, this and like mobile, I. The same thing can be said about like chat assistance with AI and mobile. 00:38:45.744 --> 00:38:48.474 It's the same idea, like, oh, it's our first taste of the technology. 00:38:48.474 --> 00:38:53.844 We're just trying to shove it into the quickest to delivery format that we can, that we're used to. 00:38:53.844 --> 00:38:58.614 And right now that's a chat conversation that's like the most intuitive way to extract things from it. 00:38:58.614 --> 00:39:05.800 But it will obviously evolve into other forms that are more specialized, and that kind of have more intention behind them. 00:39:06.074 --> 00:39:29.228 And I, I wanna, I want take the conversation at this point and kind of shift it into how, this kind of thinking is going to impact, engineering leaders, in our space and about how they're going to have to really rethink, how they build software, how they train their teams, and also how they are going to, bring their software through its entire, you know, SDLC like how does, working with this type of technology impact. 00:39:29.438 --> 00:39:32.048 And so I have some questions that are kind of top of mind for you. 00:39:32.318 --> 00:39:35.679 They're kind of things that might be burning questions in our audience mind as well. 00:39:35.989 --> 00:39:43.728 one of them is really like when you're working with a realtime technology, I think the stakes are a lot different. 00:39:43.909 --> 00:39:52.099 And failure, realtime failure looks a lot different than like a server going down somewhere on like a request response architecture. 00:39:52.349 --> 00:40:03.222 I'm wondering from your, standpoint, what are some fundamental things about engineering that you and your, your organization have had to rethink when building something like LiveKit? 00:40:04.092 --> 00:40:06.432 I think that there's kind of two sets of challenges here, right? 00:40:06.483 --> 00:40:15.847 There are challenges and questions and approaches for engineers, developer teams, companies that are kind of building applications for this new world. 00:40:16.476 --> 00:40:22.900 And then there are the challenges that we've kind of had to solve as a company or as a pro for, for our product. 00:40:22.900 --> 00:40:40.769 And, these are kind of two distinct buckets in a way, just because, The problems that we've had to solve for LiveKit around failure modes, technical challenges, scaling, of that reliability, those problems we had to solve independent of ai. 00:40:41.400 --> 00:40:43.663 just because what LiveKit. 00:40:44.398 --> 00:40:51.327 Provides is not too dissimilar from what we to provide during the pandemic. 00:40:51.327 --> 00:41:00.838 When the open source project started, it was that like we were building infrastructure that made it really easy for a human to connect with another human anywhere in the world, right? 00:41:01.148 --> 00:41:08.197 and the way that human is connecting with that other human is using cameras and microphones, from either of our computers and we're teleporting that data. 00:41:08.907 --> 00:41:10.737 Over some wires to the other person. 00:41:10.737 --> 00:41:12.117 That's what we're doing right now as well. 00:41:12.748 --> 00:41:21.059 And what we're doing is we're leveraging the same technology, but you're connecting those cameras and microphones with the machine. 00:41:21.679 --> 00:41:27.800 can use the same technology because the machine now takes the

inputs in the same way that you do, right? 00:41:27.849 --> 00:41:33.309 Looking at a camera that is capturing something that I'm seeing, or a camera that's pointed at me. 00:41:33.460 --> 00:41:36.010 So either a camera pointed at me or a camera pointed out at the world. 00:41:36.369 --> 00:41:38.590 And so it's seeing what I'm seeing. 00:41:38.940 --> 00:41:41.070 and it's hearing what I'm hearing, right? 00:41:41.070 --> 00:41:44.500 When I speak to you, you're like listening to me, with your ears. 00:41:44.550 --> 00:41:55.474 and the computer is also doing something very similar So the same technology can be used, for connecting humans to other humans as it can be for connecting humans to machines, and thus. 00:41:55.994 --> 00:42:07.695 The problems to solve there around like making sure it's as low latency as possible, making sure you have servers to the edge for any user or AI model to connect to or agent to connect to. 00:42:08.005 --> 00:42:14.880 making sure that if a server goes down, you can transparently fail over to another machine so you can kind of horizontally scale these servers. 00:42:15.411 --> 00:42:21.742 and then making sure that you can load balance across these servers and they can scale up and handle millions and millions of concurrent connections. 00:42:22.222 --> 00:42:28.041 All of those problems that we had to solve for video conferencing and for live streaming before, 00:42:28.298 --> 00:42:28.719 Mm-hmm. 00:42:28.851 --> 00:42:29.871 whole AI thing happened. 00:42:30.806 --> 00:42:38.780 There's, almost a hundred percent overlap with kind of this, proliferation of AI and, voice and vision interfaces to an AI model. 00:42:38.990 --> 00:42:43.521 So we've had to solve from that perspective have been the same. 00:42:43.521 --> 00:42:59.394 Now, there are new problems that we've had to solve that are aligned with kind of what I would say the challenges are for the application developer, the engineering teams out there that are using something like LiveKit to build an application There, it's really about how do you make sure that you do this safely? 00:43:00.775 --> 00:43:18.356 when I talk to another human being, know, there's, there's a few things, that let's say with an AI model, If you were to draw a parallel to a human being, like if I said something to you and you like, nodded your head and then you just said nothing, or you shut off your camera and disconnected from this call or whatever, that would be a pretty weird experience, right? 00:43:18.356 --> 00:43:21.117 So like, reliability is definitely something that we have to care about. 00:43:21.117 --> 00:43:21.597 Make sure that. 00:43:22.036 --> 00:43:28.067 You know, you have that failover and and and there's this continuity between the conversation that I'm having with an ai. 00:43:28.396 --> 00:43:31.637 in the same way that you want that continuity when you're in a Zoom call with someone. 00:43:31.637 --> 00:43:31.956 Right. 00:43:32.664 --> 00:43:33.235

Scenario. 00:43:33.235 --> 00:43:37.594 And so you gotta make sure that those fundamentals work and that's, you know, what we take on as our job. 00:43:37.954 --> 00:43:48.625 but then what we will help developers do, and we're gonna be building stuff in this direction, but that developers also have the burden of solving, as a challenge is. 00:43:48.875 --> 00:43:52.025 How do you make sure that the model is like saying the right thing, right? 00:43:52.025 --> 00:44:07.387 Or how do you make sure it's not hallucinating and, let's just say like swearing or saying something inappropriate, or if it's a model that is, generative video model, how do you make sure that it's not like generating, inappropriate images or things like that? 00:44:07.438 --> 00:44:18.237 I think that what is tricky with real time and with audio and video in particular is that, when it's real time, you do not have a lot of room for, corrective measures, right? 00:44:18.574 --> 00:44:18.864 Yeah. 00:44:19.318 --> 00:44:21.507 this information is being generated on the fly. 00:44:21.608 --> 00:44:24.818 The second problem versus text is that. 00:44:25.867 --> 00:44:30.907 You can make assertions about text, like they're strings, right, of characters. 00:44:31.148 --> 00:44:35.588 And you can, we've been writing assertions about strings of characters in code for a long time. 00:44:35.855 --> 00:44:36.125 Yep. 00:44:36.637 --> 00:44:46.617 but now you have to suddenly like have assertions around like waveforms or around images and, How fast can you, can you make those assertions, right? 00:44:46.617 --> 00:44:59.967 You gotta like, take audio that is coming out of a model and you have to maybe like send it to another model that is looking at or somehow determining whether the there's appropriate, uh, you know, it's inappropriate or not, and flagging it. 00:45:00.268 --> 00:45:01.498 And then like, if it's. 00:45:01.702 --> 00:45:02.663 Inappropriate. 00:45:02.693 --> 00:45:04.012 Like how do you correct for that? 00:45:04.012 --> 00:45:07.076 Have you already delivered the audio bites, to the user's device? 00:45:07.076 --> 00:45:14.695 Like, do you kind of do like an ask for forgiveness, thing or like an ask for, for permission and, and so that's a challenge, right? 00:45:14.695 --> 00:45:20.246 And one of the ways that you can kind of solve that challenge is through like simulation and evaluation. 00:45:20.525 --> 00:45:23.155 where you have like one model or. 00:45:23.585 --> 00:45:27.905 a human even sometimes like go and run through like these testing calls. 00:45:28.266 --> 00:45:36.246 or you're spinning up Voice agents that are going and like having a call with another, uh, voice agent and making sure that it's, it's doing the right thing. 00:45:36.396 --> 00:45:46.235 What you're ultimately trying to do here is build a statistical confidence that you are gonna ship to users is going to act and behave appropriately. 00:45:46.235 --> 00:45:49.416 That's fundamentally what

you're trying to do between two humans. 00:45:49.626 --> 00:45:51.726 The way that you do that is through, I don't know, like. 00:45:52.244 --> 00:45:57.344 Reading their resumes or making sure you have common LinkedIn connections or whatever, right? 00:45:57.344 --> 00:45:57.525 Like 00:45:57.847 --> 00:45:58.197 Right, 00:45:58.695 --> 00:46:05.364 There are all these mechanisms that humans use, to kind of build trust, including just looking at someone, right? 00:46:05.364 --> 00:46:09.144 Like we make these judgments just based on like kind of, uh, how people appear. 00:46:09.144 --> 00:46:10.344 And then, and then there. 00:46:10.375 --> 00:46:13.974 Then oh my gosh, it like enters into this other question about like. 00:46:14.815 --> 00:46:18.894 How do you make sure that you're not like having some kind of inherent bias, right? 00:46:18.981 --> 00:46:19.072 right. 00:46:19.144 --> 00:46:24.074 but also like, making a good judgment around safety based on the input, that you're getting. 00:46:24.074 --> 00:46:25.664 It's, really tricky. 00:46:25.664 --> 00:46:27.974 it's like a world that we're still figuring out. 00:46:28.025 --> 00:46:31.894 but it's something that is a real thing to consider when you're shipping these systems. 00:46:32.396 --> 00:46:39.387 I really like how you explained that in terms of at the end there, especially where, we have like a test case, right? 00:46:39.387 --> 00:46:43.117 Like we write assertions on strings on math, all the time. 00:46:43.117 --> 00:46:44.376 That's a solved problem. 00:46:44.376 --> 00:46:46.717 And, and tech, and that's also like table stakes. 00:46:46.887 --> 00:46:55.467 Right, like you're making an application and you're shipping it, like you better have some unit tests in there somewhere, or someone's probably gonna be unhappy and it's probably gonna be you and your users. 00:46:55.706 --> 00:47:06.447 And so in the case of like a, using multimodal tech, you're kind of solving from zero on a lot of those, test case scenarios and figuring out how would a human validate. 00:47:06.447 --> 00:47:08.751 This kind of input from another human. 00:47:08.961 --> 00:47:13.282 You know, you and I are having conversation, like you said, like I'm not nodding my head, you're nodding your head. 00:47:13.282 --> 00:47:14.782 We can see that from each other. 00:47:15.021 --> 00:47:24.112 But if I change my tone of voice or if I close my computer or if I turn off my camera, like it's going to fundamentally shift the conversation, right? 00:47:24.112 --> 00:47:32.202 And so being able to react to those in real time to understand when that needs to change behavior of an application is really important. 00:47:32.512 --> 00:47:38.431 but really just kind of solving that Totally unsolved problem of testing a voice conversation. 00:47:38.612 --> 00:47:41.612 I like the innovative idea of spinning up a voice agent. 00:47:41.791 --> 00:47:43.742 It's almost like a test case. 00:47:43.742 --> 00:47:45.481 And it's

like, here's your script. 00:47:45.722 --> 00:47:50.911 And that is like the test, the unit test or something, and then you're gonna go through this flow. 00:47:51.041 --> 00:47:53.081 and then that allows you to scale it. 00:47:53.081 --> 00:48:06.324 and I think that's ultimately, the big thing right, as you build all of these solutions for keeping your technology secure, keeping it delivering, going from zero to one, whenever you come across one of those new scenarios, 00:48:06.847 --> 00:48:07.838 Yeah, exactly. 00:48:08.208 --> 00:48:09.498 early days for it, right? 00:48:09.498 --> 00:48:12.378 Like we're, we're still a little bit in the stone ages of all of this. 00:48:12.378 --> 00:48:17.007 And, and trying to figure out how do we actually scale these systems in a way that feels safe? 00:48:17.007 --> 00:48:24.628 Because the other issue is if you, cost of getting it wrong is very, very high. 00:48:24.927 --> 00:48:27.807 it's not as severe as, say, like a self-driving car. 00:48:29.038 --> 00:48:29.938 Killing someone. 00:48:30.168 --> 00:48:33.737 when that happens, you know, everyone's like, well, we can't use this anymore. 00:48:33.737 --> 00:48:48.498 Like, even if a self-driving car is safer than a human driver, there's something weird about like, when you give your agency up to, this machine, if it gets it wrong, then all of a sudden we kind of have this expectation that machines are deterministic and 00:48:48.498 --> 00:48:48.987 that they get it 00:48:49.007 --> 00:48:49.788 right every time. 00:48:49.788 --> 00:48:54.827 But ai, the magical thing, the magical and scary thing about. 00:48:55.297 --> 00:49:00.608 This new wave of AI is that this approach to it is probabilistic. 00:49:00.608 --> 00:49:03.427 We have a probabilistic computer, right? 00:49:03.728 --> 00:49:13.552 I'm not gonna say it learns exactly the same way that we do, or thinks exactly the same way that we do, but there are similarities and, ultimately like at a high level. 00:49:14.402 --> 00:49:17.012 We are probabilistic machines and so are these. 00:49:17.012 --> 00:49:30.289 And so, we have to kind of adjust our expectations as well as a society for, you know, what the outputs might be and start to get a bit comfortable with, some of the adversarial cases, that may come up. 00:49:30.650 --> 00:49:41.719 at the same time, devise ways that we can start to build more and more statistical confidence and kind of bring some determinism or at least statistical determinism, to these systems. 00:49:42.436 --> 00:49:42.797 Right. 00:49:42.797 --> 00:49:47.586 We can't throw away those things that have worked for us before just because the table's changed a bit. 00:49:47.586 --> 00:49:52.456 We still have to care about, being certain, we're working with technology that is uncertain. 00:49:52.456 --> 00:50:06.206 It's probabilistic, like you said, and with that opens up so many opportunities For it to do things that a deterministic machine cannot, but with it also, so

many pitfalls, so many first time, errors or issues that you have to encounter. 00:50:06.206 --> 00:50:07.317 So many learning issues. 00:50:07.317 --> 00:50:12.336 And when you are talking about technology that can interact with our world, you can talk about even like this. 00:50:12.411 --> 00:50:20.311 The stakes being as high as like someone's life and whether it's like, uh, someone's life is getting saved on a nine one one call or someone's in a car that's driving itself. 00:50:20.532 --> 00:50:28.992 it opens up a lot of interesting like societal questions about agency, about who is responsible for the machines that make these decisions. 00:50:29.211 --> 00:50:31.251 is it the user, is it the people who build it? 00:50:31.251 --> 00:50:32.751 Is it the somewhere in between? 00:50:33.081 --> 00:50:34.942 'cause it can't just be the machine itself, right? 00:50:34.942 --> 00:50:36.202 And so there, these are a lot of. 00:50:36.302 --> 00:50:38.641 Questions that I think we're all still figuring out. 00:50:38.942 --> 00:50:47.192 It's really interesting to learn how those questions kind of, um, are imprinted in the technology itself as it's getting built. 00:50:47.192 --> 00:50:51.389 Like you're testing, the safety of a voice conversation. 00:50:51.579 --> 00:50:53.259 you're putting things right on the edge. 00:50:53.498 --> 00:50:56.199 That way you can have as low as latency as possible. 00:50:56.389 --> 00:50:58.608 you know, you're solving the problems in different ways. 00:50:58.608 --> 00:51:00.798 And with them there, there comes different problems. 00:51:01.099 --> 00:51:06.313 I think something too that really stands out to me is how it impacts, high stakes environments. 00:51:06.364 --> 00:51:10.547 like the 9 1 1 use case is incredible I think those are huge frontiers. 00:51:10.867 --> 00:51:13.985 and just in general, this technology, I think, this is just the start of it. 00:51:14.175 --> 00:51:17.295 I, I think we're talking about something that's still in really early stages. 00:51:17.295 --> 00:51:18.315 I'm excited to see. 00:51:18.795 --> 00:51:21.885 Where LiveKit goes, how this kind of technology takes off. 00:51:21.885 --> 00:51:28.034 And, you know, maybe we can talk again more in the future as, as that journey kind of continues, because I think it's gonna be a really interesting one. 00:51:28.344 --> 00:51:41.364 this was a total blast, you know, for me to have you on the pod to, to kind of dig into your brain about how you're thinking about building technology, the different ways you and your team have to like orient your mind around delivering this kind of software. 00:51:41.735 --> 00:51:45.184 Russ, where can people go to learn more about LiveKit and what you're building? 00:51:46.190 --> 00:51:48.081 you can check out a few different places. 00:51:48.101 --> 00:51:50.960 Uh, so the website, of course is LiveKit.io. 00:51:50.981 --> 00:52:00.768 Um, I think also important LiveKit is an open source kind of project and company like we built the, our commercial offering around, our entire open source stack. 00:52:00.768 --> 00:52:10.012 So github.com/LiveKit is where there's all of the repos in the code and you can self-host it, and, and build with that without necessarily. 00:52:10.427 --> 00:52:11.237 Giving us money. 00:52:11.297 --> 00:52:12.797 though if you wanna give us money, I won't complain. 00:52:13.097 --> 00:52:20.038 Um, and, uh, and of course, uh, LiveKit, is also on x, so x.com/LiveKit. 00:52:21.148 --> 00:52:28.588 you can also DM me on Twitter, um, at dsa three letters on that keyboard that you won't be using in a few years. 00:52:28.887 --> 00:52:29.188 Um. 00:52:29.588 --> 00:52:30.907 From, from right to left. 00:52:31.268 --> 00:52:40.518 Uh, but, uh, it was, it was really a pleasure to chat about this stuff and I'd love to do a, a check-in down the road and, see where we've ended up and how things are have progressed. 00:52:40.568 --> 00:52:41.467 Yeah, absolutely. 00:52:41.467 --> 00:52:49.597 You know, we'll definitely be staying in touch and we will get those, uh, links, uh, to our show notes as well so that our listeners can go check out the project, can learn a little bit more about its impact. 00:52:49.958 --> 00:52:52.387 thanks for joining us on today's, um, episode. 00:52:52.447 --> 00:52:53.858 Clearly, if you're hearing me say this. 00:52:53.932 --> 00:52:55.972 That you really liked it, you stuck around to the end. 00:52:56.222 --> 00:52:58.228 Uh, so thank you so much for making it this far. 00:52:58.409 --> 00:53:08.253 if you are listening, please be sure to go and, like our podcast, wherever you are listening to it, as well as Read Our substack newsletter comes out every Tuesday where we deliver our podcast. 00:53:08.583 --> 00:53:11.103 If you're only listening to the podcast, you're only getting about half the story. 00:53:11.103 --> 00:53:15.123 So be sure to check out that newsletter, Be sure to interact with us on LinkedIn. 00:53:15.273 --> 00:53:17.822 You know, Russ give you a lot of great places where you can go find us. 00:53:17.853 --> 00:53:19.293 Uh, you can also find us on there. 00:53:19.322 --> 00:53:23.123 We'd love to continue the conversation and we'll see you next time. 00:53:23.628 --> 00:53:24.429 Thanks so much, Andrew