התמלול המלא של How to Evaluate RAG Systems When Ground Truth Keeps Changing

nobody knew what this would do. 61 00:04:15.225 --> 00:04:20.505 Anyway, I I used this, for another startup that I started at the time in health care. 62 00:04:20.985 --> 00:04:22.425 We used GPT two. 63 00:04:22.825 --> 00:04:32.480 We trained it on health care data and fine tune it. Yeah. And that's kinda been my journey from the early days until today. In the process, of course, I've learned a lot about 64 00:04:33.200 --> 00:04:35.120 how to build machine learning models, 65 00:04:35.440 --> 00:04:36.320 how to 66 00:04:36.640 --> 00:04:40.000 release them to production, how to test them in production to 67 00:04:40.320 --> 00:04:41.680 ensure that they 68 00:04:42.320 --> 00:04:44.160 don't regress in their performance 69 00:04:44.555 --> 00:04:46.875 and in different fields. So it's been 70 00:04:47.115 --> 00:04:48.635 in financial services, 71 00:04:48.715 --> 00:04:52.155 in travel, in health care, and other industries. 72 00:04:52.475 --> 00:05:26.905 Now digging into the topic at hand, as you mentioned, you wrote a book on the topic of RAG and bringing it into production, and RAG as a concept was one of the first ones that really gained a lot of attention as LLMs were exploding onto the scene because it was something that was fairly straightforward and easy to engineer because we didn't have all the frameworks. We didn't have all of this concept of agentic capabilities, and the models themselves weren't really strong enough to do much with that. But we understood how to do some of that few shot learning where you do the in context 73 00:05:27.000 --> 00:05:28.040 parameterization 74 00:05:28.040 --> 00:05:32.040 of providing some examples so that the LLM could then synthesize that. 75 00:05:32.440 --> 00:05:35.080 And I'm wondering if you can just talk to 76 00:05:35.480 --> 00:05:36.840 what you see now 77 00:05:37.080 --> 00:05:52.985 that we're a few years in the role that RAG is playing in some of the actual systems that people are running. And now that we've moved beyond it being a toy and just a... An easy way to get a quick demo running to something that people are relying on for actual production applications. 78 00:05:54.500 --> 00:05:55.780 Sure. Yes. So, 79 00:05:56.340 --> 00:06:02.100 about, you know, two, two and a half years ago, you're absolutely right. Rag was the hottest thing. Right? 80 00:06:02.420 --> 00:06:08.820 And let me clarify a little bit the reason for that, because we started by having LLMs and they were answering questions really well. 81 00:06:09.425 --> 00:06:11.905 And the main challenge or the main opportunity, 82 00:06:11.905 --> 00:06:13.505 I guess, also with Rag was 83 00:06:14.065 --> 00:06:18.305 I want to connect it to my enterprise data. I want to connect to the data that's 84 00:06:18.705 --> 00:06:21.905 documents on Google Drive or on S3 or 85 00:06:22.449 --> 00:06:28.850 in Notion or in anywhere like that. And the LM doesn't know about your private data. So the main 86 00:06:29.090 --> 00:06:33.410 reason to do Rag was to ingest the data from these data sources, 87 00:06:34.129 --> 00:06:35.650 prepare them for retrieval 88 00:06:35.925 --> 00:06:47.685 so that you can use retrieval as part of REG. That's the R in REG there, it's retrieval augmented generation, right? And so the R is the prepare the data for semantic search and retrieve it at query time 89 00:06:47.940 --> 00:06:48.980 so you could 90 00:06:49.300 --> 00:06:50.820 use it inside of 91 00:06:51.380 --> 00:06:55.220 Reg. And the Reg was a simple idea. It was a prompt that said, 92 00:06:55.700 --> 00:06:58.260 you gave the element a prompt that said, here's my question, 93 00:06:59.460 --> 00:07:04.235 and then here's some relevant documents or chunks of documents that might have relevant data. 94 00:07:05.034 --> 00:07:05.675 And 95 00:07:05.835 --> 00:07:09.115 the end of the prompt was, please answer this question 96 00:07:09.435 --> 00:07:11.595 given these documents that are relevant. 97 00:07:11.835 --> 00:07:15.914 And the LM will process everything, summarize those documents, and sort of give you the right answer. 98 00:07:16.880 --> 00:07:20.160 And you're right too, Tobias. It was quite magical. 99 00:07:20.800 --> 00:07:21.840 And a lot of people 100 00:07:22.320 --> 00:07:30.000 said, like, there was a time where, like, oh, look at me. Talking to my PDF was, like, the magical sort of simple use case that 101 00:07:30.480 --> 00:07:35.245 everybody really liked. It was easy to do. I mean, you had a vector database. There was, 102 00:07:35.805 --> 00:07:38.685 at some point, like, I think 50 of those available, 103 00:07:38.764 --> 00:07:43.085 both commercial and open source. It was an explosion of of vector databases. 104 00:07:43.485 --> 00:07:48.500 And you just did some chunking of the documents, parsing the document, checking the documents, 105 00:07:48.820 --> 00:07:56.020 put them in even in memory because it was a small demo. And it was it was... It created a new capability that people really 106 00:07:56.655 --> 00:08:02.415 liked. Instead of reading the documents for 100 pages, they could really like ask it questions, which was really nice. 107 00:08:02.895 --> 00:08:07.935 Now that is very different than doing this in a real production environment. Right? 108 00:08:08.495 --> 00:08:09.375 And 109 00:08:09.640 --> 00:08:15.320 I think that's where companies that are... That build right systems, like I worked at one company called Vectara, 110 00:08:15.320 --> 00:08:27.155 which is where I wrote this book as well, has done a lot to create that retrieval and query pipeline to be much stronger and much more useful. Let me talk a little bit about where some of the challenges are in that. 111 00:08:27.555 --> 00:08:28.515 So first, 112 00:08:28.835 --> 00:08:31.315 I mentioned ingestion. 113 00:08:31.795 --> 00:08:33.875 Right? So you ingest the data 114 00:08:34.355 --> 00:08:37.315 into a semantic search engine like a vector database. 115

00:08:37.830 --> 00:08:52.055 And just in itself, when you think about it, at scale is much more difficult. Let me highlight a couple couple example reasons. Right? There... There's there's big companies who have PDFs that are... And I've I've seen one of those. 15,000 116 00:08:52.055 --> 00:09:02.375 pages of a PDF. Right? So think about that scale. Like, you run something in Python, let's say, and you you know, it could take, I don't know, hours to just process the PDF. And maybe in 117 00:09:02.949 --> 00:09:09.670 the last two pages, it runs in an error and fails and you have to try again. So you have to think about sort of normal engineering 118 00:09:09.750 --> 00:09:12.790 best practices, right, of how do you tackle 119 00:09:13.350 --> 00:09:27.865 really large files so there's no... It doesn't go out of memory. You do it in small batches or something like that. The other thing is just the number of files. I mean, it could be a million files, right, or 20,000,000 files. It doesn't have to be files, by the way. Could be, Notion documents. It could be, 120 00:09:28.425 --> 00:09:29.625 Jira tickets. 121 00:09:29.945 --> 00:09:32.290 It could be, you know, stuff from 122 00:09:32.610 --> 00:09:34.850 SharePoint or or whenever. 123 00:09:35.089 --> 00:09:37.890 But again, if you have a lot of these documents, 124 00:09:38.290 --> 00:09:42.930 you have to build a production system that not only ingests all those documents for the first time, 125 00:09:43.505 --> 00:09:47.905 which again, could take days or weeks and you have to paralyze it maybe and 126 00:09:48.385 --> 00:09:55.825 build retries in case some documents run into errors. And you also have to build things for refresh. 127 00:09:55.904 --> 00:09:57.745 Right? If I have, let's say, 128 00:09:58.660 --> 00:10:02.260 a Google Drive source of data and I ingest all the files there, 129 00:10:02.980 --> 00:10:07.780 new files come in every day or files get changed. How do you refresh without 130 00:10:08.100 --> 00:10:15.755 re ingesting all the files? Right? And then the last thing I wanna mention on ingest, which is really critical to understand is, 131 00:10:16.555 --> 00:10:17.595 multimodal. 132 00:10:17.595 --> 00:10:25.274 Right? So a lot of real world use cases do have multimodal data and quite a bit of it. So for example, tables, 133 00:10:26.490 --> 00:10:35.370 images, diagrams, you know, flowchart, things like that. And a lot of times the the value of of answering a question in Rag or in agents is 134 00:10:35.610 --> 00:10:40.330 really embedded in that image. Right? You might have a graph that shows the revenues 135 00:10:41.395 --> 00:10:51.235 of a company and it's not in any of the numbers, it's just in a graph. So you have to really understand that image to understand what it is. You can even have videos. A lot of companies have 136 00:10:51.635 --> 00:10:52.995 training videos, for example, 137 00:10:53.520 --> 00:10:55.440 a lot of them for staff, 138 00:10:55.440 --> 00:10:56.480 for sales, 139 00:10:56.480 --> 00:11:03.200 for customers, etcetera. So you wanna be able to understand video and audio and transcription and so so forth, 140 00:11:04.160 --> 00:11:04.800 and so on. 141 00:11:05.585 --> 00:11:06.385 And so, 142 00:11:07.585 --> 00:11:12.465 that is a a key interesting set of capabilities that you have to develop 143 00:11:12.705 --> 00:11:13.585 beyond 144 00:11:13.665 --> 00:11:15.825 what's involved in ingesting text. 145 00:11:17.265 --> 00:11:26.640 So that's the Indios part. Now another set of challenges is is I'm... I kinda laughed it out a little bit that everybody built it with a vector database, but there's a lot more to retrieval 146 00:11:26.640 --> 00:11:27.440 than that. 147 00:11:28.080 --> 00:11:31.600 So today, the sort of state of the art retrieval engine 148 00:11:32.080 --> 00:11:34.320 does include a vector database, 149 00:11:35.464 --> 00:11:46.185 but also a what's called hybrid search. So there's also a lot of use cases where you do want to have the old school, I'll call it that, search capabilities like with TFIDF 150 00:11:46.185 --> 00:11:47.625 or BM25 151 00:11:47.704 --> 00:11:50.345 type of lexical or keyword searching. 152 00:11:51.080 --> 00:11:55.720 And hybrid search is the ability to merge semantic search, vector search with that together 153 00:11:56.680 --> 00:11:57.240 to 154 00:11:57.720 --> 00:11:58.440 create 155 00:11:58.840 --> 00:12:02.760 a way to identify both keyword search use cases and 156 00:12:03.415 --> 00:12:05.335 semantic search use cases. 157 00:12:05.895 --> 00:12:09.415 And then people use re rankers quite extensively too because 158 00:12:09.815 --> 00:12:14.375 vector search and embedding model, what's known as embedding models, which are the models that you 159 00:12:14.775 --> 00:12:15.575 embed... 160 00:12:16.295 --> 00:12:18.535 You create vectors from from text, 161 00:12:18.980 --> 00:12:21.540 are pretty good for for initial selection. 162 00:12:22.019 --> 00:12:22.899 But, actually, 163 00:12:23.300 --> 00:12:27.060 they're not as accurate in terms of the semantic ranking. 164 00:12:27.060 --> 00:12:40.425 And so you use re anchors, and there's a lot of work in the ranking space as well. So those are, you know, a couple of examples from the the querying part. Right? And then, I mean, if you do Rag, which is sort of a a single shot, 165 00:12:40.825 --> 00:12:49.390 so I think you have to now think about the generation part. How do you craft the prompt properly so it's accurately taking all the chunks together 166 00:12:49.790 --> 00:12:52.430 and creating the best, most accurate result? 167 00:12:52.830 --> 00:12:57.150 And then that leads into the topic you mentioned earlier about evaluation. How do you, 168 00:12:57.870 --> 00:12:58.670 in production, 169 00:12:59.150 --> 00:13:00.830 evaluate that the results are 170 00:13:01.274 --> 00:13:03.995 accurate, that the retrieval is accurate, 171 00:13:04.394 --> 00:13:06.954

that there's no hallucinations. It's a big topic, 172 00:13:07.514 --> 00:13:12.154 and so on. So there's a there's a whole area of evaluation that, is very, very important. 173 00:13:12.795 --> 00:13:15.195 And on that note of evaluation, 174 00:13:15.274 --> 00:13:21.860 that I think is probably the hardest part of RAG despite all of the other complexities that really come into it because 175 00:13:21.860 --> 00:13:28.740 changing any one piece of the system can have a completely unpredictable result on the output where 176 00:13:29.105 --> 00:13:54.110 you might change a piece of data in the system, and then that might actually have been the keystone that allowed the LLM to be able to provide the accuracy that you wanted or maybe changing the model because of either different thinking parameters or different size of the scale of the model or maybe you're changing the retrieval or reranking system. Any one of those things can have an outsized impact on the overall. 177 00:13:54.510 --> 00:13:56.830 And so there have been numerous 178 00:13:57.375 --> 00:13:59.215 frameworks and approaches 179 00:13:59.215 --> 00:14:04.495 developed to be able to do some sort of at least heuristic testing of the outcome. 180 00:14:04.815 --> 00:14:12.015 A lot of the, at least, earlier iterations were reliant on LLM as a judge where you just had another language model 181 00:14:13.000 --> 00:14:20.920 gauge some sampling of the outputs to say, yes. This looks good or this looks bad. And as long as everything is about 182 00:14:20.920 --> 00:14:21.880 average, 183 00:14:22.040 --> 00:14:23.480 then you ship it. 184 00:14:23.880 --> 00:14:24.600 But, also, 185 00:14:25.175 --> 00:14:32.214 there's the offline versus online testing where with offline, you say, here is my set of examples and here are my 186 00:14:32.375 --> 00:14:41.014 expected outputs. As long as everything looks about right at the end of the day, then we're we're fine. But, really, the changes in these systems come about 187 00:14:41.470 --> 00:14:48.110 when you actually put them in front of people because the way that somebody asks the question or the way that the prompt is structured 188 00:14:48.270 --> 00:14:51.790 can also have all these different variables involved. 189 00:14:52.510 --> 00:15:02.015 And so I'm wondering if you can talk to some of the current set of approaches that people are using for these production or enterprise grade RAG systems that they 190 00:15:02.335 --> 00:15:04.895 rely on to actually build and maintain 191 00:15:05.375 --> 00:15:10.495 confidence in the accuracy and reliability of the overall stack. 192 00:15:11.839 --> 00:15:15.519 Yeah. A 100%. Super... You're right. It's a super important, 193 00:15:15.759 --> 00:15:17.120 topic in Reagan. 194 00:15:17.279 --> 00:15:42.210 I wanna mention also one that people don't necessarily think about upfront. They just build first, and then they realize, oh, we gotta build something like this. So let me let me try to give the full picture here. So first, yes, you're right. People use LM as a judge. What is that? That's like just to give us some some eval prompt and you try to evaluate the accuracy of the results. But I want to separate between retrieval accuracy 195 00:15:42.610 --> 00:15:44.210 and generation accuracy. 196 00:15:44.290 --> 00:16:04.525 Right? Two different things. So retrieval accuracy, we just described, you know, vector vector search and hybrid search and all that. So the end of that phase of RAG is a set of, they're called chunks. They're parts of documents, let's say, a couple paragraphs sometimes or something like that, that are deemed to be relevant for this particular query. 197 00:16:04.845 --> 00:16:08.970 And so the first question is, did my retrieval pipeline 198 00:16:09.210 --> 00:16:21.610 retrieve the right set of chunks, the most relevant set of chunks? Now you have to think about, do I have the relevant set of chunks? The first problem is, like, sometimes the question doesn't have an answer in the documents that I ingested. 199 00:16:22.065 --> 00:16:23.585 And then maybe I need to 200 00:16:24.305 --> 00:16:27.505 figure out where those documents are and bring them in. 201 00:16:27.905 --> 00:16:36.305 It's not obvious in a big enterprise that you have the right, you know, baseline ground truth for... To answer every question that a user might ask. 202 00:16:36.980 --> 00:16:38.740 But assuming you do have them, 203 00:16:39.060 --> 00:16:40.500 your retrieval pipeline, 204 00:16:40.980 --> 00:16:54.905 like you said, might have shifted. You may have been putting a new re ranker. You might have increased the number of results, reduced number of results. You may have read down how you do chunking to so many levers that you could have changed, and every one of those could have influenced 205 00:16:54.905 --> 00:16:56.345 your retrieval pipeline. 206 00:16:56.505 --> 00:16:57.225 And so 207 00:16:57.785 --> 00:16:58.985 on retrieval, 208 00:16:59.225 --> 00:17:00.825 you know, people tend to 209 00:17:01.225 --> 00:17:04.265 do LLM as a judge, or they try to use the regular 210 00:17:04.900 --> 00:17:15.940 techniques that we have, like precision recall, F1 from machine learning. Right? The problem with that is those techniques, those metrics are pretty simple or pretty proven, 211 00:17:16.020 --> 00:17:21.365 require a source of truth. Right? So precision, for example, is how many of the retrieved chunks 212 00:17:22.005 --> 00:17:30.245 are actually should be in... Let's say I have top 10. How many of the top 10 that I actually retrieved should be in the top 10 are actually relevant. Right? 213 00:17:30.565 --> 00:17:32.245 And you have to 214 00:17:32.700 --> 00:17:35.739 somebody has to sit down on all the chunks in your, 215 00:17:35.980 --> 00:17:38.140 say, 20,000,000 document organization 216 00:17:38.140 --> 00:17:39.580 document set and 217 00:17:39.980 --> 00:17:46.874 look at all the chunks and say, are those

relevant for this query and that other query? And I I said 20 mil to to, 218 00:17:47.515 --> 00:17:53.195 you know, emphasize it's it's completely impossible to do it in practice. Right? It's completely infeasible. 219 00:17:53.195 --> 00:17:58.554 Nobody can do that. And so in theory, those techniques work, but in practice, it's not working. 220 00:18:00.110 --> 00:18:13.790 In in the answer side, so the generation, let's say that I don't do this on the the retrieval side, but I go to the generation side, people have done in practices. They said, I'm gonna come up with a thousand questions that I think are relevant. Or sometimes 221 00:18:14.175 --> 00:18:22.975 if they're smarter or they have the system in in in production already, they'll look at the most common questions or take a random sample of questions that users actually ask. 222 00:18:23.535 --> 00:18:27.855 And then they'll they'll have somebody sit down and generate a thousand answers. 223 00:18:28.335 --> 00:18:28.655 Okay? 224 00:18:29.460 --> 00:18:34.820 It's a manual process, but, you know, it's doable. You can sit for a week and... Or a couple people and 225 00:18:35.059 --> 00:18:38.980 and generate that. And then they'll have some metrics to measure 226 00:18:39.140 --> 00:18:40.179 to the... Know, 227 00:18:41.075 --> 00:18:46.914 is this... Is the generated answer similar enough to the to the ground truth answer? 228 00:18:47.315 --> 00:18:49.875 And I've seen a lot of people try that. 229 00:18:50.835 --> 00:18:53.955 There's two problems with that approach, which also leads to LLM as a judge. 230 00:18:55.980 --> 00:19:00.220 The first one, of course, as I said, like, the questions themselves could be hard to generate sometimes. 231 00:19:00.540 --> 00:19:04.940 Remember, you have to generate not just questions that people ask, but also questions that people 232 00:19:05.580 --> 00:19:18.484 might ask later. And so it keeps changing. The tendency is to ask questions that have obvious answers, but you wanna ask also tough questions that don't have answers necessarily and things like that. But the answer is... That's where, again, I see a scalability issue 233 00:19:18.485 --> 00:19:23.445 in practice. Sometimes these organizations are so complicated. They've existed for 234 00:19:23.740 --> 00:19:24.700 thirty years, 235 00:19:25.500 --> 00:19:26.779 and nobody 236 00:19:26.779 --> 00:19:34.779 knows the answer to some questions. It's like in so many systems. And so you can even find the engineer or the product manager or 237 00:19:34.860 --> 00:19:37.260 whatever the function is, the production 238 00:19:37.260 --> 00:19:37.659 manager 239 00:19:38.215 --> 00:19:40.055 that knows the answer to that question, 240 00:19:40.695 --> 00:19:48.535 to validate it, that's the right... To to write that golden answer. Right? And again, it keeps changing. Like like you mentioned earlier, like some of the answers might 241 00:19:49.015 --> 00:19:56.250 might change over time because

I have a new version of a product or I have a new set of, capabilities in my my API. 242 00:19:56.730 --> 00:20:00.330 So it's very hard to keep track of in in practice. 243 00:20:00.570 --> 00:20:02.889 So let me talk about two approaches 244 00:20:02.970 --> 00:20:09.865 that, are using LM as a judge that, I also write up in the book that are really cool to solve both retrieval 245 00:20:10.425 --> 00:20:11.544 and generation. 246 00:20:11.705 --> 00:20:13.304 The first one is called umbrella. 247 00:20:13.945 --> 00:20:21.865 And and... Oh, by the way, before I go there, I wanna mention this has been really interesting research that, we did at Vectara along with the 248 00:20:22.210 --> 00:20:25.650 Jimmy Lynn's lab in University of Waterloo. I wanna give, 249 00:20:25.970 --> 00:20:31.410 all credits to those guys. They did a lot of the research, and we just collaborated to implement this in something 250 00:20:31.650 --> 00:20:39.164 called OpenReque eval that we created later. But Umbrella is really cool. It's a very simple algorithm if you think about it first, 251 00:20:39.725 --> 00:20:42.284 and this is to identify chunks that are relevant. 252 00:20:43.085 --> 00:20:46.124 So you take every chunk in your top 10, 253 00:20:46.445 --> 00:20:49.885 and you run an LM as a judge that gives us a score between 254 00:20:50.410 --> 00:20:54.570 zero and three. Zero means the chunk is completely irrelevant for the query. 255 00:20:54.970 --> 00:20:57.130 One means it's somewhat relevant. 256 00:20:57.930 --> 00:20:59.289 Two is like it's 257 00:20:59.370 --> 00:21:04.890 pretty good relevance. And three is like it's super relevant and might even solve the whole problem. Right? 258 00:21:06.545 --> 00:21:07.184 And, 259 00:21:07.425 --> 00:21:16.465 you know, for those who are with that line as a judge would ask me like, okay, great. I can generate this. Like, what's so research y about this? Why is it so special? 260 00:21:16.705 --> 00:21:26.190 And the reason it's so special is... And that's the algorithm. You take this and you rank it like this. Right? But the the really cool part about it is that the 261 00:21:26.510 --> 00:21:30.270 the lab for for Jimmie Lin, the the research they did is they 262 00:21:30.430 --> 00:21:33.310 tested this approach with a particular prompting 263 00:21:33.310 --> 00:21:33.870 language, 264 00:21:35.305 --> 00:21:36.905 also with human beings. 265 00:21:37.385 --> 00:21:39.625 They looked at the chunks and they 266 00:21:39.945 --> 00:21:53.490 had a bunch of data sets and they compared the results of human judgment with this, you know, over time, a couple iterations of crafting the particular prompt sentence, and they found that the correlation is very high. I think it was like something like 267 00:21:53.810 --> 00:21:56.049 maybe 80%, if I remember correctly. 268 00:21:56.370 --> 00:21:59.330 So very high correlation between what this particular 269 00:21:59.650 --> 00:22:01.570 prompting LLM would do 270

00:22:02.145 --> 00:22:05.025 compared to what a human would rank it as. 271 00:22:05.665 --> 00:22:09.825 And so you have confidence that if you use it in this way, 272 00:22:10.225 --> 00:22:15.585 you get the zero to three actually mean what they do. And that's a general problem of LLM as a judge. 273 00:22:16.830 --> 00:22:30.350 It's a fantastically easy algorithm to implement for a lot of different use cases. The problem is, do you trust it? Right? It's got its own biases. It's got all kinds of problems. The LLM might change and what you get might change too and stuff like that. 274 00:22:31.325 --> 00:22:35.005 Yeah. With the LLM as a judge, you then have two nondeterministic 275 00:22:35.005 --> 00:22:45.245 systems that you have to have some sort of evaluation for, and then you're in the catch 22 of which one do you tune when and how do you decide which one regressed versus improved. 276 00:22:46.620 --> 00:22:56.940 Exactly. Exactly. So with Umbrella, you kinda wanna keep it the way it was implemented first. It's been proven. It's shown the the sort of correlation, and that's why it's it's powerful. 277 00:22:57.500 --> 00:23:01.404 I'll just mention briefly the other one, And, you know, it's called 278 00:23:01.804 --> 00:23:02.845 autonogatizer. 279 00:23:02.845 --> 00:23:11.004 And it's a little bit more complex approach, but it's similar in concept. Again, NLM is a judge approach that takes those chunks and 280 00:23:11.644 --> 00:23:14.960 ranks them relevant to the the generated 281 00:23:14.960 --> 00:23:17.600 answer. This is for for the answer relevancy. 282 00:23:18.080 --> 00:23:20.080 And, what you do is you you 283 00:23:20.400 --> 00:23:21.440 then see, 284 00:23:21.919 --> 00:23:24.720 it tests for a different thing, which is is this 285 00:23:25.365 --> 00:23:31.205 answer that was generated by the LLM in Rag, does it use all the relevant chunks properly? 286 00:23:31.685 --> 00:23:34.404 Right? So if one of the chunks was rated three, 287 00:23:34.725 --> 00:23:36.725 like the three, like the highest rated thing, 288 00:23:37.340 --> 00:23:40.940 you know, you wanted to make sure that that piece of information was 289 00:23:41.340 --> 00:23:42.940 moved over into the answer. 290 00:23:43.420 --> 00:23:48.220 And, again, it's not deterministic, so you don't really know. So that's the goal of that algorithm. It's, 291 00:23:48.780 --> 00:23:49.900 again, something was, 292 00:23:50.365 --> 00:23:52.044 something actually called Nuggetizer, 293 00:23:52.044 --> 00:23:55.085 which is deterministic, and then Auto Nuggetizer is this 294 00:23:55.325 --> 00:24:01.244 generation of nuggets using LLMs and then sort of evaluating this in the context of the generated answer. 295 00:24:01.725 --> 00:24:05.404 So so those are two LLMs as a judge approaches that I wanted to highlight. 296 00:24:06.150 --> 00:24:07.429 And as you mentioned, 297 00:24:08.070 --> 00:24:09.030 one of the 298 00:24:09.590 --> 00:24:19.590 standard ways of

doing the evaluation is that you have that set of labeled question and answer pairs of this is the question that I have. This is the answer that I expect, 299 00:24:19.670 --> 00:24:24.475 and that can be a very time consuming and potentially manual process, 300 00:24:24.475 --> 00:24:28.315 which can also be expensive and also requires that you have some 301 00:24:28.315 --> 00:24:29.515 concept of 302 00:24:29.675 --> 00:24:32.155 what data you have, what types of questions 303 00:24:32.235 --> 00:24:34.555 or inquiries are being performed, 304 00:24:35.120 --> 00:24:40.240 and that requires some amount of bootstrapping that can again become a catch 22. 305 00:24:40.480 --> 00:24:44.559 And I'm wondering what are some of the ways that people are tackling that 306 00:24:45.280 --> 00:24:52.914 start from zero position of I have the LLM. I have a way of generating the embeddings, producing the context, 307 00:24:53.475 --> 00:24:55.075 building out that whole machinery, 308 00:24:55.235 --> 00:25:02.835 but I don't have the flywheel in action yet. So I need to be able to either generate synthetic data to start with or 309 00:25:03.330 --> 00:25:04.210 hypothesize 310 00:25:04.210 --> 00:25:06.289 a certain set of questions and answers, 311 00:25:06.370 --> 00:25:20.645 or maybe I just do it where I have a live feedback loop in production and I'm doing online evaluation so that the system can evolve and improve as it's operating rather than having to get a running head start first? 312 00:25:21.365 --> 00:25:25.284 Yeah. So first of all, yes, synthetic generation is actually a great technique. 313 00:25:25.525 --> 00:25:26.725 You can point 314 00:25:27.685 --> 00:25:32.085 kind of an LM at at a good sample of your data that you just ingested 315 00:25:32.480 --> 00:25:33.919 and actually generate 316 00:25:34.000 --> 00:25:39.440 some really good... With a good prompt, you can generate a diverse set of questions based on this data. 317 00:25:39.680 --> 00:25:45.840 It's not perfect because usually, we'll do it one document at a time, so it cannot generate very... 318 00:25:45.920 --> 00:25:46.080 Like, 319 00:25:46.755 --> 00:25:50.914 you know, questions that necessarily require information from multiple 320 00:25:51.235 --> 00:25:52.195 documents. 321 00:25:52.195 --> 00:25:53.635 That's a little more tricky. 322 00:25:53.795 --> 00:26:00.355 But people do that all the time. I also highly recommend looking at your logs if your system is already... Even a little bit of time in production. 323 00:26:01.270 --> 00:26:02.629 What types of questions 324 00:26:02.790 --> 00:26:14.554 people actually ask that they use the system and ask your subject matter experts, you know, for suggestions for different questions that might be useful. So that's a good way to kick this off. And 325 00:26:15.035 --> 00:26:20.075 sort of from a, like, a release perspective. Right? You wanna build that system to be 326 00:26:20.555 --> 00:26:30.150 sort of a part of your

aggression suite in a way. Right? So you wanna be... Every time I I do something, we kinda mentioned it earlier. If you change a component, you change something, you 327 00:26:30.710 --> 00:26:42.735 fix the bug that changes your chunking strategy or something, everything impacts the system. So you want... Every time as part of your testing, as part of your regression testing or something, you want to be able to run this complete set. 328 00:26:43.054 --> 00:26:50.095 And it's got to be, you know, versioned and controlled and sort of so you know that you at least got the same results as before 329 00:26:50.255 --> 00:26:52.494 and hopefully better over time as you 330 00:26:52.975 --> 00:26:53.775 improve the system. 331 00:26:54.390 --> 00:26:58.710 Now let's talk about online testing a little bit. So it's a very interesting 332 00:26:59.270 --> 00:27:00.070 approach, 333 00:27:00.070 --> 00:27:11.955 but it's more difficult, right? And it's more difficult for two reasons. One is you don't want to increase latency, right? You don't want to have yet another call to an LM to test before you produce the answer. 334 00:27:12.195 --> 00:27:13.635 It also, of course, costs 335 00:27:13.795 --> 00:27:16.275 a lot more. Right? It may be... Yeah. So, 336 00:27:16.595 --> 00:27:22.195 you have another LM call. It's more expensive. People worry about cost of LMs in 2026 337 00:27:22.195 --> 00:27:29.820 much more than previous years, I think it's becoming more and more of an issue. So what I've seen in my my general recommendation is to do online, 338 00:27:30.220 --> 00:27:32.059 maybe as a phase two, like, 339 00:27:32.940 --> 00:27:38.299 if you want, or you can do it in phase one if you're up to it. But, like, it's okay to do it later on after you do the offline, 340 00:27:38.460 --> 00:27:39.100 sort of 341 00:27:39.595 --> 00:27:43.114 release. And also do it as a sampling. Right? You don't have to 342 00:27:43.595 --> 00:27:50.714 do it on every one of the queries that come into your system. So the way this works, right, is your your user query comes in, 343 00:27:51.115 --> 00:27:52.875 you go through your Rag pipeline, 344 00:27:53.860 --> 00:27:55.700 and then you test. 345 00:27:56.340 --> 00:28:18.405 And you don't have to test on all the queries. There may be a lot. Right? Some people have, like, I don't a million queries a month. Right? It will be a lot of cost to run another LLM call or maybe multiple LLM calls on each of these. And so you can say, wanna test on a random 10% of the queries or 20% or 5% or something like this to control your your budget 346 00:28:18.565 --> 00:28:25.229 and also do it out of band. So I don't recommend doing it like waiting and holding off the user until 347 00:28:26.110 --> 00:28:27.309 it's finished. 348 00:28:28.190 --> 00:28:41.304 Think of it as diagnosis for your system. You did off... Offline. You store the results, and then you have some analysis later on and see what you can learn from it. You can learn that, hey. This query, the result wasn't that great, 349 00:28:42.185 --> 00:28:49.065 etcetera. And then you go and fix it back. And something that goes out of band of your release cycle, but gives you, like, more immediate responses. 350 00:28:49.639 --> 00:28:51.720 And one of the other evolutions 351 00:28:51.720 --> 00:29:18.000 of RAG that came about as the idea of agents came into the picture more strongly and as the models became more capable is agentic RAG. And I'm wondering if you can talk to some of the ways that that is distinct from the first generation of rag stacks and some of the ways that it fits into the current ecosystem and maybe how it complicates the picture of how to actually evaluate the effectiveness of it. 352 00:29:18.960 --> 00:29:31.475 Yeah. So the way I've seen this kind of evolve over time is, as we said at the beginning, you know, RAG was the kind of hot thing and and everything happened. But at some point, people saw some of the challenges 353 00:29:32.115 --> 00:29:35.875 of this one shot flow that Wright has. Query, 354 00:29:35.875 --> 00:29:36.835 retrieval, 355 00:29:36.915 --> 00:29:41.715 generated answer. And so it evolved into what's called AgenTik RAG. And 356 00:29:42.115 --> 00:29:43.715 AgenTik RAG is essentially 357 00:29:44.850 --> 00:29:54.929 a simple form of the agents we have today. A very simple form. It's just... It allows you to do maybe a couple of iterations. Right? So the agent takes the user query. 358 00:29:55.330 --> 00:29:57.410 The first thing it can do is it actually 359 00:29:59.145 --> 00:30:00.184 it can 360 00:30:00.505 --> 00:30:01.945 rephrase the query 361 00:30:02.105 --> 00:30:04.184 in a way that's better, 362 00:30:04.424 --> 00:30:07.384 maybe appropriate for the retrieval part. 363 00:30:07.625 --> 00:30:08.424 So it can 364 00:30:08.745 --> 00:30:13.065 rewrite the query in in a different way. It can actually rewrite it a couple times 365 00:30:13.650 --> 00:30:19.410 and maybe apply to the retrieval pipeline a few times to get a couple of different results back. 366 00:30:19.650 --> 00:30:21.410 And that became possible 367 00:30:21.410 --> 00:30:29.624 as our LLMs became possible too. So it didn't just happen in a vacuum. We had longer context windows. You can not just put five results, 368 00:30:29.705 --> 00:30:31.624 and that's all space you had to have. 369 00:30:32.025 --> 00:30:36.585 You could do 10 or 20 or 50 or a 100 because the the context windows became 370 00:30:36.825 --> 00:30:40.664 longer, and you can put more results in there. So AgenTiGrag, 371 00:30:40.664 --> 00:30:45.509 the primary value I see is that ability to do multiple iterations 372 00:30:45.830 --> 00:30:51.509 and also rewrite the queries. So a lot of times if you run a query the way the user wrote it, 373 00:30:52.070 --> 00:30:53.350 it may not be 374 00:30:53.670 --> 00:31:01.275 the correct way that is best suited for accurate retrieval. Right? User may have an acronym there,

for example, 375 00:31:01.675 --> 00:31:02.235 that 376 00:31:03.275 --> 00:31:10.075 is unknown to the system, and the LM might actually translate it using some glossary that you give it or something like that. 377 00:31:10.810 --> 00:31:15.690 So that's been actually really helpful to just improve accuracy of retrieval in general. 378 00:31:16.490 --> 00:31:21.370 And then this evolved into what we know today as AI agents. Right? So agents are 379 00:31:22.745 --> 00:31:30.425 sort of an evolution of that where we have what we call the harness, which is the evolution of the AgenTiC Rag loop, the simple loop that we had at the beginning, 380 00:31:30.665 --> 00:31:32.345 and now it has access to tools. 381 00:31:32.745 --> 00:31:39.309 So if you think about it in that in that sense, AgenTiC Rag was an agent with one tool for retrieval 382 00:31:39.710 --> 00:31:43.309 using the Rag kind of retrieval pipeline without the generation. 383 00:31:43.550 --> 00:31:46.270 And now we can have access to tools anywhere from 384 00:31:46.430 --> 00:31:52.270 retrieval tools like with the Rag, but also tools that go directly to target systems. Like there's 385 00:31:52.565 --> 00:31:56.004 tools to query databases with text to SQL, for example. 386 00:31:56.805 --> 00:32:01.845 There's tools to go to the native systems, like, you know, search capabilities, 387 00:32:01.845 --> 00:32:02.404 in 388 00:32:02.645 --> 00:32:07.285 in different places in Notion and other places. They all provide this kind of API type capabilities 389 00:32:07.470 --> 00:32:08.830 all through MCP. 390 00:32:08.830 --> 00:32:13.149 And that's what the the harnesses we have today for for AI agents building. 391 00:32:14.190 --> 00:32:19.710 And that idea of having the tools be the retrieval 392 00:32:20.510 --> 00:32:22.190 engine has also 393 00:32:23.315 --> 00:32:31.715 maybe diminished some of the initial supremacy of vector databases as the end all be all for that retrieval step 394 00:32:31.955 --> 00:32:32.674 and 395 00:32:32.755 --> 00:32:37.315 changes some of the equations as far as what you need to do for the 396 00:32:37.920 --> 00:32:53.440 seeding of the data where you don't necessarily have to do the whole pipeline of collect the data, chunk it, stick it into a vector database full of embeddings in order to be able to get anything out of it. But it is still a necessary element. I'm curious how you're seeing that 397 00:32:54.265 --> 00:33:07.990 change the ways that people are thinking about their overall retrieval and context system, particularly for these more agentic use cases as well as some of the way... Reasons why somebody might want to stick with a more straightforward 398 00:33:08.070 --> 00:33:11.270 rag system in the kind of first generation 399 00:33:11.270 --> 00:33:12.230 definition. 400 00:33:13.430 --> 00:33:14.870 Yeah. Excellent. 401 00:33:14.870 --> 00:33:17.110 Excellent point. So, you know, 402 00:33:17.430 --> 00:33:22.630 dragging all the data

over to ingest it into the sort of the the rag pipeline 403 00:33:22.924 --> 00:33:40.330 was hard. You had to connect to the source system. You had to essentially replicate all the data. So you had to duplicate. Those were the downsides. Right? And you had, like we talked earlier, we had to refresh it every once in a while. So you had to sort of create like a hot copy of the data only in a way that's, 404 00:33:40.650 --> 00:33:46.090 sort of in a vector database or in a lexical database for ready to be queried. 405 00:33:46.330 --> 00:33:56.705 But it didn't have the advantage of semantic search and hybrid search and all these things we talked about that created better results. And so today, you're right, as things evolve, 406 00:33:57.025 --> 00:33:58.545 more and more vendors 407 00:33:59.105 --> 00:33:59.985 created 408 00:34:00.065 --> 00:34:01.825 tools for their system 409 00:34:01.985 --> 00:34:07.159 to be able to create that system. But the downside of that still, and it keeps improving, 410 00:34:07.160 --> 00:34:08.920 is that you are dependent 411 00:34:09.080 --> 00:34:12.760 on the quality of retrieval of that tool. Right? So if I call... 412 00:34:13.080 --> 00:34:18.360 And and I'm I'm... I wanna say Notion, but I think actually Notion search is not that bad. Not not 413 00:34:19.080 --> 00:34:24.345 kind of telling anything about them, but just because it's a it's a common thing. If you go to... 414 00:34:25.065 --> 00:34:27.705 If you call a tool of Notion instead of bringing 415 00:34:27.865 --> 00:34:32.025 it back to search, you depend on whether Notion Search is good or not. 416 00:34:33.080 --> 00:34:36.600 Does it do multimodal search properly? Does it do 417 00:34:37.080 --> 00:34:40.920 the text search properly? All these things we talked about become important. 418 00:34:41.320 --> 00:34:42.360 And if they're not, 419 00:34:42.760 --> 00:34:48.785 then it doesn't help your agent. It's not gonna get the right information. Similarly with Google Drive, similarly with SharePoint, 420 00:34:48.785 --> 00:34:51.985 similarly with, you know, Jira or or anything else that you have. 421 00:34:52.225 --> 00:34:54.545 So that's the the trade off you make. 422 00:34:54.945 --> 00:35:02.705 And I think as an industry, we're kind of in the middle of that. So over the last two years, a lot of these sort of API search functionality has improved. 423 00:35:03.599 --> 00:35:11.760 But not all these companies have, you know, it's it's... You know, search is a kind of its own little sets of capabilities 424 00:35:11.839 --> 00:35:13.680 and sort of know how. 425 00:35:13.920 --> 00:35:24.535 And I think it's not super easy for all of them to sort of improve the search to a level that it can be as good as you might need it to be. The other challenge is a lot of companies are now moving to 426 00:35:25.095 --> 00:35:25.975 kind of 427 00:35:26.695 --> 00:35:29.975 what's called... There's a word for it. It's called AI sovereignty. 428 00:35:29.975 --> 00:35:36.020 Right? Like... But kind of on premise. Right? When you go to on premise... And we haven't touched this yet, but, like, just to... 429 00:35:36.500 --> 00:35:37.540 On this point, 430 00:35:37.860 --> 00:35:41.860 like, they want everything in control. So they're calling a local LLM. 431 00:35:42.420 --> 00:35:43.460 Maybe it's 432 00:35:43.940 --> 00:35:44.820 OpenAI's 433 00:35:44.820 --> 00:35:58.395 open source model, or maybe it's the NVIDIA one or or thinking machines, whatever it may be, it's all... Nothing goes outside. Nothing that goes outside. And so again, if you have to do that, a lot of times you might wanna ingest the data 434 00:35:59.210 --> 00:36:04.089 earlier into that local environment so they can be part of what can be queried, 435 00:36:04.329 --> 00:36:08.650 and you don't have to make any external API calls in sort of these air gapped environments. 436 00:36:09.609 --> 00:36:13.769 One of the other interesting pieces of bringing agents into the mix is, 437 00:36:14.335 --> 00:36:15.375 in particular, 438 00:36:15.535 --> 00:36:18.495 on the side of data pipelines where 439 00:36:19.455 --> 00:36:34.310 the first generation of RAG required a lot of upfront engineering effort to build that pipeline of doing the chunking and embedding, etcetera, keeping that up to date. Now agents are doing a lot of the work of actual code generation. They're being brought into data pipelines, 440 00:36:34.310 --> 00:36:36.310 and so that also potentially 441 00:36:36.550 --> 00:36:39.510 acts as an input to the RAG system, 442 00:36:39.750 --> 00:36:43.349 which can be yet another source of errors or improvements. 443 00:36:43.885 --> 00:36:47.965 And I'm wondering what are some of the ways that you're seeing that come full circle where 444 00:36:48.045 --> 00:36:50.365 we're bringing LLMs and agents 445 00:36:50.525 --> 00:36:59.910 fully into the process of managing the end to end system for the data seeding as well as doing the retrieval and as well as doing the evaluation 446 00:37:00.470 --> 00:37:04.230 and just some of the ways that that maybe unlocks some new patterns that you're seeing? 447 00:37:05.030 --> 00:37:07.590 Yeah. Quick question. So, for example, 448 00:37:07.910 --> 00:37:14.335 there's a lot of different places. I see, for example, an ingesting of data. Right? There used to be a data engineering pipeline, 449 00:37:14.974 --> 00:37:20.255 that did sort of, let's say, take a PDF file. You have to parse the PDFs, put it into pages, 450 00:37:21.454 --> 00:37:30.800 see where tables and images are, do the chunking, all that stuff. Now people do it with with agents. Right? You have an agent that just does all that process for you. That's one area. 451 00:37:31.279 --> 00:37:32.000 Generally, 452 00:37:32.079 --> 00:37:36.079 the other interesting thing is all these systems used to be with APIs, 453 00:37:36.400 --> 00:37:44.935 and they had very elaborate documentation about the APIs. And you

had to, as an engineer, as a developer, have to read all of that and figure out how to implement it and 454 00:37:45.335 --> 00:37:52.935 if it has changed and everything. Today, you know, I run Code or Codecs or Copilot or whatever, and I just point it at the APIs and 455 00:37:53.415 --> 00:37:53.975 that's it. 456 00:37:55.569 --> 00:38:01.570 It produces a really good system for querying for my actual application that uses the API. 457 00:38:01.970 --> 00:38:06.050 It produces a good thing for indexing the data or ingesting the data. 458 00:38:06.369 --> 00:38:11.089 So like everywhere else in engineering, the coding agents are fantastically powerful 459 00:38:12.025 --> 00:38:14.665 to help us build system much, much faster. 460 00:38:15.545 --> 00:38:16.425 And honestly, 461 00:38:17.865 --> 00:38:19.065 with less 462 00:38:20.105 --> 00:38:34.160 mistakes, like we still make mistakes as humans. I know we all say like, oh, is the coding agent really doing a good job? And in some cases, it does not, and we can talk about that too. There's... I have some experience with that too. One of the things I like to do is to write something in Cloud Code and then 463 00:38:34.400 --> 00:38:54.820 have codex review it, kinda have this adversarial between them. Because I found that a lot of times that really helps. You show something to Cloud Code and you ask Codex to review it, it says, oh, I found three bugs that CodeCode said everything was okay. Or or the other way around. It's not, I guess, one or the other. But, anyway, getting to your earlier point, yeah, I think, I've seen this in ingestion. 464 00:38:55.140 --> 00:38:56.180 I've seen this in, 465 00:38:57.300 --> 00:38:58.580 sort of just writing 466 00:38:58.660 --> 00:39:00.820 other parts of, the system. 467 00:39:01.140 --> 00:39:10.885 I think in the infrastructure of the of the the right pipelines, like where embedding happens and things like this on the retrieval itself, I haven't seen a lot of people use it yet. 468 00:39:11.204 --> 00:39:21.430 And also in evaluation, I haven't seen it yet, but that's just maybe me. I haven't, you know, bumped into people to do that. I I could expect that this could be helpful in any any part of this of this. Another 469 00:39:21.510 --> 00:39:26.630 term that has been percolating through the ecosystem is context engineering, 470 00:39:26.710 --> 00:39:27.190 which 471 00:39:27.830 --> 00:39:38.875 if you squint is the same thing as rag, but people like to come up with new terms for the same thing. And I'm wondering what are some of the ways that you think about the overlap and some of the differentiation 472 00:39:38.875 --> 00:39:42.715 between the ideas of Rag and the idea of context engineering. 473 00:39:43.434 --> 00:39:57.360 So, yeah, the context engineering is... Let let me sort of first state what I kind of my understanding of it just so we align on because the terms are... Could be very confusing. Right? So context engineering is this idea that you as the engineer developing your system 474 00:39:57.680 --> 00:39:58.720 control 475 00:39:59.244 --> 00:40:03.245 what fits into the context of the LM before you send a request. Right? 476 00:40:03.645 --> 00:40:10.365 So by the way, when I started using GPT two, for those who don't know in the audience, it had 512 477 00:40:10.365 --> 00:40:16.740 tokens context limit. That's it. We think about now we have a million or 10,000,000. It it sounds ridiculous, but, 478 00:40:17.140 --> 00:40:19.780 that was it. And then it became a thousand, 479 00:40:20.339 --> 00:40:23.140 ten twenty four. I was like, oh my god. This is double. 480 00:40:23.220 --> 00:40:24.580 Like... So, 481 00:40:25.059 --> 00:40:27.059 context has grown a lot, which is 482 00:40:27.655 --> 00:40:28.455 amazing. 483 00:40:28.615 --> 00:40:30.695 But... So context engineering 484 00:40:30.855 --> 00:40:32.535 now is this practice 485 00:40:32.535 --> 00:40:35.975 of when you create your system of deciding 486 00:40:36.855 --> 00:40:42.055 what's in your system prompt, what do you tell the the agent to be like its guiding principles, 487 00:40:42.400 --> 00:40:44.560 And what sort of intermediate 488 00:40:44.640 --> 00:40:47.840 data you bring into the the same context, 489 00:40:48.880 --> 00:40:50.960 that you put in front of the LLM 490 00:40:51.119 --> 00:41:02.205 in terms of... We think about RAC, it could be chunks that come in. If it's a more advanced agent, it could be sort of the tools that it may be allowed to call and their definitions. 491 00:41:02.765 --> 00:41:05.405 It could be historical information about the user. 492 00:41:05.485 --> 00:41:09.965 People now talk about memory. So there's a lot of things that you can pull into the the context, 493 00:41:10.605 --> 00:41:25.320 but it's the same thing. It's it's how do you take all this stuff and then you send it to the LM and ask it for help to do something, either call one of the tools, call many of the tools, whatever it needs to do, that'll answer the question or perform a task for you. 494 00:41:25.974 --> 00:41:30.935 Yeah. So that's kind of the evolution that, one thing I I I I should mention is, 495 00:41:31.974 --> 00:41:33.895 one way I think about RAG 496 00:41:34.055 --> 00:41:35.095 in a way 497 00:41:35.415 --> 00:41:41.720 is a little bit... I don't... I haven't heard it anywhere else, but I thought it was an interesting way to think about it is REG could be 498 00:41:42.040 --> 00:41:45.400 the kind of attention mechanism for context engineering. 499 00:41:45.560 --> 00:41:53.720 And what do I mean by that? People know about attention, you know, the famous word in deep learning. Right? Attention is a mechanism in deep learning itself where 500 00:41:54.045 --> 00:41:55.085 it sort of 501 00:41:55.724 --> 00:41:57.245 looks at the history 502 00:41:57.565 --> 00:41:58.605 of the 503 00:41:58.925 --> 00:42:04.365 the conversation so far and picks the right words or the right tokens in the

history 504 00:42:04.605 --> 00:42:10.090 that are more important to generate the next token, right? That's what the retention mechanism is 505 00:42:10.250 --> 00:42:19.610 in general. And so sort of taking that idea and saying, I have a question for the user now, and I have all of my corporate, all my enterprise data out there. 506 00:42:20.335 --> 00:42:26.335 And REG sort of picks the right sets of information out of there that needs to be pulled in 507 00:42:26.655 --> 00:42:29.215 to the context of the LLM 508 00:42:29.535 --> 00:42:33.455 in that sense. And so I think of it as, again, a selection... Selector 509 00:42:33.855 --> 00:42:34.575 operator 510 00:42:34.800 --> 00:42:37.280 that helps you pick what's important, 511 00:42:38.000 --> 00:42:41.680 into the the context engineering sort of, landscape. 512 00:42:42.560 --> 00:42:58.775 And as you have been working in this space and particularly in your capacity doing developer relations, working with people out in the community? What are some of the most interesting or innovative or unexpected ways that you've seen teams deal with the production requirements of RAG systems? 513 00:42:59.575 --> 00:43:05.140 The first thing I I wanted to mention there is not necessarily unexpected, actually expected, 514 00:43:05.140 --> 00:43:06.660 but important 515 00:43:06.900 --> 00:43:09.540 is how do you deal with roles and permissions. 516 00:43:10.260 --> 00:43:11.300 Right? So 517 00:43:11.700 --> 00:43:16.660 this is an important problem in Rag and in agents. Right? So I... Let's say I have 518 00:43:17.345 --> 00:43:20.225 all my Google Drive ingested into my Rag pipeline. 519 00:43:20.464 --> 00:43:24.465 Maybe the salaries of the company, there's Excel sheet that's all... Also there. 520 00:43:24.865 --> 00:43:27.265 Only the CEO and HR have access to it. 521 00:43:27.664 --> 00:43:32.305 But if I ingest it without any protections, then everybody can can pull information out of that. Right? 522 00:43:33.480 --> 00:43:35.000 So you wanna make sure 523 00:43:35.240 --> 00:43:44.680 that you have this, role based access control the same way the company implements it be applied in your retrieval pipeline. That's a really important thing to do. 524 00:43:45.160 --> 00:43:50.145 And the... I guess the kind of the the innovative ways people implemented this 525 00:43:50.865 --> 00:43:52.065 is just by 526 00:43:52.385 --> 00:43:54.465 implementing some form of filtering. 527 00:43:54.865 --> 00:43:59.984 Right? So if you say attach to every document that you ingest into your pipeline, 528 00:44:00.730 --> 00:44:05.690 you touch some some notion of permission. And, again, permission is a whole world of complexity. 529 00:44:06.250 --> 00:44:09.210 And a lot of companies have multiple permissioning systems, 530 00:44:09.370 --> 00:44:13.450 and, you know, it's it's a mess. But if you can normalize your permission, 531 00:44:13.875 --> 00:44:18.755 your role into something that you can sort of add as, say, metadata to the document, 532 00:44:19.395 --> 00:44:25.715 and then when a query comes in, you can sort of say, if if the CEO is asking the question, 533 00:44:26.690 --> 00:44:28.130 that... Then I'm gonna 534 00:44:28.450 --> 00:44:35.570 I'm gonna allow them to see everything. If it's, somebody on the... Whatever, on on another team that's not supposed to see the salaries, 535 00:44:36.050 --> 00:44:41.195 then he would then filter the documents that are sort of have the the permission block 536 00:44:41.435 --> 00:44:45.035 in the metadata and not include that in the checks even if it's relevant. 537 00:44:45.435 --> 00:44:47.915 So so the overs... Supersede 538 00:44:48.155 --> 00:44:50.315 relevance with with filtering beforehand. 539 00:44:50.970 --> 00:44:52.250 And that's been 540 00:44:52.650 --> 00:44:53.530 really helpful. 541 00:44:53.690 --> 00:44:54.730 That's one thing. 542 00:44:55.450 --> 00:44:58.970 I think multimodal has been an area that I've seen a lot of innovation. 543 00:44:59.530 --> 00:45:01.690 So I mentioned this a little bit in the beginning 544 00:45:02.090 --> 00:45:09.515 on the indexing or the ingestion side. But think about multimodal also on the UI side. Right? Your application needs to show 545 00:45:10.235 --> 00:45:14.635 the response. But if you have an image, you kinda wanna integrate it into the response. 546 00:45:15.115 --> 00:45:19.435 So, again, a lot of people don't think about it. If you have a video, you wanna embed the video in there to show 547 00:45:19.830 --> 00:45:25.430 where the response was retrieved from and maybe even point the video to the right second. 548 00:45:25.750 --> 00:45:30.630 Right? So you kinda wanna have some innovation on the UI side to make the user experience, like, really helpful. 549 00:45:31.350 --> 00:45:32.150 Ultimately, 550 00:45:32.310 --> 00:45:35.590 everywhere else, every other application we've ever built 551 00:45:35.885 --> 00:45:41.965 in tech, people are lazy. They wanna get the answers right away, and they want it to be appropriate for them. So you kinda work 552 00:45:42.365 --> 00:45:46.925 really hard to make this work in this way in in multi... Multimodal data. 553 00:45:47.580 --> 00:45:52.780 I think the other thing is, as I mentioned, I think people are working really hard to make it 554 00:45:53.100 --> 00:45:54.220 be viable, 555 00:45:54.300 --> 00:45:55.820 both REG and AgenTik, 556 00:45:56.140 --> 00:45:57.580 in sort 557 00:45:58.460 --> 00:46:00.140 of the air gapped environments. 558 00:46:00.365 --> 00:46:04.445 That's a really important use case for a lot of big companies. 559 00:46:04.845 --> 00:46:05.885 Think about, 560 00:46:06.845 --> 00:46:13.885 you know, people working in the government industry where there's, like, different whatever agencies and things like this that they can never leave, or 561 00:46:14.460 --> 00:46:17.340 companies in the defense industry or semiconductors 562 00:46:17.340 --> 00:46:18.860 where

there's a lot of competition. 563 00:46:19.180 --> 00:46:20.540 Those are areas where we see, 564 00:46:20.940 --> 00:46:25.580 even financial services, again, a lot of different things. So if you unpack that, 565 00:46:26.140 --> 00:46:27.980 you have to do a lot of innovative stuff 566 00:46:29.465 --> 00:46:35.625 to do this. So one of the surprising things that I bumped into was that, well, somewhat surprising, would say, is 567 00:46:36.185 --> 00:46:43.865 people are afraid to use the Chinese models, right? Chinese models, the open source models are very, very good in terms of performance, right? They are 568 00:46:45.720 --> 00:46:48.600 on par with the commercial models that are API based 569 00:46:49.080 --> 00:46:53.400 to some... Almost almost there, I think, like, MiniMax three, 570 00:46:53.480 --> 00:47:00.535 Kymi two, you know, three. Like, there's a lot of recent models, QN, like DeepSeek. All of them are... By the benchmarks, they're really good. 571 00:47:01.095 --> 00:47:02.695 And they can be hosted locally, 572 00:47:02.855 --> 00:47:09.734 and there's not too many models that compete that are sort of by US companies. But I see a lot of resistance of using these models 573 00:47:10.320 --> 00:47:11.040 in 574 00:47:11.280 --> 00:47:13.600 at least US and European based companies. 575 00:47:13.840 --> 00:47:14.240 And, 576 00:47:15.520 --> 00:47:18.800 I can see why that would be concerning to some extent, but... 577 00:47:20.000 --> 00:47:32.615 And so I I really like, for example, that, thinking machines recently came with a very good use based model, and I see there's a lot of movement there. So that's something that initially surprised me because open source weights are just weights. 578 00:47:32.855 --> 00:47:33.975 They can't, like... 579 00:47:34.295 --> 00:47:37.255 I don't see... As a technical person, I don't see a way 580 00:47:37.700 --> 00:47:41.540 how this... They would send something or the Chinese could control them. 581 00:47:41.940 --> 00:47:46.420 Maybe I haven't thought this through enough, and I'm not an expert on security and 582 00:47:47.220 --> 00:48:04.135 relationships like this. Maybe it's about the fact that people might get used to them. And then once there's a need for additional code, they'll forget about it or or something like this. I I don't know. I'm really not sure about this. But I've seen a lot of resistance to that for sure in this sort of, server and AI space. 583 00:48:04.775 --> 00:48:09.450 And in your experience of working in this area of 584 00:48:09.610 --> 00:48:16.170 building and evaluating RAG systems, what are some of the most interesting or unexpected or challenging lessons that you learned personally? 585 00:48:17.290 --> 00:48:21.225 Well, first of all, it's something I mentioned earlier that teams don't think about 586 00:48:21.625 --> 00:48:23.945 the challenges upfront. They just... 587 00:48:24.265 --> 00:48:36.140 Mean, they rush to build a system. And like we said earlier, it's... It was a lot... It used to be a lot of engineering. Now it's a little bit less, but it's a lot of work to build a scalable, robust sort of, like, system 588 00:48:36.460 --> 00:48:38.060 with all the bits and pieces. 589 00:48:38.300 --> 00:48:39.340 And a lot of teams... 590 00:48:39.820 --> 00:48:40.940 I was I was 591 00:48:41.820 --> 00:48:43.580 surprised that people don't 592 00:48:43.580 --> 00:48:50.825 apply the same rigor that we do to any other system that we ever built in microservices or whatever other technology in the past to 593 00:48:51.065 --> 00:48:51.945 the system. 594 00:48:52.265 --> 00:48:54.025 I think part of it is because 595 00:48:54.825 --> 00:48:59.865 this has been a lot... It's happened so fast and there's been a lot of push like, hey, we gotta go into AI. We gotta 596 00:49:00.279 --> 00:49:04.839 AI fire systems or whatever it may be. Right? So I think that's 597 00:49:05.799 --> 00:49:08.440 that's something that I've seen, like, the rush 598 00:49:08.680 --> 00:49:15.395 and not thinking through it and then being surprised that it's more complicated than we thought when you go to production scale. 599 00:49:15.795 --> 00:49:19.475 And the second thing is really... And we talked about it too, is, like, just devaluation, 600 00:49:19.475 --> 00:49:21.475 how how little people think about it. 601 00:49:21.875 --> 00:49:24.195 And they're then surprised that 602 00:49:24.515 --> 00:49:28.515 results are not good, and then they put it... Slap it on the system after the fact. 603 00:49:29.530 --> 00:49:34.810 And then realize, okay, how how difficult it is. It's not just creating questions and answers that becomes complicated. 604 00:49:35.210 --> 00:49:36.890 So people are used to 605 00:49:37.770 --> 00:49:44.515 regular applications, right, where where the metrics... So people understand metrics. Right? You have to have measure latency, measure downtime, 606 00:49:44.515 --> 00:49:46.115 you know, all these things that we're 607 00:49:46.915 --> 00:49:50.355 accustomed to from the kind of DevOps world. Right? But 608 00:49:50.915 --> 00:49:52.195 these new metrics 609 00:49:52.275 --> 00:49:55.955 that are ML based and sort of LLM based are new 610 00:49:56.300 --> 00:50:00.060 and more difficult and people, like, don't think about them upfront. 611 00:50:00.460 --> 00:50:05.260 What I've seen is if you you have an ML team internally that is powerful and strong, 612 00:50:05.580 --> 00:50:13.645 then they might tell you about this and you might think about it upfront, but many companies don't, and then they they don't think about it in that sense. 613 00:50:13.965 --> 00:50:17.165 And as you continue to explore 614 00:50:17.245 --> 00:50:25.440 and work in this space and keep tabs on the ways that the industry is evolving, what are some of the predictions or 615 00:50:25.520 --> 00:50:30.800 hopes that you see for the future of RAG systems and context engineering 616 00:50:30.800 --> 00:50:36.160

and just the overall space of data driven LLM and agentic applications? 617 00:50:38.445 --> 00:50:39.005 So... 618 00:50:39.565 --> 00:50:42.765 Yeah. First of all, I think retrieval will continue to evolve, 619 00:50:43.245 --> 00:50:49.005 not... Probably without much the g. The g is still... Some people still use NaiveRag 620 00:50:49.005 --> 00:50:57.550 in in the old way, and it's available for them. But I think more and more will go in... Certainly go into AgenTik, and we'll just use the retrieval part where it fits 621 00:50:57.790 --> 00:51:05.630 and continue to improve that. I still see new embedding models being created that are better, new ranking models that are better. So there's there's still work to do. 622 00:51:06.295 --> 00:51:07.335 People use 623 00:51:07.495 --> 00:51:09.015 glossaries, for example, 624 00:51:09.335 --> 00:51:09.895 to... 625 00:51:10.295 --> 00:51:15.655 And build models that are specific. They're fine tuned specifically to that particular 626 00:51:16.055 --> 00:51:16.855 industry, 627 00:51:17.015 --> 00:51:20.455 which helps with fields. So there's a lot a lot more to do in retrieval and search. 628 00:51:21.170 --> 00:51:23.170 I definitely think we will see the 629 00:51:23.490 --> 00:51:24.370 retrieval 630 00:51:24.370 --> 00:51:27.250 of other systems that you connect to directly 631 00:51:27.410 --> 00:51:31.890 using a MCP tool or something like this become better and 632 00:51:32.130 --> 00:51:33.090 create better results. 633 00:51:33.835 --> 00:51:36.635 I still see a a big gap in the multimodality 634 00:51:36.635 --> 00:51:41.195 in that respect, and I hope those those vendors will will keep improving that as well. 635 00:51:41.595 --> 00:51:44.155 So that's one thing. Context engineering, 636 00:51:44.235 --> 00:51:47.515 again, is... Can used to be a buzzword that's very popular. 637 00:51:48.570 --> 00:51:52.730 It kinda goes into what people think of as memory systems now. 638 00:51:53.290 --> 00:51:56.650 So people talk about, like, the immediate memory, like, 639 00:51:57.210 --> 00:51:59.850 in a in a particular session in a chatbot. 640 00:52:00.330 --> 00:52:09.845 How do we talk, you know, back and forth me in the chatbot? And so if I say something that we talked about before, I wanna remember that. That's the simplest form. But then there's things like, 641 00:52:10.164 --> 00:52:12.565 oh, twenty days ago, we talked about something 642 00:52:13.204 --> 00:52:14.405 related to, 643 00:52:14.964 --> 00:52:16.885 I don't know, maybe 644 00:52:17.200 --> 00:52:18.880 different banks in my area. 645 00:52:19.600 --> 00:52:28.160 And I told you, the agent, that I like this bank, so don't ask me that again. So, like, historical context of earlier conversations. 646 00:52:28.480 --> 00:52:31.680 And I think, both of these are really interesting, 647 00:52:31.760 --> 00:52:34.775 and people attack them in kind of, different ways. 648 00:52:35.335 --> 00:52:42.135 But I think, again, that'll probably be commoditized and be, like, certain two or three approaches that people do, and they'll be really easy to use. 649 00:52:42.454 --> 00:52:47.095 In terms of agents, of course, I think there's a lot of interesting things coming up. So one... 650 00:52:47.720 --> 00:52:52.920 And let me talk about regular agents and then coding agents because I like talking about coding agents separately too. 651 00:52:53.079 --> 00:52:54.280 On regular agents, 652 00:52:54.519 --> 00:53:02.760 you know, the agent... The LLMs do really well with calling tools and deciding what to do with agents. There's no doubt about it. If you compare it to a year ago, it's incredibly better. 653 00:53:03.575 --> 00:53:08.455 But we've seen the agents move to what's called long term horizon 654 00:53:08.535 --> 00:53:09.575 sort of things. 655 00:53:09.974 --> 00:53:11.655 And I think they'll still suffer 656 00:53:12.295 --> 00:53:16.295 from accuracy when the task is... Takes a lot of steps 657 00:53:16.779 --> 00:53:19.099 Because what happens is the context becomes 658 00:53:19.500 --> 00:53:20.540 sort of like 659 00:53:21.980 --> 00:53:29.259 poison a little bit. There's like what's called context rot a little bit. And it'll start doing the wrong things a little bit or asking questions again. 660 00:53:30.015 --> 00:53:36.815 So for example, when I use a coding agent like Cloud Code or Codex, a lot of times when I use the same session for a long time, 661 00:53:37.855 --> 00:53:39.615 after about 200,000 662 00:53:39.615 --> 00:53:41.615 tokens or something, I start seeing it 663 00:53:41.934 --> 00:53:43.934 not act as well. And I 664 00:53:44.079 --> 00:53:55.599 sometimes benefit from, like, killing it starting a new session. Right? So there's... This happens in regular agents too. And I think we're... We have to... As an industry, I'm looking forward to seeing what solutions people come up for for that. 665 00:53:56.475 --> 00:53:59.835 On coding agents, I think we're seeing this interesting 666 00:54:00.315 --> 00:54:06.715 evolution from one coding agent to a swarm of agents. And that's really something we're working on at Band 667 00:54:07.515 --> 00:54:11.435 about creating this... Using Band to... Which is agent agent communication 668 00:54:12.099 --> 00:54:15.140 to create this idea of multiple coding agents 669 00:54:15.460 --> 00:54:18.819 that talk to each other to solve a problem, that that collaborate 670 00:54:18.819 --> 00:54:19.540 together. 671 00:54:19.700 --> 00:54:22.740 Just like we humans in a team can collaborate together. Right? 672 00:54:23.300 --> 00:54:25.140 So I think this idea of 673 00:54:25.645 --> 00:54:26.765 coding agents 674 00:54:26.765 --> 00:54:41.599 that specialize in a particular area, so you can have a coding agent to write a code, a coding agent to plan, a coding agent to review the code or to review the plan, a coding agent for compliance, a coding agent for machine learning, for design, for different roles.

675 00:54:42.400 --> 00:54:49.040 And they have a very precise context. They don't have context of everything, so they can work more precisely on each of these. 676 00:54:49.599 --> 00:55:01.984 And then they collaborate and hand off work to each other as needed when they need to develop code for us and and do that. So I think in my mind, this will create a a much better outcome for everybody. 677 00:55:02.224 --> 00:55:03.744 And especially if you use 678 00:55:04.865 --> 00:55:08.305 multiple agents, you can take advantage of 679 00:55:08.799 --> 00:55:17.119 Clubcode and Codex and Copilot and, I know, Kiro or everybody else. And there's gonna be so many of these, coding agents produced in the antigravity. 680 00:55:17.520 --> 00:55:20.000 Because every one of those has a little bit of a different harness, 681 00:55:20.815 --> 00:55:22.495 so it behaves a little bit differently. 682 00:55:22.655 --> 00:55:30.655 It has a different LLM that drives it. So you will see... You will keep seeing them. I think we all will keep seeing them sort of have 683 00:55:30.655 --> 00:55:31.855 different personalities, 684 00:55:31.855 --> 00:55:33.375 different, like, strength and weaknesses. 685 00:55:34.430 --> 00:55:35.710 And as we get to 686 00:55:35.950 --> 00:55:38.030 complex code generation, 687 00:55:38.910 --> 00:55:40.590 long horizon tasks, 688 00:55:40.830 --> 00:55:45.150 benefiting from sort of adversarial review and seeing multiple, 689 00:55:45.310 --> 00:55:47.950 points of view would actually create higher quality outcomes. 690 00:55:49.225 --> 00:55:51.705 Are there any other aspects of 691 00:55:51.865 --> 00:55:59.305 the work that you're doing in this space or just the overall challenge of building and operating and maintaining these 692 00:55:59.305 --> 00:56:03.545 rag based systems that we didn't discuss yet that you'd like to cover before we close out the show? 693 00:56:04.800 --> 00:56:20.685 Well, I wanted to mention one more area, which is kinda SQL based data that we haven't talked about, and that's interesting. So there is a lot of work in text to SQL. And, again, think of it as a tool that agents might call. The reason it's interesting is very obvious, right? Like most mission critical 694 00:56:20.845 --> 00:56:21.565 data 695 00:56:21.885 --> 00:56:29.325 is still in databases, in Snowflake or Databricks or Microsoft or whatever it is, right? And Text to SQL is going through... 696 00:56:29.805 --> 00:56:38.430 First of all, it's really good, Right? It's still... It it is pretty good now. And it's gonna get better over time. There's models being developed specifically to solve this problem. 697 00:56:39.550 --> 00:56:41.070 But I think one of the 698 00:56:41.310 --> 00:56:42.510 the interesting 699 00:56:43.070 --> 00:56:44.030 things that are happening 700 00:56:44.875 --> 00:56:49.035 is it's related to this world of semantic modeling, right? There's 701 00:56:49.915 --> 00:56:55.515 was Looker that started this, and there's now the new version of this called Malloy, which is open source. 702 00:56:55.915 --> 00:57:02.280 And there's other companies that do this kind of thing to build semantic models to enable enterprise to 703 00:57:02.360 --> 00:57:03.640 sort of define 704 00:57:04.120 --> 00:57:09.320 what something means in SQL. Because the LM, again, like with REG, doesn't have context 705 00:57:09.400 --> 00:57:10.680 of what things mean. 706 00:57:11.160 --> 00:57:14.120 So if you ask it, tell me about the churn 707 00:57:14.744 --> 00:57:16.825 in our recent customer base, 708 00:57:16.984 --> 00:57:26.984 churn can be defined as somebody that dropped off for fifteen days or thirty days or a hundred days or whatever. So there's no definition the LM knows is the the absolute truth. 709 00:57:28.040 --> 00:57:37.880 And so the ability to define this is really important. And it was important before we had this with data and analysts and things like this, but now we have to define it for agents 710 00:57:38.119 --> 00:57:43.640 because there's gonna be so many more agents asking so many more questions out of these stacks of SQL pipelines 711 00:57:43.914 --> 00:57:50.395 that we need to enable, make sure that if agents are gonna do this at scale and we can trust it, it has to be enabled. 712 00:57:50.875 --> 00:57:52.714 So it's just one thing I wanted to mention, 713 00:57:53.194 --> 00:57:55.035 that we didn't didn't cover. 714 00:57:55.994 --> 00:58:03.530 Yeah. That's definitely still an interesting space, and I see the word semantic layer and semantic modeling 715 00:58:03.530 --> 00:58:07.850 probably at least once a day, definitely multiple times a week these days. 716 00:58:08.730 --> 00:58:24.075 Alright. Well, for anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gaps in the tooling technology or human training that's available for AI systems today. 717 00:58:25.250 --> 00:58:29.170 The thing that... First of all, let me start with human training. 718 00:58:29.410 --> 00:58:31.650 I still see a huge gap 719 00:58:31.810 --> 00:58:49.585 in terms of people adopting AI. Now I think you and I and maybe others in the audience, some of us live in the tech world, and it's... We track these things every... We try our best actually to track these things every day. Right? It's it's so difficult. There's, like, 50 things happening every week, and it's hard to to keep track of everything. But, 720 00:58:50.625 --> 00:58:57.980 AI is so powerful and so useful in a lot of different ways. And I'd love to see more. I see a lot, but I'd love to see more 721 00:58:58.140 --> 00:59:00.940 people and companies help with education 722 00:59:00.940 --> 00:59:01.660 of 723 00:59:02.060 --> 00:59:05.020 everybody of how to take advantage of AI. 724

00:59:05.260 --> 00:59:06.940 This started from people 725 00:59:07.095 --> 00:59:12.215 don't even know how to use ChatGPT. Now I think more people know how to use that, but, like, how to use it effectively, 726 00:59:12.935 --> 00:59:18.215 how to connect it to your your own, you know, systems and things like this. Very, very important. 727 00:59:18.615 --> 00:59:23.330 I also... And a little bit in a narrower context, but related to that, I think 728 00:59:23.810 --> 00:59:26.850 the education system itself, the education industry, 729 00:59:26.930 --> 00:59:30.290 teaching, you know, kids, teaching adults 730 00:59:30.370 --> 00:59:31.890 that do later education, 731 00:59:32.210 --> 00:59:33.810 I think has to change. Right? 732 00:59:34.365 --> 00:59:35.805 We have to integrate, 733 00:59:35.885 --> 00:59:42.605 in my mind, how to use AI effectively in every part of education. And it's a massive shift 734 00:59:42.845 --> 00:59:47.325 that has to happen because learning, memorizing things, and and, 735 00:59:47.565 --> 00:59:48.525 kind of 736 00:59:50.010 --> 00:59:53.850 answering questions based on memory is just not gonna work anymore. 737 00:59:54.250 --> 00:59:56.650 I'm very well connected to the education 738 00:59:57.370 --> 01:00:02.090 ecosystem here in in Silicon Valley, and I've seen this a lot in the last couple of years 739 01:00:02.945 --> 01:00:07.585 where teachers are looking for help too. How do I how do I teach Python? 740 01:00:08.065 --> 01:00:09.345 Like, how do I test? 741 01:00:09.665 --> 01:00:22.530 Like, because the the test is usually solve this problem, and you give it to the coding agent. And the second letter, you have the answer. So I think people need help, not just, learning how to use AI, but how to teach AI. So I'm really looking forward to seeing how that... 742 01:00:22.850 --> 01:00:28.850 And I know a lot of people are working on it. I'm not saying it's not solved, but it's it's it's, I'd like to see, people help with that. 743 01:00:29.615 --> 01:00:34.734 Yeah. And, generally, I'm really excited to see what some of the commercial labs, 744 01:00:34.895 --> 01:00:36.575 come up with. I know there's a lot of, 745 01:00:36.974 --> 01:00:38.655 people that's, kind of, 746 01:00:39.295 --> 01:00:41.615 came out of the labs now or starting a lot of new 747 01:00:43.140 --> 01:00:43.940 companies, 748 01:00:43.940 --> 01:00:44.500 like, 749 01:00:45.300 --> 01:00:47.940 you know, Ilia and his company, SSI, 750 01:00:47.940 --> 01:00:49.780 and Thinking Machines, 751 01:00:49.859 --> 01:00:50.580 and and 752 01:00:50.820 --> 01:00:57.460 Coral Automation, and other companies like that that are trying to solve the next generation of of LMs. And also 753 01:00:58.065 --> 01:01:01.265 very excited about image and video generation. I think, 754 01:01:02.224 --> 01:01:04.545 I've been tracking that space a little bit too. And, 755 01:01:05.905 --> 01:01:08.385 kind of the way I like to characterize it, I feel like it's, 756 01:01:08.945 --> 01:01:12.385 it's in a GPT two, GPT three moment of its evolution. 757 01:01:12.960 --> 01:01:16.640 It's very powerful. We use it all the time, but I think it's... 758 01:01:16.960 --> 01:01:20.080 The potential will be unlocked in maybe a couple years. 759 01:01:20.560 --> 01:01:23.360 We're all gonna see this amazing innovation 760 01:01:24.000 --> 01:01:24.480 and, 761 01:01:24.800 --> 01:01:28.560 much more, direct use cases that are that are more immediate in that sense. 762 01:01:29.214 --> 01:01:30.095 Absolutely. 763 01:01:30.095 --> 01:01:52.020 Well, thank you very much for taking the time today to join me and share the work that you've been doing and some of your insights on how to evaluate rag systems and get them running well production. It's definitely a very interesting and continually complex problem and one that keeps shifting. So I appreciate you taking the time to share your insights on that, and I hope you enjoy the rest of your day. 764 01:01:52.500 --> 01:01:53.619 Thanks, Tobias. 765 01:01:53.940 --> 01:01:55.540 Glad to be here. Thanks for having me. 766 01:02:01.425 --> 01:02:09.780 Thank you for listening. Don't forget to check out our other shows. The data engineering podcast covers the latest on modern data management, and podcast.init 767 01:02:09.780 --> 01:02:13.860 covers the Python language, its community, and the innovative ways it is being used. 768 01:02:14.100 --> 01:02:24.730 Visit the site to subscribe to the show, sign up for the mailing list, read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email hosts@aiengineeringpodcast.com 769 01:02:24.730 --> 01:02:25.450 with your story.