How to Evaluate RAG Systems When Ground Truth Keeps Changing

WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 10/07/2026 00:15:00
Duration: 3751.206
Channels: 1
1
00:00:05.120 --> 00:00:08.000
Hello, and welcome to the AI Engineering Podcast,
2
00:00:08.000 --> 00:00:12.160
your guide to the fast moving world of building scalable and maintainable AI systems.
3
00:00:19.605 --> 00:00:35.980
Your host is Tobias Macey, and today I'm interviewing Ofer Mendelevitch about how to evaluate your RAG systems when you have incomplete or changing data to grade against. So, Ofer, can you start by introducing yourself? Hi. Thanks, Tobias. Great to be here. I was a machine learning and kind
4
00:00:35.980 --> 00:00:40.620
of recently deep learning leader in here in the Bay Area for the last maybe two decades.
5
00:00:41.135 --> 00:00:43.855
I actually started my career in machine learning.
6
00:00:44.175 --> 00:00:47.695
I had a fun story to tell about this, which is when I started,
7
00:00:48.094 --> 00:00:53.454
Naive Bayes was a leading algorithm at the time. I don't know if anybody in the audience know what that is,
8
00:00:54.110 --> 00:00:57.230
but it's probably the first machine learning algorithm ever
9
00:00:57.629 --> 00:00:59.230
created that I've known of.
10
00:00:59.550 --> 00:01:02.589
And nobody is using it today, of course, because it's so old.
11
00:01:02.910 --> 00:01:10.284
But, yeah, so I've been I've been around for a while from the time that we called it machine learning to data science to now we call everything AI.
12
00:01:10.845 --> 00:01:12.284
In addition to that, I was
13
00:01:13.244 --> 00:01:28.700
sort of in engineering too. So part of the engineering stack, was VP of engineering, a couple of startups in Silicon Valley. So I've seen a lot of the ups and downs of the Valley over the last two decades. Yeah. More recently, I've been, involved in, some efforts in also developer relations
14
00:01:29.020 --> 00:01:31.740
and, for a company called Band and other companies,
15
00:01:32.375 --> 00:01:33.015
And
16
00:01:33.255 --> 00:01:38.055
wrote a book actually with O'Reilly recently called Hands on Rag for Production,
17
00:01:38.295 --> 00:01:44.455
which I think we're going to talk a little bit about today too, that kind of tries to summarize all of my experiences in
18
00:01:45.070 --> 00:01:48.350
bringing AI systems, Rag, and today agents
19
00:01:48.590 --> 00:01:52.829
into production and what that means and all the challenges that are, part of that.
20
00:01:53.869 --> 00:02:05.045
So you mentioned that you've been in the space for a long time. I'm wondering if you can just start by giving a bit of an overview of how you first got started in the overall field of what used to be machine learning and has now been
21
00:02:05.365 --> 00:02:08.485
subsumed by AI and what it is that has kept you here.
22
00:02:09.125 --> 00:02:11.845
Yeah. So, I actually started with,
23
00:02:12.165 --> 00:02:16.725
machine learning in my first startup, which was founded in year 2,000,
24
00:02:16.410 --> 00:02:19.850
more or less. And as I mentioned earlier, we did a
25
00:02:20.410 --> 00:02:26.570
very elaborate system. It's funny to think about it these days that did the document classification.
26
00:02:27.050 --> 00:02:30.275
It was called Quiver. The company later was acquired by
27
00:02:30.834 --> 00:02:32.595
another company called Inktomi.
28
00:02:32.595 --> 00:02:34.995
But at the time, it was all new. Like,
29
00:02:35.875 --> 00:02:38.115
people who did machine learning were only PhDs.
30
00:02:38.194 --> 00:02:39.875
That's the only way you could do it.
31
00:02:40.435 --> 00:02:42.995
You had to have these very specialized expertise.
32
00:02:43.340 --> 00:02:46.140
But that's how I got started. I started a company and we required
33
00:02:46.780 --> 00:02:47.980
this classification
34
00:02:47.980 --> 00:02:49.500
classification engine
35
00:02:49.739 --> 00:02:50.460
to
36
00:02:50.540 --> 00:02:54.780
do our product, to implement our product. And I just learned on the job, essentially.
37
00:02:56.055 --> 00:03:00.215
Then this, of course, was very interesting to me. So I kept in this field.
38
00:03:00.455 --> 00:03:04.295
Then things evolved into, you know, decision trees and
39
00:03:04.295 --> 00:03:06.055
what we know now as XGBoost
40
00:03:06.055 --> 00:03:06.695
and,
41
00:03:07.095 --> 00:03:09.735
you know, all these other algorithms that evolved over the years.
42
00:03:10.480 --> 00:03:11.120
And
43
00:03:11.760 --> 00:03:12.560
around
44
00:03:13.520 --> 00:03:14.160
twenty
45
00:03:14.720 --> 00:03:16.160
eighteentwenty nineteen,
46
00:03:16.480 --> 00:03:21.360
I actually had my first meeting with the LM space. So a friend of mine who
47
00:03:21.600 --> 00:03:24.160
used to work at OpenAI in the past
48
00:03:24.435 --> 00:03:29.395
told me to take a look at this, and it was... I was very fortunate that I had this friend. And
49
00:03:31.075 --> 00:03:33.555
this is the time when GPT was GPT one,
50
00:03:33.795 --> 00:03:36.515
and GPT two just came out early twenty nineteen.
51
00:03:37.090 --> 00:03:42.930
For those of you who haven't seen that, there's a whole set of articles published by OpenAI
52
00:03:43.010 --> 00:03:49.650
on those years. It's really funny because they... Well, funny and interesting. They released GPT two in three phases.
53
00:03:50.315 --> 00:03:52.795
They had the small, the medium, and the large,
54
00:03:53.355 --> 00:03:53.835
and
55
00:03:54.475 --> 00:03:57.835
and they had extra large. So they did not... They decided not to release. Now
56
00:03:58.235 --> 00:04:00.155
I'm saying it's funny because
57
00:04:00.475 --> 00:04:02.875
the extra large, which was deemed very dangerous,
58
00:04:03.370 --> 00:04:05.050
was about 1,500,000,000
59
00:04:05.050 --> 00:04:11.930
parameters, which is kind of laughable in today's terms. But I think they had the right idea of thinking about alignment and thinking about,
60
00:04:12.330 --> 00:04:14.490
protecting because nobody knew what this would do.
61
00:04:15.225 --> 00:04:20.505
Anyway, I I used this, for another startup that I started at the time in health care.
62
00:04:20.985 --> 00:04:22.425
We used GPT two.
63
00:04:22.825 --> 00:04:32.480
We trained it on health care data and fine tune it. Yeah. And that's kinda been my journey from the early days until today. In the process, of course, I've learned a lot about
64
00:04:33.200 --> 00:04:35.120
how to build machine learning models,
65
00:04:35.440 --> 00:04:36.320
how to
66
00:04:36.640 --> 00:04:40.000
release them to production, how to test them in production to
67
00:04:40.320 --> 00:04:41.680
ensure that they
68
00:04:42.320 --> 00:04:44.160
don't regress in their performance
69
00:04:44.555 --> 00:04:46.875
and in different fields. So it's been
70
00:04:47.115 --> 00:04:48.635
in financial services,
71
00:04:48.715 --> 00:04:52.155
in travel, in health care, and other industries.
72
00:04:52.475 --> 00:05:26.905
Now digging into the topic at hand, as you mentioned, you wrote a book on the topic of RAG and bringing it into production, and RAG as a concept was one of the first ones that really gained a lot of attention as LLMs were exploding onto the scene because it was something that was fairly straightforward and easy to engineer because we didn't have all the frameworks. We didn't have all of this concept of agentic capabilities, and the models themselves weren't really strong enough to do much with that. But we understood how to do some of that few shot learning where you do the in context
73
00:05:27.000 --> 00:05:28.040
parameterization
74
00:05:28.040 --> 00:05:32.040
of providing some examples so that the LLM could then synthesize that.
75
00:05:32.440 --> 00:05:35.080
And I'm wondering if you can just talk to
76
00:05:35.480 --> 00:05:36.840
what you see now
77
00:05:37.080 --> 00:05:52.985
that we're a few years in the role that RAG is playing in some of the actual systems that people are running. And now that we've moved beyond it being a toy and just a... An easy way to get a quick demo running to something that people are relying on for actual production applications.
78
00:05:54.500 --> 00:05:55.780
Sure. Yes. So,
79
00:05:56.340 --> 00:06:02.100
about, you know, two, two and a half years ago, you're absolutely right. Rag was the hottest thing. Right?
80
00:06:02.420 --> 00:06:08.820
And let me clarify a little bit the reason for that, because we started by having LLMs and they were answering questions really well.
81
00:06:09.425 --> 00:06:11.905
And the main challenge or the main opportunity,
82
00:06:11.905 --> 00:06:13.505
I guess, also with Rag was
83
00:06:14.065 --> 00:06:18.305
I want to connect it to my enterprise data. I want to connect to the data that's
84
00:06:18.705 --> 00:06:21.905
documents on Google Drive or on S3 or
85
00:06:22.449 --> 00:06:28.850
in Notion or in anywhere like that. And the LM doesn't know about your private data. So the main
86
00:06:29.090 --> 00:06:33.410
reason to do Rag was to ingest the data from these data sources,
87
00:06:34.129 --> 00:06:35.650
prepare them for retrieval
88
00:06:35.925 --> 00:06:47.685
so that you can use retrieval as part of REG. That's the R in REG there, it's retrieval augmented generation, right? And so the R is the prepare the data for semantic search and retrieve it at query time
89
00:06:47.940 --> 00:06:48.980
so you could
90
00:06:49.300 --> 00:06:50.820
use it inside of
91
00:06:51.380 --> 00:06:55.220
Reg. And the Reg was a simple idea. It was a prompt that said,
92
00:06:55.700 --> 00:06:58.260
you gave the element a prompt that said, here's my question,
93
00:06:59.460 --> 00:07:04.235
and then here's some relevant documents or chunks of documents that might have relevant data.
94
00:07:05.034 --> 00:07:05.675
And
95
00:07:05.835 --> 00:07:09.115
the end of the prompt was, please answer this question
96
00:07:09.435 --> 00:07:11.595
given these documents that are relevant.
97
00:07:11.835 --> 00:07:15.914
And the LM will process everything, summarize those documents, and sort of give you the right answer.
98
00:07:16.880 --> 00:07:20.160
And you're right too, Tobias. It was quite magical.
99
00:07:20.800 --> 00:07:21.840
And a lot of people
100
00:07:22.320 --> 00:07:30.000
said, like, there was a time where, like, oh, look at me. Talking to my PDF was, like, the magical sort of simple use case that
101
00:07:30.480 --> 00:07:35.245
everybody really liked. It was easy to do. I mean, you had a vector database. There was,
102
00:07:35.805 --> 00:07:38.685
at some point, like, I think 50 of those available,
103
00:07:38.764 --> 00:07:43.085
both commercial and open source. It was an explosion of of vector databases.
104
00:07:43.485 --> 00:07:48.500
And you just did some chunking of the documents, parsing the document, checking the documents,
105
00:07:48.820 --> 00:07:56.020
put them in even in memory because it was a small demo. And it was it was... It created a new capability that people really
106
00:07:56.655 --> 00:08:02.415
liked. Instead of reading the documents for 100 pages, they could really like ask it questions, which was really nice.
107
00:08:02.895 --> 00:08:07.935
Now that is very different than doing this in a real production environment. Right?
108
00:08:08.495 --> 00:08:09.375
And
109
00:08:09.640 --> 00:08:15.320
I think that's where companies that are... That build right systems, like I worked at one company called Vectara,
110
00:08:15.320 --> 00:08:27.155
which is where I wrote this book as well, has done a lot to create that retrieval and query pipeline to be much stronger and much more useful. Let me talk a little bit about where some of the challenges are in that.
111
00:08:27.555 --> 00:08:28.515
So first,
112
00:08:28.835 --> 00:08:31.315
I mentioned ingestion.
113
00:08:31.795 --> 00:08:33.875
Right? So you ingest the data
114
00:08:34.355 --> 00:08:37.315
into a semantic search engine like a vector database.
115
00:08:37.830 --> 00:08:52.055
And just in itself, when you think about it, at scale is much more difficult. Let me highlight a couple couple example reasons. Right? There... There's there's big companies who have PDFs that are... And I've I've seen one of those. 15,000
116
00:08:52.055 --> 00:09:02.375
pages of a PDF. Right? So think about that scale. Like, you run something in Python, let's say, and you you know, it could take, I don't know, hours to just process the PDF. And maybe in
117
00:09:02.949 --> 00:09:09.670
the last two pages, it runs in an error and fails and you have to try again. So you have to think about sort of normal engineering
118
00:09:09.750 --> 00:09:12.790
best practices, right, of how do you tackle
119
00:09:13.350 --> 00:09:27.865
really large files so there's no... It doesn't go out of memory. You do it in small batches or something like that. The other thing is just the number of files. I mean, it could be a million files, right, or 20,000,000 files. It doesn't have to be files, by the way. Could be, Notion documents. It could be,
120
00:09:28.425 --> 00:09:29.625
Jira tickets.
121
00:09:29.945 --> 00:09:32.290
It could be, you know, stuff from
122
00:09:32.610 --> 00:09:34.850
SharePoint or or whenever.
123
00:09:35.089 --> 00:09:37.890
But again, if you have a lot of these documents,
124
00:09:38.290 --> 00:09:42.930
you have to build a production system that not only ingests all those documents for the first time,
125
00:09:43.505 --> 00:09:47.905
which again, could take days or weeks and you have to paralyze it maybe and
126
00:09:48.385 --> 00:09:55.825
build retries in case some documents run into errors. And you also have to build things for refresh.
127
00:09:55.904 --> 00:09:57.745
Right? If I have, let's say,
128
00:09:58.660 --> 00:10:02.260
a Google Drive source of data and I ingest all the files there,
129
00:10:02.980 --> 00:10:07.780
new files come in every day or files get changed. How do you refresh without
130
00:10:08.100 --> 00:10:15.755
re ingesting all the files? Right? And then the last thing I wanna mention on ingest, which is really critical to understand is,
131
00:10:16.555 --> 00:10:17.595
multimodal.
132
00:10:17.595 --> 00:10:25.274
Right? So a lot of real world use cases do have multimodal data and quite a bit of it. So for example, tables,
133
00:10:26.490 --> 00:10:35.370
images, diagrams, you know, flowchart, things like that. And a lot of times the the value of of answering a question in Rag or in agents is
134
00:10:35.610 --> 00:10:40.330
really embedded in that image. Right? You might have a graph that shows the revenues
135
00:10:41.395 --> 00:10:51.235
of a company and it's not in any of the numbers, it's just in a graph. So you have to really understand that image to understand what it is. You can even have videos. A lot of companies have
136
00:10:51.635 --> 00:10:52.995
training videos, for example,
137
00:10:53.520 --> 00:10:55.440
a lot of them for staff,
138
00:10:55.440 --> 00:10:56.480
for sales,
139
00:10:56.480 --> 00:11:03.200
for customers, etcetera. So you wanna be able to understand video and audio and transcription and so so forth,
140
00:11:04.160 --> 00:11:04.800
and so on.
141
00:11:05.585 --> 00:11:06.385
And so,
142
00:11:07.585 --> 00:11:12.465
that is a a key interesting set of capabilities that you have to develop
143
00:11:12.705 --> 00:11:13.585
beyond
144
00:11:13.665 --> 00:11:15.825
what's involved in ingesting text.
145
00:11:17.265 --> 00:11:26.640
So that's the Indios part. Now another set of challenges is is I'm... I kinda laughed it out a little bit that everybody built it with a vector database, but there's a lot more to retrieval
146
00:11:26.640 --> 00:11:27.440
than that.
147
00:11:28.080 --> 00:11:31.600
So today, the sort of state of the art retrieval engine
148
00:11:32.080 --> 00:11:34.320
does include a vector database,
149
00:11:35.464 --> 00:11:46.185
but also a what's called hybrid search. So there's also a lot of use cases where you do want to have the old school, I'll call it that, search capabilities like with TFIDF
150
00:11:46.185 --> 00:11:47.625
or BM25
151
00:11:47.704 --> 00:11:50.345
type of lexical or keyword searching.
152
00:11:51.080 --> 00:11:55.720
And hybrid search is the ability to merge semantic search, vector search with that together
153
00:11:56.680 --> 00:11:57.240
to
154
00:11:57.720 --> 00:11:58.440
create
155
00:11:58.840 --> 00:12:02.760
a way to identify both keyword search use cases and
156
00:12:03.415 --> 00:12:05.335
semantic search use cases.
157
00:12:05.895 --> 00:12:09.415
And then people use re rankers quite extensively too because
158
00:12:09.815 --> 00:12:14.375
vector search and embedding model, what's known as embedding models, which are the models that you
159
00:12:14.775 --> 00:12:15.575
embed...
160
00:12:16.295 --> 00:12:18.535
You create vectors from from text,
161
00:12:18.980 --> 00:12:21.540
are pretty good for for initial selection.
162
00:12:22.019 --> 00:12:22.899
But, actually,
163
00:12:23.300 --> 00:12:27.060
they're not as accurate in terms of the semantic ranking.
164
00:12:27.060 --> 00:12:40.425
And so you use re anchors, and there's a lot of work in the ranking space as well. So those are, you know, a couple of examples from the the querying part. Right? And then, I mean, if you do Rag, which is sort of a a single shot,
165
00:12:40.825 --> 00:12:49.390
so I think you have to now think about the generation part. How do you craft the prompt properly so it's accurately taking all the chunks together
166
00:12:49.790 --> 00:12:52.430
and creating the best, most accurate result?
167
00:12:52.830 --> 00:12:57.150
And then that leads into the topic you mentioned earlier about evaluation. How do you,
168
00:12:57.870 --> 00:12:58.670
in production,
169
00:12:59.150 --> 00:13:00.830
evaluate that the results are
170
00:13:01.274 --> 00:13:03.995
accurate, that the retrieval is accurate,
171
00:13:04.394 --> 00:13:06.954
that there's no hallucinations. It's a big topic,
172
00:13:07.514 --> 00:13:12.154
and so on. So there's a there's a whole area of evaluation that, is very, very important.
173
00:13:12.795 --> 00:13:15.195
And on that note of evaluation,
174
00:13:15.274 --> 00:13:21.860
that I think is probably the hardest part of RAG despite all of the other complexities that really come into it because
175
00:13:21.860 --> 00:13:28.740
changing any one piece of the system can have a completely unpredictable result on the output where
176
00:13:29.105 --> 00:13:54.110
you might change a piece of data in the system, and then that might actually have been the keystone that allowed the LLM to be able to provide the accuracy that you wanted or maybe changing the model because of either different thinking parameters or different size of the scale of the model or maybe you're changing the retrieval or reranking system. Any one of those things can have an outsized impact on the overall.
177
00:13:54.510 --> 00:13:56.830
And so there have been numerous
178
00:13:57.375 --> 00:13:59.215
frameworks and approaches
179
00:13:59.215 --> 00:14:04.495
developed to be able to do some sort of at least heuristic testing of the outcome.
180
00:14:04.815 --> 00:14:12.015
A lot of the, at least, earlier iterations were reliant on LLM as a judge where you just had another language model
181
00:14:13.000 --> 00:14:20.920
gauge some sampling of the outputs to say, yes. This looks good or this looks bad. And as long as everything is about
182
00:14:20.920 --> 00:14:21.880
average,
183
00:14:22.040 --> 00:14:23.480
then you ship it.
184
00:14:23.880 --> 00:14:24.600
But, also,
185
00:14:25.175 --> 00:14:32.214
there's the offline versus online testing where with offline, you say, here is my set of examples and here are my
186
00:14:32.375 --> 00:14:41.014
expected outputs. As long as everything looks about right at the end of the day, then we're we're fine. But, really, the changes in these systems come about
187
00:14:41.470 --> 00:14:48.110
when you actually put them in front of people because the way that somebody asks the question or the way that the prompt is structured
188
00:14:48.270 --> 00:14:51.790
can also have all these different variables involved.
189
00:14:52.510 --> 00:15:02.015
And so I'm wondering if you can talk to some of the current set of approaches that people are using for these production or enterprise grade RAG systems that they
190
00:15:02.335 --> 00:15:04.895
rely on to actually build and maintain
191
00:15:05.375 --> 00:15:10.495
confidence in the accuracy and reliability of the overall stack.
192
00:15:11.839 --> 00:15:15.519
Yeah. A 100%. Super... You're right. It's a super important,
193
00:15:15.759 --> 00:15:17.120
topic in Reagan.
194
00:15:17.279 --> 00:15:42.210
I wanna mention also one that people don't necessarily think about upfront. They just build first, and then they realize, oh, we gotta build something like this. So let me let me try to give the full picture here. So first, yes, you're right. People use LM as a judge. What is that? That's like just to give us some some eval prompt and you try to evaluate the accuracy of the results. But I want to separate between retrieval accuracy
195
00:15:42.610 --> 00:15:44.210
and generation accuracy.
196
00:15:44.290 --> 00:16:04.525
Right? Two different things. So retrieval accuracy, we just described, you know, vector vector search and hybrid search and all that. So the end of that phase of RAG is a set of, they're called chunks. They're parts of documents, let's say, a couple paragraphs sometimes or something like that, that are deemed to be relevant for this particular query.
197
00:16:04.845 --> 00:16:08.970
And so the first question is, did my retrieval pipeline
198
00:16:09.210 --> 00:16:21.610
retrieve the right set of chunks, the most relevant set of chunks? Now you have to think about, do I have the relevant set of chunks? The first problem is, like, sometimes the question doesn't have an answer in the documents that I ingested.
199
00:16:22.065 --> 00:16:23.585
And then maybe I need to
200
00:16:24.305 --> 00:16:27.505
figure out where those documents are and bring them in.
201
00:16:27.905 --> 00:16:36.305
It's not obvious in a big enterprise that you have the right, you know, baseline ground truth for... To answer every question that a user might ask.
202
00:16:36.980 --> 00:16:38.740
But assuming you do have them,
203
00:16:39.060 --> 00:16:40.500
your retrieval pipeline,
204
00:16:40.980 --> 00:16:54.905
like you said, might have shifted. You may have been putting a new re ranker. You might have increased the number of results, reduced number of results. You may have read down how you do chunking to so many levers that you could have changed, and every one of those could have influenced
205
00:16:54.905 --> 00:16:56.345
your retrieval pipeline.
206
00:16:56.505 --> 00:16:57.225
And so
207
00:16:57.785 --> 00:16:58.985
on retrieval,
208
00:16:59.225 --> 00:17:00.825
you know, people tend to
209
00:17:01.225 --> 00:17:04.265
do LLM as a judge, or they try to use the regular
210
00:17:04.900 --> 00:17:15.940
techniques that we have, like precision recall, F1 from machine learning. Right? The problem with that is those techniques, those metrics are pretty simple or pretty proven,
211
00:17:16.020 --> 00:17:21.365
require a source of truth. Right? So precision, for example, is how many of the retrieved chunks
212
00:17:22.005 --> 00:17:30.245
are actually should be in... Let's say I have top 10. How many of the top 10 that I actually retrieved should be in the top 10 are actually relevant. Right?
213
00:17:30.565 --> 00:17:32.245
And you have to
214
00:17:32.700 --> 00:17:35.739
somebody has to sit down on all the chunks in your,
215
00:17:35.980 --> 00:17:38.140
say, 20,000,000 document organization
216
00:17:38.140 --> 00:17:39.580
document set and
217
00:17:39.980 --> 00:17:46.874
look at all the chunks and say, are those relevant for this query and that other query? And I I said 20 mil to to,
218
00:17:47.515 --> 00:17:53.195
you know, emphasize it's it's completely impossible to do it in practice. Right? It's completely infeasible.
219
00:17:53.195 --> 00:17:58.554
Nobody can do that. And so in theory, those techniques work, but in practice, it's not working.
220
00:18:00.110 --> 00:18:13.790
In in the answer side, so the generation, let's say that I don't do this on the the retrieval side, but I go to the generation side, people have done in practices. They said, I'm gonna come up with a thousand questions that I think are relevant. Or sometimes
221
00:18:14.175 --> 00:18:22.975
if they're smarter or they have the system in in in production already, they'll look at the most common questions or take a random sample of questions that users actually ask.
222
00:18:23.535 --> 00:18:27.855
And then they'll they'll have somebody sit down and generate a thousand answers.
223
00:18:28.335 --> 00:18:28.655
Okay?
224
00:18:29.460 --> 00:18:34.820
It's a manual process, but, you know, it's doable. You can sit for a week and... Or a couple people and
225
00:18:35.059 --> 00:18:38.980
and generate that. And then they'll have some metrics to measure
226
00:18:39.140 --> 00:18:40.179
to the... Know,
227
00:18:41.075 --> 00:18:46.914
is this... Is the generated answer similar enough to the to the ground truth answer?
228
00:18:47.315 --> 00:18:49.875
And I've seen a lot of people try that.
229
00:18:50.835 --> 00:18:53.955
There's two problems with that approach, which also leads to LLM as a judge.
230
00:18:55.980 --> 00:19:00.220
The first one, of course, as I said, like, the questions themselves could be hard to generate sometimes.
231
00:19:00.540 --> 00:19:04.940
Remember, you have to generate not just questions that people ask, but also questions that people
232
00:19:05.580 --> 00:19:18.484
might ask later. And so it keeps changing. The tendency is to ask questions that have obvious answers, but you wanna ask also tough questions that don't have answers necessarily and things like that. But the answer is... That's where, again, I see a scalability issue
233
00:19:18.485 --> 00:19:23.445
in practice. Sometimes these organizations are so complicated. They've existed for
234
00:19:23.740 --> 00:19:24.700
thirty years,
235
00:19:25.500 --> 00:19:26.779
and nobody
236
00:19:26.779 --> 00:19:34.779
knows the answer to some questions. It's like in so many systems. And so you can even find the engineer or the product manager or
237
00:19:34.860 --> 00:19:37.260
whatever the function is, the production
238
00:19:37.260 --> 00:19:37.659
manager
239
00:19:38.215 --> 00:19:40.055
that knows the answer to that question,
240
00:19:40.695 --> 00:19:48.535
to validate it, that's the right... To to write that golden answer. Right? And again, it keeps changing. Like like you mentioned earlier, like some of the answers might
241
00:19:49.015 --> 00:19:56.250
might change over time because I have a new version of a product or I have a new set of, capabilities in my my API.
242
00:19:56.730 --> 00:20:00.330
So it's very hard to keep track of in in practice.
243
00:20:00.570 --> 00:20:02.889
So let me talk about two approaches
244
00:20:02.970 --> 00:20:09.865
that, are using LM as a judge that, I also write up in the book that are really cool to solve both retrieval
245
00:20:10.425 --> 00:20:11.544
and generation.
246
00:20:11.705 --> 00:20:13.304
The first one is called umbrella.
247
00:20:13.945 --> 00:20:21.865
And and... Oh, by the way, before I go there, I wanna mention this has been really interesting research that, we did at Vectara along with the
248
00:20:22.210 --> 00:20:25.650
Jimmy Lynn's lab in University of Waterloo. I wanna give,
249
00:20:25.970 --> 00:20:31.410
all credits to those guys. They did a lot of the research, and we just collaborated to implement this in something
250
00:20:31.650 --> 00:20:39.164
called OpenReque eval that we created later. But Umbrella is really cool. It's a very simple algorithm if you think about it first,
251
00:20:39.725 --> 00:20:42.284
and this is to identify chunks that are relevant.
252
00:20:43.085 --> 00:20:46.124
So you take every chunk in your top 10,
253
00:20:46.445 --> 00:20:49.885
and you run an LM as a judge that gives us a score between
254
00:20:50.410 --> 00:20:54.570
zero and three. Zero means the chunk is completely irrelevant for the query.
255
00:20:54.970 --> 00:20:57.130
One means it's somewhat relevant.
256
00:20:57.930 --> 00:20:59.289
Two is like it's
257
00:20:59.370 --> 00:21:04.890
pretty good relevance. And three is like it's super relevant and might even solve the whole problem. Right?
258
00:21:06.545 --> 00:21:07.184
And,
259
00:21:07.425 --> 00:21:16.465
you know, for those who are with that line as a judge would ask me like, okay, great. I can generate this. Like, what's so research y about this? Why is it so special?
260
00:21:16.705 --> 00:21:26.190
And the reason it's so special is... And that's the algorithm. You take this and you rank it like this. Right? But the the really cool part about it is that the
261
00:21:26.510 --> 00:21:30.270
the lab for for Jimmie Lin, the the research they did is they
262
00:21:30.430 --> 00:21:33.310
tested this approach with a particular prompting
263
00:21:33.310 --> 00:21:33.870
language,
264
00:21:35.305 --> 00:21:36.905
also with human beings.
265
00:21:37.385 --> 00:21:39.625
They looked at the chunks and they
266
00:21:39.945 --> 00:21:53.490
had a bunch of data sets and they compared the results of human judgment with this, you know, over time, a couple iterations of crafting the particular prompt sentence, and they found that the correlation is very high. I think it was like something like
267
00:21:53.810 --> 00:21:56.049
maybe 80%, if I remember correctly.
268
00:21:56.370 --> 00:21:59.330
So very high correlation between what this particular
269
00:21:59.650 --> 00:22:01.570
prompting LLM would do
270
00:22:02.145 --> 00:22:05.025
compared to what a human would rank it as.
271
00:22:05.665 --> 00:22:09.825
And so you have confidence that if you use it in this way,
272
00:22:10.225 --> 00:22:15.585
you get the zero to three actually mean what they do. And that's a general problem of LLM as a judge.
273
00:22:16.830 --> 00:22:30.350
It's a fantastically easy algorithm to implement for a lot of different use cases. The problem is, do you trust it? Right? It's got its own biases. It's got all kinds of problems. The LLM might change and what you get might change too and stuff like that.
274
00:22:31.325 --> 00:22:35.005
Yeah. With the LLM as a judge, you then have two nondeterministic
275
00:22:35.005 --> 00:22:45.245
systems that you have to have some sort of evaluation for, and then you're in the catch 22 of which one do you tune when and how do you decide which one regressed versus improved.
276
00:22:46.620 --> 00:22:56.940
Exactly. Exactly. So with Umbrella, you kinda wanna keep it the way it was implemented first. It's been proven. It's shown the the sort of correlation, and that's why it's it's powerful.
277
00:22:57.500 --> 00:23:01.404
I'll just mention briefly the other one, And, you know, it's called
278
00:23:01.804 --> 00:23:02.845
autonogatizer.
279
00:23:02.845 --> 00:23:11.004
And it's a little bit more complex approach, but it's similar in concept. Again, NLM is a judge approach that takes those chunks and
280
00:23:11.644 --> 00:23:14.960
ranks them relevant to the the generated
281
00:23:14.960 --> 00:23:17.600
answer. This is for for the answer relevancy.
282
00:23:18.080 --> 00:23:20.080
And, what you do is you you
283
00:23:20.400 --> 00:23:21.440
then see,
284
00:23:21.919 --> 00:23:24.720
it tests for a different thing, which is is this
285
00:23:25.365 --> 00:23:31.205
answer that was generated by the LLM in Rag, does it use all the relevant chunks properly?
286
00:23:31.685 --> 00:23:34.404
Right? So if one of the chunks was rated three,
287
00:23:34.725 --> 00:23:36.725
like the three, like the highest rated thing,
288
00:23:37.340 --> 00:23:40.940
you know, you wanted to make sure that that piece of information was
289
00:23:41.340 --> 00:23:42.940
moved over into the answer.
290
00:23:43.420 --> 00:23:48.220
And, again, it's not deterministic, so you don't really know. So that's the goal of that algorithm. It's,
291
00:23:48.780 --> 00:23:49.900
again, something was,
292
00:23:50.365 --> 00:23:52.044
something actually called Nuggetizer,
293
00:23:52.044 --> 00:23:55.085
which is deterministic, and then Auto Nuggetizer is this
294
00:23:55.325 --> 00:24:01.244
generation of nuggets using LLMs and then sort of evaluating this in the context of the generated answer.
295
00:24:01.725 --> 00:24:05.404
So so those are two LLMs as a judge approaches that I wanted to highlight.
296
00:24:06.150 --> 00:24:07.429
And as you mentioned,
297
00:24:08.070 --> 00:24:09.030
one of the
298
00:24:09.590 --> 00:24:19.590
standard ways of doing the evaluation is that you have that set of labeled question and answer pairs of this is the question that I have. This is the answer that I expect,
299
00:24:19.670 --> 00:24:24.475
and that can be a very time consuming and potentially manual process,
300
00:24:24.475 --> 00:24:28.315
which can also be expensive and also requires that you have some
301
00:24:28.315 --> 00:24:29.515
concept of
302
00:24:29.675 --> 00:24:32.155
what data you have, what types of questions
303
00:24:32.235 --> 00:24:34.555
or inquiries are being performed,
304
00:24:35.120 --> 00:24:40.240
and that requires some amount of bootstrapping that can again become a catch 22.
305
00:24:40.480 --> 00:24:44.559
And I'm wondering what are some of the ways that people are tackling that
306
00:24:45.280 --> 00:24:52.914
start from zero position of I have the LLM. I have a way of generating the embeddings, producing the context,
307
00:24:53.475 --> 00:24:55.075
building out that whole machinery,
308
00:24:55.235 --> 00:25:02.835
but I don't have the flywheel in action yet. So I need to be able to either generate synthetic data to start with or
309
00:25:03.330 --> 00:25:04.210
hypothesize
310
00:25:04.210 --> 00:25:06.289
a certain set of questions and answers,
311
00:25:06.370 --> 00:25:20.645
or maybe I just do it where I have a live feedback loop in production and I'm doing online evaluation so that the system can evolve and improve as it's operating rather than having to get a running head start first?
312
00:25:21.365 --> 00:25:25.284
Yeah. So first of all, yes, synthetic generation is actually a great technique.
313
00:25:25.525 --> 00:25:26.725
You can point
314
00:25:27.685 --> 00:25:32.085
kind of an LM at at a good sample of your data that you just ingested
315
00:25:32.480 --> 00:25:33.919
and actually generate
316
00:25:34.000 --> 00:25:39.440
some really good... With a good prompt, you can generate a diverse set of questions based on this data.
317
00:25:39.680 --> 00:25:45.840
It's not perfect because usually, we'll do it one document at a time, so it cannot generate very...
318
00:25:45.920 --> 00:25:46.080
Like,
319
00:25:46.755 --> 00:25:50.914
you know, questions that necessarily require information from multiple
320
00:25:51.235 --> 00:25:52.195
documents.
321
00:25:52.195 --> 00:25:53.635
That's a little more tricky.
322
00:25:53.795 --> 00:26:00.355
But people do that all the time. I also highly recommend looking at your logs if your system is already... Even a little bit of time in production.
323
00:26:01.270 --> 00:26:02.629
What types of questions
324
00:26:02.790 --> 00:26:14.554
people actually ask that they use the system and ask your subject matter experts, you know, for suggestions for different questions that might be useful. So that's a good way to kick this off. And
325
00:26:15.035 --> 00:26:20.075
sort of from a, like, a release perspective. Right? You wanna build that system to be
326
00:26:20.555 --> 00:26:30.150
sort of a part of your aggression suite in a way. Right? So you wanna be... Every time I I do something, we kinda mentioned it earlier. If you change a component, you change something, you
327
00:26:30.710 --> 00:26:42.735
fix the bug that changes your chunking strategy or something, everything impacts the system. So you want... Every time as part of your testing, as part of your regression testing or something, you want to be able to run this complete set.
328
00:26:43.054 --> 00:26:50.095
And it's got to be, you know, versioned and controlled and sort of so you know that you at least got the same results as before
329
00:26:50.255 --> 00:26:52.494
and hopefully better over time as you
330
00:26:52.975 --> 00:26:53.775
improve the system.
331
00:26:54.390 --> 00:26:58.710
Now let's talk about online testing a little bit. So it's a very interesting
332
00:26:59.270 --> 00:27:00.070
approach,
333
00:27:00.070 --> 00:27:11.955
but it's more difficult, right? And it's more difficult for two reasons. One is you don't want to increase latency, right? You don't want to have yet another call to an LM to test before you produce the answer.
334
00:27:12.195 --> 00:27:13.635
It also, of course, costs
335
00:27:13.795 --> 00:27:16.275
a lot more. Right? It may be... Yeah. So,
336
00:27:16.595 --> 00:27:22.195
you have another LM call. It's more expensive. People worry about cost of LMs in 2026
337
00:27:22.195 --> 00:27:29.820
much more than previous years, I think it's becoming more and more of an issue. So what I've seen in my my general recommendation is to do online,
338
00:27:30.220 --> 00:27:32.059
maybe as a phase two, like,
339
00:27:32.940 --> 00:27:38.299
if you want, or you can do it in phase one if you're up to it. But, like, it's okay to do it later on after you do the offline,
340
00:27:38.460 --> 00:27:39.100
sort of
341
00:27:39.595 --> 00:27:43.114
release. And also do it as a sampling. Right? You don't have to
342
00:27:43.595 --> 00:27:50.714
do it on every one of the queries that come into your system. So the way this works, right, is your your user query comes in,
343
00:27:51.115 --> 00:27:52.875
you go through your Rag pipeline,
344
00:27:53.860 --> 00:27:55.700
and then you test.
345
00:27:56.340 --> 00:28:18.405
And you don't have to test on all the queries. There may be a lot. Right? Some people have, like, I don't a million queries a month. Right? It will be a lot of cost to run another LLM call or maybe multiple LLM calls on each of these. And so you can say, wanna test on a random 10% of the queries or 20% or 5% or something like this to control your your budget
346
00:28:18.565 --> 00:28:25.229
and also do it out of band. So I don't recommend doing it like waiting and holding off the user until
347
00:28:26.110 --> 00:28:27.309
it's finished.
348
00:28:28.190 --> 00:28:41.304
Think of it as diagnosis for your system. You did off... Offline. You store the results, and then you have some analysis later on and see what you can learn from it. You can learn that, hey. This query, the result wasn't that great,
349
00:28:42.185 --> 00:28:49.065
etcetera. And then you go and fix it back. And something that goes out of band of your release cycle, but gives you, like, more immediate responses.
350
00:28:49.639 --> 00:28:51.720
And one of the other evolutions
351
00:28:51.720 --> 00:29:18.000
of RAG that came about as the idea of agents came into the picture more strongly and as the models became more capable is agentic RAG. And I'm wondering if you can talk to some of the ways that that is distinct from the first generation of rag stacks and some of the ways that it fits into the current ecosystem and maybe how it complicates the picture of how to actually evaluate the effectiveness of it.
352
00:29:18.960 --> 00:29:31.475
Yeah. So the way I've seen this kind of evolve over time is, as we said at the beginning, you know, RAG was the kind of hot thing and and everything happened. But at some point, people saw some of the challenges
353
00:29:32.115 --> 00:29:35.875
of this one shot flow that Wright has. Query,
354
00:29:35.875 --> 00:29:36.835
retrieval,
355
00:29:36.915 --> 00:29:41.715
generated answer. And so it evolved into what's called AgenTik RAG. And
356
00:29:42.115 --> 00:29:43.715
AgenTik RAG is essentially
357
00:29:44.850 --> 00:29:54.929
a simple form of the agents we have today. A very simple form. It's just... It allows you to do maybe a couple of iterations. Right? So the agent takes the user query.
358
00:29:55.330 --> 00:29:57.410
The first thing it can do is it actually
359
00:29:59.145 --> 00:30:00.184
it can
360
00:30:00.505 --> 00:30:01.945
rephrase the query
361
00:30:02.105 --> 00:30:04.184
in a way that's better,
362
00:30:04.424 --> 00:30:07.384
maybe appropriate for the retrieval part.
363
00:30:07.625 --> 00:30:08.424
So it can
364
00:30:08.745 --> 00:30:13.065
rewrite the query in in a different way. It can actually rewrite it a couple times
365
00:30:13.650 --> 00:30:19.410
and maybe apply to the retrieval pipeline a few times to get a couple of different results back.
366
00:30:19.650 --> 00:30:21.410
And that became possible
367
00:30:21.410 --> 00:30:29.624
as our LLMs became possible too. So it didn't just happen in a vacuum. We had longer context windows. You can not just put five results,
368
00:30:29.705 --> 00:30:31.624
and that's all space you had to have.
369
00:30:32.025 --> 00:30:36.585
You could do 10 or 20 or 50 or a 100 because the the context windows became
370
00:30:36.825 --> 00:30:40.664
longer, and you can put more results in there. So AgenTiGrag,
371
00:30:40.664 --> 00:30:45.509
the primary value I see is that ability to do multiple iterations
372
00:30:45.830 --> 00:30:51.509
and also rewrite the queries. So a lot of times if you run a query the way the user wrote it,
373
00:30:52.070 --> 00:30:53.350
it may not be
374
00:30:53.670 --> 00:31:01.275
the correct way that is best suited for accurate retrieval. Right? User may have an acronym there, for example,
375
00:31:01.675 --> 00:31:02.235
that
376
00:31:03.275 --> 00:31:10.075
is unknown to the system, and the LM might actually translate it using some glossary that you give it or something like that.
377
00:31:10.810 --> 00:31:15.690
So that's been actually really helpful to just improve accuracy of retrieval in general.
378
00:31:16.490 --> 00:31:21.370
And then this evolved into what we know today as AI agents. Right? So agents are
379
00:31:22.745 --> 00:31:30.425
sort of an evolution of that where we have what we call the harness, which is the evolution of the AgenTiC Rag loop, the simple loop that we had at the beginning,
380
00:31:30.665 --> 00:31:32.345
and now it has access to tools.
381
00:31:32.745 --> 00:31:39.309
So if you think about it in that in that sense, AgenTiC Rag was an agent with one tool for retrieval
382
00:31:39.710 --> 00:31:43.309
using the Rag kind of retrieval pipeline without the generation.
383
00:31:43.550 --> 00:31:46.270
And now we can have access to tools anywhere from
384
00:31:46.430 --> 00:31:52.270
retrieval tools like with the Rag, but also tools that go directly to target systems. Like there's
385
00:31:52.565 --> 00:31:56.004
tools to query databases with text to SQL, for example.
386
00:31:56.805 --> 00:32:01.845
There's tools to go to the native systems, like, you know, search capabilities,
387
00:32:01.845 --> 00:32:02.404
in
388
00:32:02.645 --> 00:32:07.285
in different places in Notion and other places. They all provide this kind of API type capabilities
389
00:32:07.470 --> 00:32:08.830
all through MCP.
390
00:32:08.830 --> 00:32:13.149
And that's what the the harnesses we have today for for AI agents building.
391
00:32:14.190 --> 00:32:19.710
And that idea of having the tools be the retrieval
392
00:32:20.510 --> 00:32:22.190
engine has also
393
00:32:23.315 --> 00:32:31.715
maybe diminished some of the initial supremacy of vector databases as the end all be all for that retrieval step
394
00:32:31.955 --> 00:32:32.674
and
395
00:32:32.755 --> 00:32:37.315
changes some of the equations as far as what you need to do for the
396
00:32:37.920 --> 00:32:53.440
seeding of the data where you don't necessarily have to do the whole pipeline of collect the data, chunk it, stick it into a vector database full of embeddings in order to be able to get anything out of it. But it is still a necessary element. I'm curious how you're seeing that
397
00:32:54.265 --> 00:33:07.990
change the ways that people are thinking about their overall retrieval and context system, particularly for these more agentic use cases as well as some of the way... Reasons why somebody might want to stick with a more straightforward
398
00:33:08.070 --> 00:33:11.270
rag system in the kind of first generation
399
00:33:11.270 --> 00:33:12.230
definition.
400
00:33:13.430 --> 00:33:14.870
Yeah. Excellent.
401
00:33:14.870 --> 00:33:17.110
Excellent point. So, you know,
402
00:33:17.430 --> 00:33:22.630
dragging all the data over to ingest it into the sort of the the rag pipeline
403
00:33:22.924 --> 00:33:40.330
was hard. You had to connect to the source system. You had to essentially replicate all the data. So you had to duplicate. Those were the downsides. Right? And you had, like we talked earlier, we had to refresh it every once in a while. So you had to sort of create like a hot copy of the data only in a way that's,
404
00:33:40.650 --> 00:33:46.090
sort of in a vector database or in a lexical database for ready to be queried.
405
00:33:46.330 --> 00:33:56.705
But it didn't have the advantage of semantic search and hybrid search and all these things we talked about that created better results. And so today, you're right, as things evolve,
406
00:33:57.025 --> 00:33:58.545
more and more vendors
407
00:33:59.105 --> 00:33:59.985
created
408
00:34:00.065 --> 00:34:01.825
tools for their system
409
00:34:01.985 --> 00:34:07.159
to be able to create that system. But the downside of that still, and it keeps improving,
410
00:34:07.160 --> 00:34:08.920
is that you are dependent
411
00:34:09.080 --> 00:34:12.760
on the quality of retrieval of that tool. Right? So if I call...
412
00:34:13.080 --> 00:34:18.360
And and I'm I'm... I wanna say Notion, but I think actually Notion search is not that bad. Not not
413
00:34:19.080 --> 00:34:24.345
kind of telling anything about them, but just because it's a it's a common thing. If you go to...
414
00:34:25.065 --> 00:34:27.705
If you call a tool of Notion instead of bringing
415
00:34:27.865 --> 00:34:32.025
it back to search, you depend on whether Notion Search is good or not.
416
00:34:33.080 --> 00:34:36.600
Does it do multimodal search properly? Does it do
417
00:34:37.080 --> 00:34:40.920
the text search properly? All these things we talked about become important.
418
00:34:41.320 --> 00:34:42.360
And if they're not,
419
00:34:42.760 --> 00:34:48.785
then it doesn't help your agent. It's not gonna get the right information. Similarly with Google Drive, similarly with SharePoint,
420
00:34:48.785 --> 00:34:51.985
similarly with, you know, Jira or or anything else that you have.
421
00:34:52.225 --> 00:34:54.545
So that's the the trade off you make.
422
00:34:54.945 --> 00:35:02.705
And I think as an industry, we're kind of in the middle of that. So over the last two years, a lot of these sort of API search functionality has improved.
423
00:35:03.599 --> 00:35:11.760
But not all these companies have, you know, it's it's... You know, search is a kind of its own little sets of capabilities
424
00:35:11.839 --> 00:35:13.680
and sort of know how.
425
00:35:13.920 --> 00:35:24.535
And I think it's not super easy for all of them to sort of improve the search to a level that it can be as good as you might need it to be. The other challenge is a lot of companies are now moving to
426
00:35:25.095 --> 00:35:25.975
kind of
427
00:35:26.695 --> 00:35:29.975
what's called... There's a word for it. It's called AI sovereignty.
428
00:35:29.975 --> 00:35:36.020
Right? Like... But kind of on premise. Right? When you go to on premise... And we haven't touched this yet, but, like, just to...
429
00:35:36.500 --> 00:35:37.540
On this point,
430
00:35:37.860 --> 00:35:41.860
like, they want everything in control. So they're calling a local LLM.
431
00:35:42.420 --> 00:35:43.460
Maybe it's
432
00:35:43.940 --> 00:35:44.820
OpenAI's
433
00:35:44.820 --> 00:35:58.395
open source model, or maybe it's the NVIDIA one or or thinking machines, whatever it may be, it's all... Nothing goes outside. Nothing that goes outside. And so again, if you have to do that, a lot of times you might wanna ingest the data
434
00:35:59.210 --> 00:36:04.089
earlier into that local environment so they can be part of what can be queried,
435
00:36:04.329 --> 00:36:08.650
and you don't have to make any external API calls in sort of these air gapped environments.
436
00:36:09.609 --> 00:36:13.769
One of the other interesting pieces of bringing agents into the mix is,
437
00:36:14.335 --> 00:36:15.375
in particular,
438
00:36:15.535 --> 00:36:18.495
on the side of data pipelines where
439
00:36:19.455 --> 00:36:34.310
the first generation of RAG required a lot of upfront engineering effort to build that pipeline of doing the chunking and embedding, etcetera, keeping that up to date. Now agents are doing a lot of the work of actual code generation. They're being brought into data pipelines,
440
00:36:34.310 --> 00:36:36.310
and so that also potentially
441
00:36:36.550 --> 00:36:39.510
acts as an input to the RAG system,
442
00:36:39.750 --> 00:36:43.349
which can be yet another source of errors or improvements.
443
00:36:43.885 --> 00:36:47.965
And I'm wondering what are some of the ways that you're seeing that come full circle where
444
00:36:48.045 --> 00:36:50.365
we're bringing LLMs and agents
445
00:36:50.525 --> 00:36:59.910
fully into the process of managing the end to end system for the data seeding as well as doing the retrieval and as well as doing the evaluation
446
00:37:00.470 --> 00:37:04.230
and just some of the ways that that maybe unlocks some new patterns that you're seeing?
447
00:37:05.030 --> 00:37:07.590
Yeah. Quick question. So, for example,
448
00:37:07.910 --> 00:37:14.335
there's a lot of different places. I see, for example, an ingesting of data. Right? There used to be a data engineering pipeline,
449
00:37:14.974 --> 00:37:20.255
that did sort of, let's say, take a PDF file. You have to parse the PDFs, put it into pages,
450
00:37:21.454 --> 00:37:30.800
see where tables and images are, do the chunking, all that stuff. Now people do it with with agents. Right? You have an agent that just does all that process for you. That's one area.
451
00:37:31.279 --> 00:37:32.000
Generally,
452
00:37:32.079 --> 00:37:36.079
the other interesting thing is all these systems used to be with APIs,
453
00:37:36.400 --> 00:37:44.935
and they had very elaborate documentation about the APIs. And you had to, as an engineer, as a developer, have to read all of that and figure out how to implement it and
454
00:37:45.335 --> 00:37:52.935
if it has changed and everything. Today, you know, I run Code or Codecs or Copilot or whatever, and I just point it at the APIs and
455
00:37:53.415 --> 00:37:53.975
that's it.
456
00:37:55.569 --> 00:38:01.570
It produces a really good system for querying for my actual application that uses the API.
457
00:38:01.970 --> 00:38:06.050
It produces a good thing for indexing the data or ingesting the data.
458
00:38:06.369 --> 00:38:11.089
So like everywhere else in engineering, the coding agents are fantastically powerful
459
00:38:12.025 --> 00:38:14.665
to help us build system much, much faster.
460
00:38:15.545 --> 00:38:16.425
And honestly,
461
00:38:17.865 --> 00:38:19.065
with less
462
00:38:20.105 --> 00:38:34.160
mistakes, like we still make mistakes as humans. I know we all say like, oh, is the coding agent really doing a good job? And in some cases, it does not, and we can talk about that too. There's... I have some experience with that too. One of the things I like to do is to write something in Cloud Code and then
463
00:38:34.400 --> 00:38:54.820
have codex review it, kinda have this adversarial between them. Because I found that a lot of times that really helps. You show something to Cloud Code and you ask Codex to review it, it says, oh, I found three bugs that CodeCode said everything was okay. Or or the other way around. It's not, I guess, one or the other. But, anyway, getting to your earlier point, yeah, I think, I've seen this in ingestion.
464
00:38:55.140 --> 00:38:56.180
I've seen this in,
465
00:38:57.300 --> 00:38:58.580
sort of just writing
466
00:38:58.660 --> 00:39:00.820
other parts of, the system.
467
00:39:01.140 --> 00:39:10.885
I think in the infrastructure of the of the the right pipelines, like where embedding happens and things like this on the retrieval itself, I haven't seen a lot of people use it yet.
468
00:39:11.204 --> 00:39:21.430
And also in evaluation, I haven't seen it yet, but that's just maybe me. I haven't, you know, bumped into people to do that. I I could expect that this could be helpful in any any part of this of this. Another
469
00:39:21.510 --> 00:39:26.630
term that has been percolating through the ecosystem is context engineering,
470
00:39:26.710 --> 00:39:27.190
which
471
00:39:27.830 --> 00:39:38.875
if you squint is the same thing as rag, but people like to come up with new terms for the same thing. And I'm wondering what are some of the ways that you think about the overlap and some of the differentiation
472
00:39:38.875 --> 00:39:42.715
between the ideas of Rag and the idea of context engineering.
473
00:39:43.434 --> 00:39:57.360
So, yeah, the context engineering is... Let let me sort of first state what I kind of my understanding of it just so we align on because the terms are... Could be very confusing. Right? So context engineering is this idea that you as the engineer developing your system
474
00:39:57.680 --> 00:39:58.720
control
475
00:39:59.244 --> 00:40:03.245
what fits into the context of the LM before you send a request. Right?
476
00:40:03.645 --> 00:40:10.365
So by the way, when I started using GPT two, for those who don't know in the audience, it had 512
477
00:40:10.365 --> 00:40:16.740
tokens context limit. That's it. We think about now we have a million or 10,000,000. It it sounds ridiculous, but,
478
00:40:17.140 --> 00:40:19.780
that was it. And then it became a thousand,
479
00:40:20.339 --> 00:40:23.140
ten twenty four. I was like, oh my god. This is double.
480
00:40:23.220 --> 00:40:24.580
Like... So,
481
00:40:25.059 --> 00:40:27.059
context has grown a lot, which is
482
00:40:27.655 --> 00:40:28.455
amazing.
483
00:40:28.615 --> 00:40:30.695
But... So context engineering
484
00:40:30.855 --> 00:40:32.535
now is this practice
485
00:40:32.535 --> 00:40:35.975
of when you create your system of deciding
486
00:40:36.855 --> 00:40:42.055
what's in your system prompt, what do you tell the the agent to be like its guiding principles,
487
00:40:42.400 --> 00:40:44.560
And what sort of intermediate
488
00:40:44.640 --> 00:40:47.840
data you bring into the the same context,
489
00:40:48.880 --> 00:40:50.960
that you put in front of the LLM
490
00:40:51.119 --> 00:41:02.205
in terms of... We think about RAC, it could be chunks that come in. If it's a more advanced agent, it could be sort of the tools that it may be allowed to call and their definitions.
491
00:41:02.765 --> 00:41:05.405
It could be historical information about the user.
492
00:41:05.485 --> 00:41:09.965
People now talk about memory. So there's a lot of things that you can pull into the the context,
493
00:41:10.605 --> 00:41:25.320
but it's the same thing. It's it's how do you take all this stuff and then you send it to the LM and ask it for help to do something, either call one of the tools, call many of the tools, whatever it needs to do, that'll answer the question or perform a task for you.
494
00:41:25.974 --> 00:41:30.935
Yeah. So that's kind of the evolution that, one thing I I I I should mention is,
495
00:41:31.974 --> 00:41:33.895
one way I think about RAG
496
00:41:34.055 --> 00:41:35.095
in a way
497
00:41:35.415 --> 00:41:41.720
is a little bit... I don't... I haven't heard it anywhere else, but I thought it was an interesting way to think about it is REG could be
498
00:41:42.040 --> 00:41:45.400
the kind of attention mechanism for context engineering.
499
00:41:45.560 --> 00:41:53.720
And what do I mean by that? People know about attention, you know, the famous word in deep learning. Right? Attention is a mechanism in deep learning itself where
500
00:41:54.045 --> 00:41:55.085
it sort of
501
00:41:55.724 --> 00:41:57.245
looks at the history
502
00:41:57.565 --> 00:41:58.605
of the
503
00:41:58.925 --> 00:42:04.365
the conversation so far and picks the right words or the right tokens in the history
504
00:42:04.605 --> 00:42:10.090
that are more important to generate the next token, right? That's what the retention mechanism is
505
00:42:10.250 --> 00:42:19.610
in general. And so sort of taking that idea and saying, I have a question for the user now, and I have all of my corporate, all my enterprise data out there.
506
00:42:20.335 --> 00:42:26.335
And REG sort of picks the right sets of information out of there that needs to be pulled in
507
00:42:26.655 --> 00:42:29.215
to the context of the LLM
508
00:42:29.535 --> 00:42:33.455
in that sense. And so I think of it as, again, a selection... Selector
509
00:42:33.855 --> 00:42:34.575
operator
510
00:42:34.800 --> 00:42:37.280
that helps you pick what's important,
511
00:42:38.000 --> 00:42:41.680
into the the context engineering sort of, landscape.
512
00:42:42.560 --> 00:42:58.775
And as you have been working in this space and particularly in your capacity doing developer relations, working with people out in the community? What are some of the most interesting or innovative or unexpected ways that you've seen teams deal with the production requirements of RAG systems?
513
00:42:59.575 --> 00:43:05.140
The first thing I I wanted to mention there is not necessarily unexpected, actually expected,
514
00:43:05.140 --> 00:43:06.660
but important
515
00:43:06.900 --> 00:43:09.540
is how do you deal with roles and permissions.
516
00:43:10.260 --> 00:43:11.300
Right? So
517
00:43:11.700 --> 00:43:16.660
this is an important problem in Rag and in agents. Right? So I... Let's say I have
518
00:43:17.345 --> 00:43:20.225
all my Google Drive ingested into my Rag pipeline.
519
00:43:20.464 --> 00:43:24.465
Maybe the salaries of the company, there's Excel sheet that's all... Also there.
520
00:43:24.865 --> 00:43:27.265
Only the CEO and HR have access to it.
521
00:43:27.664 --> 00:43:32.305
But if I ingest it without any protections, then everybody can can pull information out of that. Right?
522
00:43:33.480 --> 00:43:35.000
So you wanna make sure
523
00:43:35.240 --> 00:43:44.680
that you have this, role based access control the same way the company implements it be applied in your retrieval pipeline. That's a really important thing to do.
524
00:43:45.160 --> 00:43:50.145
And the... I guess the kind of the the innovative ways people implemented this
525
00:43:50.865 --> 00:43:52.065
is just by
526
00:43:52.385 --> 00:43:54.465
implementing some form of filtering.
527
00:43:54.865 --> 00:43:59.984
Right? So if you say attach to every document that you ingest into your pipeline,
528
00:44:00.730 --> 00:44:05.690
you touch some some notion of permission. And, again, permission is a whole world of complexity.
529
00:44:06.250 --> 00:44:09.210
And a lot of companies have multiple permissioning systems,
530
00:44:09.370 --> 00:44:13.450
and, you know, it's it's a mess. But if you can normalize your permission,
531
00:44:13.875 --> 00:44:18.755
your role into something that you can sort of add as, say, metadata to the document,
532
00:44:19.395 --> 00:44:25.715
and then when a query comes in, you can sort of say, if if the CEO is asking the question,
533
00:44:26.690 --> 00:44:28.130
that... Then I'm gonna
534
00:44:28.450 --> 00:44:35.570
I'm gonna allow them to see everything. If it's, somebody on the... Whatever, on on another team that's not supposed to see the salaries,
535
00:44:36.050 --> 00:44:41.195
then he would then filter the documents that are sort of have the the permission block
536
00:44:41.435 --> 00:44:45.035
in the metadata and not include that in the checks even if it's relevant.
537
00:44:45.435 --> 00:44:47.915
So so the overs... Supersede
538
00:44:48.155 --> 00:44:50.315
relevance with with filtering beforehand.
539
00:44:50.970 --> 00:44:52.250
And that's been
540
00:44:52.650 --> 00:44:53.530
really helpful.
541
00:44:53.690 --> 00:44:54.730
That's one thing.
542
00:44:55.450 --> 00:44:58.970
I think multimodal has been an area that I've seen a lot of innovation.
543
00:44:59.530 --> 00:45:01.690
So I mentioned this a little bit in the beginning
544
00:45:02.090 --> 00:45:09.515
on the indexing or the ingestion side. But think about multimodal also on the UI side. Right? Your application needs to show
545
00:45:10.235 --> 00:45:14.635
the response. But if you have an image, you kinda wanna integrate it into the response.
546
00:45:15.115 --> 00:45:19.435
So, again, a lot of people don't think about it. If you have a video, you wanna embed the video in there to show
547
00:45:19.830 --> 00:45:25.430
where the response was retrieved from and maybe even point the video to the right second.
548
00:45:25.750 --> 00:45:30.630
Right? So you kinda wanna have some innovation on the UI side to make the user experience, like, really helpful.
549
00:45:31.350 --> 00:45:32.150
Ultimately,
550
00:45:32.310 --> 00:45:35.590
everywhere else, every other application we've ever built
551
00:45:35.885 --> 00:45:41.965
in tech, people are lazy. They wanna get the answers right away, and they want it to be appropriate for them. So you kinda work
552
00:45:42.365 --> 00:45:46.925
really hard to make this work in this way in in multi... Multimodal data.
553
00:45:47.580 --> 00:45:52.780
I think the other thing is, as I mentioned, I think people are working really hard to make it
554
00:45:53.100 --> 00:45:54.220
be viable,
555
00:45:54.300 --> 00:45:55.820
both REG and AgenTik,
556
00:45:56.140 --> 00:45:57.580
in sort
557
00:45:58.460 --> 00:46:00.140
of the air gapped environments.
558
00:46:00.365 --> 00:46:04.445
That's a really important use case for a lot of big companies.
559
00:46:04.845 --> 00:46:05.885
Think about,
560
00:46:06.845 --> 00:46:13.885
you know, people working in the government industry where there's, like, different whatever agencies and things like this that they can never leave, or
561
00:46:14.460 --> 00:46:17.340
companies in the defense industry or semiconductors
562
00:46:17.340 --> 00:46:18.860
where there's a lot of competition.
563
00:46:19.180 --> 00:46:20.540
Those are areas where we see,
564
00:46:20.940 --> 00:46:25.580
even financial services, again, a lot of different things. So if you unpack that,
565
00:46:26.140 --> 00:46:27.980
you have to do a lot of innovative stuff
566
00:46:29.465 --> 00:46:35.625
to do this. So one of the surprising things that I bumped into was that, well, somewhat surprising, would say, is
567
00:46:36.185 --> 00:46:43.865
people are afraid to use the Chinese models, right? Chinese models, the open source models are very, very good in terms of performance, right? They are
568
00:46:45.720 --> 00:46:48.600
on par with the commercial models that are API based
569
00:46:49.080 --> 00:46:53.400
to some... Almost almost there, I think, like, MiniMax three,
570
00:46:53.480 --> 00:47:00.535
Kymi two, you know, three. Like, there's a lot of recent models, QN, like DeepSeek. All of them are... By the benchmarks, they're really good.
571
00:47:01.095 --> 00:47:02.695
And they can be hosted locally,
572
00:47:02.855 --> 00:47:09.734
and there's not too many models that compete that are sort of by US companies. But I see a lot of resistance of using these models
573
00:47:10.320 --> 00:47:11.040
in
574
00:47:11.280 --> 00:47:13.600
at least US and European based companies.
575
00:47:13.840 --> 00:47:14.240
And,
576
00:47:15.520 --> 00:47:18.800
I can see why that would be concerning to some extent, but...
577
00:47:20.000 --> 00:47:32.615
And so I I really like, for example, that, thinking machines recently came with a very good use based model, and I see there's a lot of movement there. So that's something that initially surprised me because open source weights are just weights.
578
00:47:32.855 --> 00:47:33.975
They can't, like...
579
00:47:34.295 --> 00:47:37.255
I don't see... As a technical person, I don't see a way
580
00:47:37.700 --> 00:47:41.540
how this... They would send something or the Chinese could control them.
581
00:47:41.940 --> 00:47:46.420
Maybe I haven't thought this through enough, and I'm not an expert on security and
582
00:47:47.220 --> 00:48:04.135
relationships like this. Maybe it's about the fact that people might get used to them. And then once there's a need for additional code, they'll forget about it or or something like this. I I don't know. I'm really not sure about this. But I've seen a lot of resistance to that for sure in this sort of, server and AI space.
583
00:48:04.775 --> 00:48:09.450
And in your experience of working in this area of
584
00:48:09.610 --> 00:48:16.170
building and evaluating RAG systems, what are some of the most interesting or unexpected or challenging lessons that you learned personally?
585
00:48:17.290 --> 00:48:21.225
Well, first of all, it's something I mentioned earlier that teams don't think about
586
00:48:21.625 --> 00:48:23.945
the challenges upfront. They just...
587
00:48:24.265 --> 00:48:36.140
Mean, they rush to build a system. And like we said earlier, it's... It was a lot... It used to be a lot of engineering. Now it's a little bit less, but it's a lot of work to build a scalable, robust sort of, like, system
588
00:48:36.460 --> 00:48:38.060
with all the bits and pieces.
589
00:48:38.300 --> 00:48:39.340
And a lot of teams...
590
00:48:39.820 --> 00:48:40.940
I was I was
591
00:48:41.820 --> 00:48:43.580
surprised that people don't
592
00:48:43.580 --> 00:48:50.825
apply the same rigor that we do to any other system that we ever built in microservices or whatever other technology in the past to
593
00:48:51.065 --> 00:48:51.945
the system.
594
00:48:52.265 --> 00:48:54.025
I think part of it is because
595
00:48:54.825 --> 00:48:59.865
this has been a lot... It's happened so fast and there's been a lot of push like, hey, we gotta go into AI. We gotta
596
00:49:00.279 --> 00:49:04.839
AI fire systems or whatever it may be. Right? So I think that's
597
00:49:05.799 --> 00:49:08.440
that's something that I've seen, like, the rush
598
00:49:08.680 --> 00:49:15.395
and not thinking through it and then being surprised that it's more complicated than we thought when you go to production scale.
599
00:49:15.795 --> 00:49:19.475
And the second thing is really... And we talked about it too, is, like, just devaluation,
600
00:49:19.475 --> 00:49:21.475
how how little people think about it.
601
00:49:21.875 --> 00:49:24.195
And they're then surprised that
602
00:49:24.515 --> 00:49:28.515
results are not good, and then they put it... Slap it on the system after the fact.
603
00:49:29.530 --> 00:49:34.810
And then realize, okay, how how difficult it is. It's not just creating questions and answers that becomes complicated.
604
00:49:35.210 --> 00:49:36.890
So people are used to
605
00:49:37.770 --> 00:49:44.515
regular applications, right, where where the metrics... So people understand metrics. Right? You have to have measure latency, measure downtime,
606
00:49:44.515 --> 00:49:46.115
you know, all these things that we're
607
00:49:46.915 --> 00:49:50.355
accustomed to from the kind of DevOps world. Right? But
608
00:49:50.915 --> 00:49:52.195
these new metrics
609
00:49:52.275 --> 00:49:55.955
that are ML based and sort of LLM based are new
610
00:49:56.300 --> 00:50:00.060
and more difficult and people, like, don't think about them upfront.
611
00:50:00.460 --> 00:50:05.260
What I've seen is if you you have an ML team internally that is powerful and strong,
612
00:50:05.580 --> 00:50:13.645
then they might tell you about this and you might think about it upfront, but many companies don't, and then they they don't think about it in that sense.
613
00:50:13.965 --> 00:50:17.165
And as you continue to explore
614
00:50:17.245 --> 00:50:25.440
and work in this space and keep tabs on the ways that the industry is evolving, what are some of the predictions or
615
00:50:25.520 --> 00:50:30.800
hopes that you see for the future of RAG systems and context engineering
616
00:50:30.800 --> 00:50:36.160
and just the overall space of data driven LLM and agentic applications?
617
00:50:38.445 --> 00:50:39.005
So...
618
00:50:39.565 --> 00:50:42.765
Yeah. First of all, I think retrieval will continue to evolve,
619
00:50:43.245 --> 00:50:49.005
not... Probably without much the g. The g is still... Some people still use NaiveRag
620
00:50:49.005 --> 00:50:57.550
in in the old way, and it's available for them. But I think more and more will go in... Certainly go into AgenTik, and we'll just use the retrieval part where it fits
621
00:50:57.790 --> 00:51:05.630
and continue to improve that. I still see new embedding models being created that are better, new ranking models that are better. So there's there's still work to do.
622
00:51:06.295 --> 00:51:07.335
People use
623
00:51:07.495 --> 00:51:09.015
glossaries, for example,
624
00:51:09.335 --> 00:51:09.895
to...
625
00:51:10.295 --> 00:51:15.655
And build models that are specific. They're fine tuned specifically to that particular
626
00:51:16.055 --> 00:51:16.855
industry,
627
00:51:17.015 --> 00:51:20.455
which helps with fields. So there's a lot a lot more to do in retrieval and search.
628
00:51:21.170 --> 00:51:23.170
I definitely think we will see the
629
00:51:23.490 --> 00:51:24.370
retrieval
630
00:51:24.370 --> 00:51:27.250
of other systems that you connect to directly
631
00:51:27.410 --> 00:51:31.890
using a MCP tool or something like this become better and
632
00:51:32.130 --> 00:51:33.090
create better results.
633
00:51:33.835 --> 00:51:36.635
I still see a a big gap in the multimodality
634
00:51:36.635 --> 00:51:41.195
in that respect, and I hope those those vendors will will keep improving that as well.
635
00:51:41.595 --> 00:51:44.155
So that's one thing. Context engineering,
636
00:51:44.235 --> 00:51:47.515
again, is... Can used to be a buzzword that's very popular.
637
00:51:48.570 --> 00:51:52.730
It kinda goes into what people think of as memory systems now.
638
00:51:53.290 --> 00:51:56.650
So people talk about, like, the immediate memory, like,
639
00:51:57.210 --> 00:51:59.850
in a in a particular session in a chatbot.
640
00:52:00.330 --> 00:52:09.845
How do we talk, you know, back and forth me in the chatbot? And so if I say something that we talked about before, I wanna remember that. That's the simplest form. But then there's things like,
641
00:52:10.164 --> 00:52:12.565
oh, twenty days ago, we talked about something
642
00:52:13.204 --> 00:52:14.405
related to,
643
00:52:14.964 --> 00:52:16.885
I don't know, maybe
644
00:52:17.200 --> 00:52:18.880
different banks in my area.
645
00:52:19.600 --> 00:52:28.160
And I told you, the agent, that I like this bank, so don't ask me that again. So, like, historical context of earlier conversations.
646
00:52:28.480 --> 00:52:31.680
And I think, both of these are really interesting,
647
00:52:31.760 --> 00:52:34.775
and people attack them in kind of, different ways.
648
00:52:35.335 --> 00:52:42.135
But I think, again, that'll probably be commoditized and be, like, certain two or three approaches that people do, and they'll be really easy to use.
649
00:52:42.454 --> 00:52:47.095
In terms of agents, of course, I think there's a lot of interesting things coming up. So one...
650
00:52:47.720 --> 00:52:52.920
And let me talk about regular agents and then coding agents because I like talking about coding agents separately too.
651
00:52:53.079 --> 00:52:54.280
On regular agents,
652
00:52:54.519 --> 00:53:02.760
you know, the agent... The LLMs do really well with calling tools and deciding what to do with agents. There's no doubt about it. If you compare it to a year ago, it's incredibly better.
653
00:53:03.575 --> 00:53:08.455
But we've seen the agents move to what's called long term horizon
654
00:53:08.535 --> 00:53:09.575
sort of things.
655
00:53:09.974 --> 00:53:11.655
And I think they'll still suffer
656
00:53:12.295 --> 00:53:16.295
from accuracy when the task is... Takes a lot of steps
657
00:53:16.779 --> 00:53:19.099
Because what happens is the context becomes
658
00:53:19.500 --> 00:53:20.540
sort of like
659
00:53:21.980 --> 00:53:29.259
poison a little bit. There's like what's called context rot a little bit. And it'll start doing the wrong things a little bit or asking questions again.
660
00:53:30.015 --> 00:53:36.815
So for example, when I use a coding agent like Cloud Code or Codex, a lot of times when I use the same session for a long time,
661
00:53:37.855 --> 00:53:39.615
after about 200,000
662
00:53:39.615 --> 00:53:41.615
tokens or something, I start seeing it
663
00:53:41.934 --> 00:53:43.934
not act as well. And I
664
00:53:44.079 --> 00:53:55.599
sometimes benefit from, like, killing it starting a new session. Right? So there's... This happens in regular agents too. And I think we're... We have to... As an industry, I'm looking forward to seeing what solutions people come up for for that.
665
00:53:56.475 --> 00:53:59.835
On coding agents, I think we're seeing this interesting
666
00:54:00.315 --> 00:54:06.715
evolution from one coding agent to a swarm of agents. And that's really something we're working on at Band
667
00:54:07.515 --> 00:54:11.435
about creating this... Using Band to... Which is agent agent communication
668
00:54:12.099 --> 00:54:15.140
to create this idea of multiple coding agents
669
00:54:15.460 --> 00:54:18.819
that talk to each other to solve a problem, that that collaborate
670
00:54:18.819 --> 00:54:19.540
together.
671
00:54:19.700 --> 00:54:22.740
Just like we humans in a team can collaborate together. Right?
672
00:54:23.300 --> 00:54:25.140
So I think this idea of
673
00:54:25.645 --> 00:54:26.765
coding agents
674
00:54:26.765 --> 00:54:41.599
that specialize in a particular area, so you can have a coding agent to write a code, a coding agent to plan, a coding agent to review the code or to review the plan, a coding agent for compliance, a coding agent for machine learning, for design, for different roles.
675
00:54:42.400 --> 00:54:49.040
And they have a very precise context. They don't have context of everything, so they can work more precisely on each of these.
676
00:54:49.599 --> 00:55:01.984
And then they collaborate and hand off work to each other as needed when they need to develop code for us and and do that. So I think in my mind, this will create a a much better outcome for everybody.
677
00:55:02.224 --> 00:55:03.744
And especially if you use
678
00:55:04.865 --> 00:55:08.305
multiple agents, you can take advantage of
679
00:55:08.799 --> 00:55:17.119
Clubcode and Codex and Copilot and, I know, Kiro or everybody else. And there's gonna be so many of these, coding agents produced in the antigravity.
680
00:55:17.520 --> 00:55:20.000
Because every one of those has a little bit of a different harness,
681
00:55:20.815 --> 00:55:22.495
so it behaves a little bit differently.
682
00:55:22.655 --> 00:55:30.655
It has a different LLM that drives it. So you will see... You will keep seeing them. I think we all will keep seeing them sort of have
683
00:55:30.655 --> 00:55:31.855
different personalities,
684
00:55:31.855 --> 00:55:33.375
different, like, strength and weaknesses.
685
00:55:34.430 --> 00:55:35.710
And as we get to
686
00:55:35.950 --> 00:55:38.030
complex code generation,
687
00:55:38.910 --> 00:55:40.590
long horizon tasks,
688
00:55:40.830 --> 00:55:45.150
benefiting from sort of adversarial review and seeing multiple,
689
00:55:45.310 --> 00:55:47.950
points of view would actually create higher quality outcomes.
690
00:55:49.225 --> 00:55:51.705
Are there any other aspects of
691
00:55:51.865 --> 00:55:59.305
the work that you're doing in this space or just the overall challenge of building and operating and maintaining these
692
00:55:59.305 --> 00:56:03.545
rag based systems that we didn't discuss yet that you'd like to cover before we close out the show?
693
00:56:04.800 --> 00:56:20.685
Well, I wanted to mention one more area, which is kinda SQL based data that we haven't talked about, and that's interesting. So there is a lot of work in text to SQL. And, again, think of it as a tool that agents might call. The reason it's interesting is very obvious, right? Like most mission critical
694
00:56:20.845 --> 00:56:21.565
data
695
00:56:21.885 --> 00:56:29.325
is still in databases, in Snowflake or Databricks or Microsoft or whatever it is, right? And Text to SQL is going through...
696
00:56:29.805 --> 00:56:38.430
First of all, it's really good, Right? It's still... It it is pretty good now. And it's gonna get better over time. There's models being developed specifically to solve this problem.
697
00:56:39.550 --> 00:56:41.070
But I think one of the
698
00:56:41.310 --> 00:56:42.510
the interesting
699
00:56:43.070 --> 00:56:44.030
things that are happening
700
00:56:44.875 --> 00:56:49.035
is it's related to this world of semantic modeling, right? There's
701
00:56:49.915 --> 00:56:55.515
was Looker that started this, and there's now the new version of this called Malloy, which is open source.
702
00:56:55.915 --> 00:57:02.280
And there's other companies that do this kind of thing to build semantic models to enable enterprise to
703
00:57:02.360 --> 00:57:03.640
sort of define
704
00:57:04.120 --> 00:57:09.320
what something means in SQL. Because the LM, again, like with REG, doesn't have context
705
00:57:09.400 --> 00:57:10.680
of what things mean.
706
00:57:11.160 --> 00:57:14.120
So if you ask it, tell me about the churn
707
00:57:14.744 --> 00:57:16.825
in our recent customer base,
708
00:57:16.984 --> 00:57:26.984
churn can be defined as somebody that dropped off for fifteen days or thirty days or a hundred days or whatever. So there's no definition the LM knows is the the absolute truth.
709
00:57:28.040 --> 00:57:37.880
And so the ability to define this is really important. And it was important before we had this with data and analysts and things like this, but now we have to define it for agents
710
00:57:38.119 --> 00:57:43.640
because there's gonna be so many more agents asking so many more questions out of these stacks of SQL pipelines
711
00:57:43.914 --> 00:57:50.395
that we need to enable, make sure that if agents are gonna do this at scale and we can trust it, it has to be enabled.
712
00:57:50.875 --> 00:57:52.714
So it's just one thing I wanted to mention,
713
00:57:53.194 --> 00:57:55.035
that we didn't didn't cover.
714
00:57:55.994 --> 00:58:03.530
Yeah. That's definitely still an interesting space, and I see the word semantic layer and semantic modeling
715
00:58:03.530 --> 00:58:07.850
probably at least once a day, definitely multiple times a week these days.
716
00:58:08.730 --> 00:58:24.075
Alright. Well, for anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gaps in the tooling technology or human training that's available for AI systems today.
717
00:58:25.250 --> 00:58:29.170
The thing that... First of all, let me start with human training.
718
00:58:29.410 --> 00:58:31.650
I still see a huge gap
719
00:58:31.810 --> 00:58:49.585
in terms of people adopting AI. Now I think you and I and maybe others in the audience, some of us live in the tech world, and it's... We track these things every... We try our best actually to track these things every day. Right? It's it's so difficult. There's, like, 50 things happening every week, and it's hard to to keep track of everything. But,
720
00:58:50.625 --> 00:58:57.980
AI is so powerful and so useful in a lot of different ways. And I'd love to see more. I see a lot, but I'd love to see more
721
00:58:58.140 --> 00:59:00.940
people and companies help with education
722
00:59:00.940 --> 00:59:01.660
of
723
00:59:02.060 --> 00:59:05.020
everybody of how to take advantage of AI.
724
00:59:05.260 --> 00:59:06.940
This started from people
725
00:59:07.095 --> 00:59:12.215
don't even know how to use ChatGPT. Now I think more people know how to use that, but, like, how to use it effectively,
726
00:59:12.935 --> 00:59:18.215
how to connect it to your your own, you know, systems and things like this. Very, very important.
727
00:59:18.615 --> 00:59:23.330
I also... And a little bit in a narrower context, but related to that, I think
728
00:59:23.810 --> 00:59:26.850
the education system itself, the education industry,
729
00:59:26.930 --> 00:59:30.290
teaching, you know, kids, teaching adults
730
00:59:30.370 --> 00:59:31.890
that do later education,
731
00:59:32.210 --> 00:59:33.810
I think has to change. Right?
732
00:59:34.365 --> 00:59:35.805
We have to integrate,
733
00:59:35.885 --> 00:59:42.605
in my mind, how to use AI effectively in every part of education. And it's a massive shift
734
00:59:42.845 --> 00:59:47.325
that has to happen because learning, memorizing things, and and,
735
00:59:47.565 --> 00:59:48.525
kind of
736
00:59:50.010 --> 00:59:53.850
answering questions based on memory is just not gonna work anymore.
737
00:59:54.250 --> 00:59:56.650
I'm very well connected to the education
738
00:59:57.370 --> 01:00:02.090
ecosystem here in in Silicon Valley, and I've seen this a lot in the last couple of years
739
01:00:02.945 --> 01:00:07.585
where teachers are looking for help too. How do I how do I teach Python?
740
01:00:08.065 --> 01:00:09.345
Like, how do I test?
741
01:00:09.665 --> 01:00:22.530
Like, because the the test is usually solve this problem, and you give it to the coding agent. And the second letter, you have the answer. So I think people need help, not just, learning how to use AI, but how to teach AI. So I'm really looking forward to seeing how that...
742
01:00:22.850 --> 01:00:28.850
And I know a lot of people are working on it. I'm not saying it's not solved, but it's it's it's, I'd like to see, people help with that.
743
01:00:29.615 --> 01:00:34.734
Yeah. And, generally, I'm really excited to see what some of the commercial labs,
744
01:00:34.895 --> 01:00:36.575
come up with. I know there's a lot of,
745
01:00:36.974 --> 01:00:38.655
people that's, kind of,
746
01:00:39.295 --> 01:00:41.615
came out of the labs now or starting a lot of new
747
01:00:43.140 --> 01:00:43.940
companies,
748
01:00:43.940 --> 01:00:44.500
like,
749
01:00:45.300 --> 01:00:47.940
you know, Ilia and his company, SSI,
750
01:00:47.940 --> 01:00:49.780
and Thinking Machines,
751
01:00:49.859 --> 01:00:50.580
and and
752
01:00:50.820 --> 01:00:57.460
Coral Automation, and other companies like that that are trying to solve the next generation of of LMs. And also
753
01:00:58.065 --> 01:01:01.265
very excited about image and video generation. I think,
754
01:01:02.224 --> 01:01:04.545
I've been tracking that space a little bit too. And,
755
01:01:05.905 --> 01:01:08.385
kind of the way I like to characterize it, I feel like it's,
756
01:01:08.945 --> 01:01:12.385
it's in a GPT two, GPT three moment of its evolution.
757
01:01:12.960 --> 01:01:16.640
It's very powerful. We use it all the time, but I think it's...
758
01:01:16.960 --> 01:01:20.080
The potential will be unlocked in maybe a couple years.
759
01:01:20.560 --> 01:01:23.360
We're all gonna see this amazing innovation
760
01:01:24.000 --> 01:01:24.480
and,
761
01:01:24.800 --> 01:01:28.560
much more, direct use cases that are that are more immediate in that sense.
762
01:01:29.214 --> 01:01:30.095
Absolutely.
763
01:01:30.095 --> 01:01:52.020
Well, thank you very much for taking the time today to join me and share the work that you've been doing and some of your insights on how to evaluate rag systems and get them running well production. It's definitely a very interesting and continually complex problem and one that keeps shifting. So I appreciate you taking the time to share your insights on that, and I hope you enjoy the rest of your day.
764
01:01:52.500 --> 01:01:53.619
Thanks, Tobias.
765
01:01:53.940 --> 01:01:55.540
Glad to be here. Thanks for having me.
766
01:02:01.425 --> 01:02:09.780
Thank you for listening. Don't forget to check out our other shows. The data engineering podcast covers the latest on modern data management, and podcast.init
767
01:02:09.780 --> 01:02:13.860
covers the Python language, its community, and the innovative ways it is being used.
768
01:02:14.100 --> 01:02:24.730
Visit the site to subscribe to the show, sign up for the mailing list, read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email hosts@aiengineeringpodcast.com
769
01:02:24.730 --> 01:02:25.450
with your story.