Nitay (00:06.51)
Emmanuel, great to have you today on the podcast here with us with with Costas. Thanks for joining. Why don't we start with give the audience a bit of your background and kind of some of the things you've worked on in the past and what you're focused on now?
Immanuel (00:18.945)
Yes, hello, Nitai and Kostas. thanks for the invitation. Super happy to be here. And yes, so my name is Immanuel Trummer. I am a professor at Cornell University, and I'm generally working in the research area of database systems and data analysis. I imagine most people here in the community would probably know me for work that in particular
Connects databases with novel technologies. In particular, I'm we are working a lot on connecting database systems with large language models. I've been working in that space since quite a while. I still remember when I was doing this work before ChatGPT came out, I always had this like five to ten minute introduction in my talks about what is a language model, why is it cool, what can we do with it, and so on.
And then over the last couple of years that essentially part of the talk has become shorter and shorter. And now I'm basically just assuming that everyone knows about a language model and it's really taking off in the database community. And there's lots of things that you can do with large language models. For instance, we are working on projects that use large language models for database tuning, for essentially
Writing a code for data processing, replacing some of the components in a traditional database management system, and also a work that essentially uses launch language models directly for data processing, which means that you can essentially expand the scope in data processing from mostly structured and relational data to many unstructured types of data as well.
if you do that, then of course things become quickly expensive because large language models. if you look at the processing overheads per amount of input data, for instance, then it's just many orders of magnitude more expensive compared to traditional relational data processing. So a lot of our work is focused on how we can scale up data processing with language models to large data sets and make it more efficient in terms of time and monetary execution fees.
Immanuel (02:33.399)
Also recently working on a start up company in that space which will focus on making data analysis with language models more efficient on large data sets.
Nitay (02:45.389)
Amazing. Sounds like a lot of very cool stuff. Why why don't we go back to the beginning a little bit? You said, you know, you guys were looking at incorporating language models before it became the obvious thing. I I remember those days of people talking about I can predict the next word and everybody around them being like, Why why is that useful? Well, what are you gonna do with that? and so back in those kind of early days, what is it that you saw about language models and and their interaction with data the with the database world and the potentials and the research you guys started on there?
Immanuel (03:12.889)
I mean I generally got got hooked to language models. I think like then playing with it a little bit and then trying it out is just an amazing an amazing feeling. I still remember how I started working. In particular, I think like starting from GPT three onwards it really became magical and it really I had a feeling that there's a just a tremendous amount of cool things that you can do with it.
So, I mean, some of the first work that I actually did in that space was focused on problems around database tuning. I mean, those database management systems, they just have hundreds to thousands of tuning parameters that you can use in order to influence the system behavior in all kinds of tasks. So for instance, like tuning how much buffer to allocate, tuning how many cores to spend in certain tasks, and so on and so forth.
And I mean the settings for those parameters can have just tremendous amount of impact on the execution performance for certain workloads. The default settings are generally suboptimal for any given workload. There was just a lot of interest in the database community about how to find good parameter settings to optimize a given performance objective, such as maximizing throughput or minimizing latency for.
A given workload. And this a really, really hard problem just already because there's just a lot of parameters and a lot of possible value combinations. So you're dealing with this huge search space. And traditionally there are no really good models that enable you to predict which settings work well for a given scenario. So you often have to essentially try out different parameter values and
The thing that I was working on in that space, which turned out very well actually, is that a human being, when faced with the same problem, wouldn't just try out random parameter settings and see how they work. But that person would hopefully first have a look at the database manual, at tutorials about database tuning, then and maybe discussions on forums like DBA Stack Exchange and so on and so forth.
Immanuel (05:28.341)
There's just a tremendous amount of useful information about database tuning contained in those text documents. So the idea was basically that now that large language models at that point, a couple of years ago, they just had essentially crossed a boundary where they got really good at understanding natural language text and extracting information from it. So what I was doing in that space was essentially to use those language models in order to extract useful information from database tuning from the entirety.
Of the web, basically. And then you can use that as a starting point for automated database tuning. So that was a project a couple of years ago. And nowadays there's actually lots of work that follows the same direction that uses large language models in order to find good parameter settings for database systems. It is extremely useful because you just narrow down this huge search space to something much more reasonable. And that is
One way in which you can use language models. After that, we have been working a lot on, for instance, new user interfaces for data analysis that are based on data analysis, also on approaches that use language models for code generation to replace certain parts of database management systems. But yes, the very first project that I worked on in the context of language models and database systems, that was really about like database tuning.
And it turned out that language models are very good at like essentially providing you with a shortcut to cut down this huge space of possible parameter configurations.
Nitay (07:07.383)
And it's interesting. So you want the approach of like because traditionally the approach I imagine others would have taken would be, okay, it's a combinatorial explosion problem. There's too many options, like you said. Let's do some clever AI search with some Monte Carlo, some whatever, like try a bunch of things and hope to get to come some good result. And you took it the next step and said, wait a minute, there's a whole bunch of kind of DVA lore, knowledge base, et cetera, that we can pull in.
And so then we're gonna build system that's gonna take that and optimize. It seems like there's a next wave, I imagine. I'm curious if like maybe this takes us to the future of then bringing in some like reinforcement learning and having like, why is it not just like, hey, here's my goal? I want this to run as fast as possible. Why do I why do I even need to care about the language model or the search or the this? Why is it not just like here you go, do it?
Immanuel (07:55.472)
That is a very good question. And in fact, before I was doing this work, the primary method for database tuning was indeed based on reinforcement learning. So the reward function that you're optimizing for, that is essentially the performance gains that you're getting compared to the default parameter configuration on a given workload. And that works also, but it just takes a very long time to converge because in order to
Get a reward value from this reward function, you essentially have to execute at least part of your representative workload with a given parameter configuration. So I should clarify that this first work that was doing in the space of language models and database systems, it actually also used reinforcement learning, but it used reinforcement learning within a much more narrow search space, basically, which enabled it to convert much faster.
So you essentially use the language borders in order to get a search space of reasonable parameter settings and then you use reinforcement learning within that space in order to find the actually best configurations.
Nitay (09:03.847)
Makes a lot of sense. very cool. And what was the most interesting or maybe surprising finding and what was kind of the biggest challenge of of doing this research? Getting to behave well.
Immanuel (09:16.495)
Well, I mean, I think that at when starting this, it actually seemed like a fairly crazy, maybe crazy is too strong, but it feel it felt like an approach that might not work. And so I think just like the general outcome that those language models have really become good enough at understanding text that this approach is valid and is working quite well. I think that was the the primary outcome.
One thing that was interesting to see is that when doing database tuning, I mean, you might sometimes think that just reading the database manual gives you pretty much all the information that you need in order to tune well. And that turned out not to be the case. I was surprised to see that in some cases, the best configuration that the system finds is based on some little snippets of information.
That it extracts somewhere from a forum like Stack Overflow or DBA Stack Exchange, for instance, some discussion about some very specific workload and some very specific scenario. And there's this tuning option that generally might not even be a good idea, but every once in a while it actually makes a huge difference. And the way in which the system often beats those more traditional database tuning tools is often when it finds one of those pieces of information somewhere on the web.
that it can use in order to get better for a very specific work.
Kostas (10:47.765)
okay, that's very interesting. And I want to I I want to ask a question about something that might be like a little bit of like a step ahead of what you are describing, Emmanuel. So and let me do like a little bit of introduction here for my for our for our audience. So traditionally database systems have been like one of like let's say the most complex systems that like we
are engineering, right? Like databases, compilers, and probably also like operating systems. anyone who has gone through like a CS degree, like they know that these are, you know, they're really, really hard like classes to go and take and probably you won't even build the whole thing like at the end, right? Because of like the complexity there. and and I think a lot of like that complexity drives, let's say, some
in the industry at least like some decisions around how the systems are built, right? So obviously we can't write a specialized database for each workload that we have out there. Like it's prohibitively expensive, like or it was prohibited expensive just to start doing like a little bit of introduction here, like to my question. so we have like we needed to have like these let's say categories. Okay, we have transactional, we have all up, let's say we have time series,
Maybe streaming. And then within these, we have a lot of knobs that we use to go and customize and update the system as the workloads change. Now, LLMs, obviously, you said one of the things that they can do is like help figure out these knobs and come up like with good configurations. But another
Interesting side effect that all of them have is that now code is pretty cheap to generate, right? And pretty complex also code. And I've seen a few papers coming up. I think in the LDB there are like a I think like at least two of them where the approach is more of like why not just go and build from scratch a dedicated database specifically for like the workload that we have, right? Which obviously a few years ago that would be a crazy, crazy idea.
Kostas (13:10.837)
tell us a little bit about that and how does this relate with what you are talking about, the optimization of like the configuration of the systems and in a way where do you see things going? Because things like with LLMs like move like extremely, extremely fast. and I think that's like part of like the the what is like really fascinating with them. Like so yeah, tell us tell us a little bit about like about this.
Immanuel (13:39.686)
So that's an absolutely a great point. And I think in certain ways this is the extended version of what I was describing. I mean, if we want to adapt a database systems to get really good at a specific workload or in a specific scenario, we can only do so much by changing the tuning parameter configurations because we already need just the right tuning parameter to do what we wanna do. So in order to have the ultimate freedom.
In order to customize a database management system, in order to make it efficient for a specific scenario, in the end, what you have to do is basically to generate code that does the data processing for you. And we do actually have two papers here at this conference about that specific topic using launch language models in order to generate code for data processing. So that's a topic that we have been working at, I would say basically since it became
Technically somewhat feasible. I did some work a while back in twenty two, twenty-three, about a system called Codex DB, which uses language models in order to generate code for data processing from scratch for specific queries and a follow-up work on a system called GenesisDB, which implements custom operator implementations for a database management system for those traditional relational operators like joints, filters, and so on and so forth.
And that worked somewhat well. I mean like for this early work, we actually typically generated code in the Python programming language. So here the focus was a little bit less on efficiency, as you can infer from the language, and more about like making it generally possible in principle. So there it was actually very challenging to generate code.
That even works correctly. And for many queries back at that time, it was essentially not possible. Even if you gave it multiple retries and if you helped it quite a bit along the way, it was really, really challenging to generate a correct code for complex queries, even in a relatively concise high-level language such as Python. Now, this has recently changed actually. So the two papers that we are presenting here at this conference, they are actually
Immanuel (16:00.01)
using like the latest generation of launch language models in particular from anthropic and open AI, together with an agentic framework in order to generate code for data processing. And now we are generating C code. Typically, even for complex queries, you get the right code and you actually get code that is very, very efficient. And that was a nice outcome. It turns out that if we give those agents a lot of freedom about like what kind of code they're writing
They're often able to write like code that outperforms traditional database management systems. So we actually we have like two flavors of this here at the conference. One project called GenDB. It essentially uses an agentic framework based on large language models in order to generate code that processes specific queries on a specific hardware on specific data.
So this project is really about specializing the generated code as much as possible in order to make it perform really, really well for specific queries. And you actually do get some significant speed ups of a factor of a multiple compared to highly optimized traditional database management systems. And it's actually really amazing like the kind of optimizations that the system considers. We check the code that it is generating. It is a, I should probably say it's an iterative process.
So the system can essentially generate code, then benchmark that code, analyze the code, and then generate better code and try to improve over the original version. And that actually works quite well in practice. And it is doing some very fine brain optimization, for instance, taking into account the sizes of the cache hierarchy on your CPU in relation to the data.
and the query properties in order to synthesize like highly specialized data structures that work best for specific queries and for specific types of hardware. And yes, so this is things that you could not achieve anymore just by changing the settings for tuning parameters in an existing system, because this is really changing like fundamentally how the system represents data and how it processes it.
Immanuel (18:21.615)
So that is one of our systems, GenDB. We have another system called JPreg, which does a little bit of a different approach. So I mean, the former system generates highly specialized code for processing specific queries on specific hardware. The latter system tries to essentially generate code that will be reused across different queries. So you're trying to essentially reply some part of the database management system that accesses database.
And again, I mean it generates very efficient code. I mean the advantage if you generate code that is reused across different queries is that for instance, you can have somebody who verifies that code once and then you can use it in the system going forward. If you generate code for specific queries, then I mean if you say I don't really trust the language model, then you would essentially have to do something for each of those queries.
or query templates in order to verify the generated code. Having said that, we have lots of mechanisms that essentially increase the chances that the code you are generating using those agents is is correct. For instance, we have agents that are tasked with generating automatically test cases that are then used in order to evaluate and test their generated code. And if there's some some if there's some this
difference to the ground truths, a result for a a specific query on a data sample, and we would regenerate the code basically. But definitely I would say that we have been working on that topic since a while, and the focus has definitely changed from just making, just trying to get correct code in a large percentage of cases to really trying to generate code that outperforms existing methods in terms of a data processing time.
Kostas (19:52.574)
Mm-hmm.
Immanuel (20:15.26)
for instance.
Kostas (20:16.607)
Yeah. okay, I have like a a question for you. what's I'm I'm personally like curious about it, but I I I think like it will be interesting also like for the audience. So traditionally and correct me if I'm wrong by the way, traditionally in the database systems there was always like these let's say like two different approaches on how let's say we take like
A query in SQL, right? Like we create like our logical plans, then we get into like the physical plans. And then we either let's say how we how do we actually like implement and execute these physical plans, right? Like there is one approach that says, okay, let's go and generate like code that can be compiled and like we can run it, right? And the other one is we have like these operators, we combine the operators together and we go like and and execute. The reason I
I'm asking that is because my understanding is that generating code for our queries always like felt that can improve a lot like the performance, because obviously like we can go and compile code that you know like the compiler can do also like really great like optimizations based on the hardware you're running, blah blah blah, like all that stuff. But my feeling was also that it never became, let's say.
As adopted as someone would imagine, right? Based on like what it promises in terms of the performance. Part of my understanding, at least, the reason behind that, two things that I I was kind of like identifying. One is the complexity, right? It's like much harder like to debug a system like that, like from the engineers who are building these things. and the other one was like.
query latency, right? Like when you have like to compile, it is like a process that takes time. So if you are doing like online analytics, like for example, okay, you can't add like that much like latency there. So let's say if if it was like a batch process, yeah, like that can potentially like work. Like it's worth taking like the penalty there. But if it is like let's say somewhere where like a human is waiting like to answer like a query, then that starts becoming like a problem. What
Kostas (22:40.761)
LLMs do right that couldn't be done using let's say the previous approaches of like generating code for like the the plans that our like databases were generating. And that's one thing. And the other thing is what about the latency again, right? Like LLMs are slow. Like if it is hard like to compile something, like in terms of how fast you can compile something using let's say like
modern compilers, I assume that waiting for an a jetting system to come up like with the code and make sure that the code is correct and all these things and like goes around is like even like slower, right? So how how what are the differences there and like also how do we fix some of these issues that traditionally we had like with generating like and compiling code for database systems.
Immanuel (23:37.102)
Okay, great questions. Okay, so I mean in general, code compilation has been around since a while in database management systems in general. The short answer about like what language models are doing differently compared to traditional code generation is that you suddenly have a lot of freedom in terms of the code you are generating. Previously you would generate the code based on code templates, basically.
That restrict the like what kind of code it is, which data structures it is using in order to represent results, for instance. So now if you would generate code from scratch using a language model, there are many more optimization opportunities that you can exploit when doing so. About your comment in terms of latency of code compilation and even more so latency of using agentic frameworks in order to generate code.
That is absolutely correct and that restricts to some extent in for which queries you want to use those approaches. If you have short-running ad-hook queries, then you definitely don't want to generate fresh code using multi-agent systems because it would just take way too long generating the code in comparison to the query execution time. But on the other side, if you have queries that have to be executed repeatedly.
Maybe on changing data. Then you have to do this only once and then you can reuse those queries going forward. Going beyond single specific queries, when looking at the workloads from industry, it turns out that actually a large percentage of the queries that you're seeing there, they are not ad hoc queries, but they comply with query templates. And those query templates are known beforehand.
And if you know those query templates beforehand, then you can generate code that applies to a specific query template, which is also something that we have been doing. In those cases, again, you can afford to spend more time in generating the code because the result will be reused across many different instances of the same query template. If you have a lot of ad hoc queries in your specific workload, then what would apply most are approaches.
Immanuel (25:51.972)
which generate code that substitutes some components in the database system and don't have to be regenerated for each new query, but you generate them once for a specific hardware, for instance, or for a specific database, and then you reuse them across different different queries.
Kostas (26:16.309)
Okay, that's that's great. now we covered already some of the parts of like a DB system. There is another one that is related, and I think it's also like very interesting with the LLMs, and I'd love to hear from you the effect that LLMs have on that, and that's the optimizer. and the reason I find it interesting is because writing like
query optimizers, I don't know, I always felt to me like some kind of
science but also art. You have like these people that they come up like with these like these rules that because at the end of the day like optimizers are like a bunch of rules that we codify there and we try like to figure out the right order like to execute them and apply them or have them let's say find all the edge cases and all these things and it's all like one of these things that
Yes, you can reason about it and like s have a systemic approach, but at the end a human will figure out that like, like when we have this kind of distribution of data there and have it might make sense like to put our cost to look like this and like put it there and like all that stuff. so how how do you see the optimizers actually being affected by the use of LLMs?
Immanuel (27:53.102)
Alright, that's a great question. And I mean there are a couple of ways in which LLMs can be used in order to improve query optimization as well. There has been quite a bit of work in our community that tries to leverage LLMs in one way or another one for query optimization. So I mean, similar for code generation, also for query optimization, you might not want to use the LLM directly on the critical path for ad hoc.
Short-running queries just because it takes a little while to use the LLM to, for instance, rewrite the query, for instance. there's also some correctness issues. So, what you can do, for instance, we have been doing some work a while back on using LLMs in order to predict the data properties from the schema. for instance, I mean one.
Problem in general in query optimization is that it is often very hard to estimate the execution costs of plan candidates. And if those execution cost estimates are incorrect, then the query optimizer will select a suboptimal plan. Now, in practice, the primary reason why those cost estimates for candidate plans are incorrect is that the query optimizer has troubles estimating.
The sizes of intermediate results that get generated during query processing. And the primary reasons why it is hard to estimate the sizes of those intermediate results is that first of all, the data might have a skewed distribution. If you use a very simple size estimation model, you might assume that if there are 10 different values in a column and I have an equality predicate selecting one of them.
Then 10% of the data will satisfy that equality predicate. But in reality, it might be that you have certain values that are much more frequent than others in the data, and that can throw off those estimation models. The somewhat harder problem to fix in that context is that different predicates in the same query might have some correlation between them. So if a row in your data satisfies one of those conditions,
Immanuel (30:12.109)
It is much more likely or much less likely to satisfy the second condition. Classical query optimizers typically assume by default that different query predicates are independent, statistically independent from each other. So they will estimate a probability to satisfy a combination of predicates that is far off the true probability, and then they get those size estimates wrong.
Now, the way language models can be helpful in that context is that certain properties of the data, they can be predicted by people actually just from the semantics of the data. If you have the schema of the database and perhaps some accompanying documentation that tells you what is the semantics of the different column names in your data, you might be able to infer some properties of the data without even doing any in-depth analysis.
For instance, if the economist called ID column, then most likely you will find unique values in that column. Or if you have certain things, for instance, you have the column about the car maker and about the car model, you will probably have a correlation between those two. And a human being would know that just from looking at the semantics of the data without even having to do some expensive data.
Profiling. And language models can do that to some extent as well. So we have shown essentially that if you apply language models in order to understand the semantics of the data and predict some properties of the data, it often turns out to be correct. And those are this is information that a query optimizer can directly use in order to in order to counter some of those common problems, such as failing to
Detect that certain predicates are correlated, failing to detect the skew in certain columns. So, in that context, language models can be helpful. And that can be applied even before processing specific queries in a pre-processing step, basically. Then you can feed that to the query optimizer to get more reliable optimization. So that is one way in which you can.
Immanuel (32:32.289)
apply language model in the context of query optimization, but people have been trying out many different ways of doing that, and that's certainly an interesting space.
Nitay (32:44.363)
I'm curious, Emmanuel, through all these kind of research and explorations you guys have done, have you had any findings of things? You know, like I I always think back to kind of the the the AlphaGo. It was like move whatever, 30, 24, I forget the number, but you know, where it did this like surprising move that everybody what thought was, you know, the most terrible move ever and you should never do it. And we've been taught for, you know, decades, centuries probably more that's terrible. And it turns out actually in very certain cases, it's the exact right move.
So I'm curious through all the like kind of explorations you guys have done that you just discussed here, what has been some of the findings or things or tunings that you later go back in the system and you say, huh, it chose to do this or that. Like why why did it choose this? And then you investigate and what's been the learnings there.
Immanuel (33:33.027)
That's a very good question. So I think that I mean, in general, I mean, in particular when looking at those projects where the language model is used to generate code, for instance, I think we had quite a few aha moments when looking at the code that is generating. And I think in in in many cases, those are things that we wouldn't have thought about of doing in order to make query processing more efficient. So I think that I mean, probably most in that project that was actually
surprising in some cases how it was able to for instance like to use unusual data structures in order to represent data and get some additional performance gains that we might have overlooked. And I think in in general, I mean like when referring to Alpha Go, I mean I I think that I mean there I mean
The kind of performance gains that we are generally able to get out of those launch language models when applying them in different scenarios has actually been very positive. I mean, I would not always I mean, in some cases, I think it's like things that a good database administrator would be able to do. But I mean, that's something that requires a lot of database expertise and lay users who
want to use database management systems but are not necessarily experts in terms of the internals about how they work, they would probably not find that. So I mean, even for us, in some cases a surprising surprising moves, so to speak. But certainly for people who are not database experts explicitly, there's a lot of useful hints that they can get from those models for database tuning.
Nitay (35:17.165)
Very interesting. what's been by the way, we we didn't actually go into it, but tell us about some of your kind of you know, some of the research that you're most excited about or the most kind of landmark, what were the actual outcomes? What was the the delta in performance that you reached with the system or the amount of I don't know, less code that that the human has to write? Like how however you were measuring kind of the the value of it. Tell us about some of the results.
Immanuel (35:38.98)
Right. Okay, right. Good question. I mean, what we have been discussing so far is actually mostly about how to make classical database management systems more efficient, systems that process relational data, tabular data, using something like SQL queries. I think one of the most significant impacts of language models in the context of data management.
Is actually really that they dramatically expand the scope of what we can do in data processing in general. I mean, traditionally with SQL, you're essentially processing fairly simple operators on tabular data. So you have for instance, an operator that would enable you to count the number of product reviews in your database that contain a certain keyword, but you couldn't process something.
That counts the number of product reviews in your database that are generally positive about the product. So you cannot process operators that require semantic understanding of the data. And then on the other side, it is traditionally restricted to processing data tables, tables where the cells contain essentially the SQL data types, text, the numbers, and so on.
But you couldn't process something like images or sound files, videos, and so on and so forth. And one way in which large language models really advance data processing is that they expand the scope of what we can do with the data and the types of data that we can process. So one domain that I'm particularly excited about in the domain of database systems, and something where
We have been doing quite a lot of work here in the past couple of years. That is what is often called semantic SQL processing. So you essentially have SQL, which is expanded with AI operators that are evaluated by large language models in the background. So you essentially would be able now to write a query that counts the number of product reviews in your database that are positive towards the product.
Immanuel (37:56.806)
You would also be able to process tables where the cells not only contain text and numbers, but also images, sound files, videos, and so on and so forth. So this is something that has become reasonably possible only since quite recently. But actually people have been trying to do something like that since quite a long time. If you go back something around 15 years ago.
In the database community, you actually had a whole line of research on so-called crowdsourced database management systems. And those are database management systems, which essentially use human crowd workers taken from platforms like Amazon Mechanical Turk, for instance, in order to evaluate some of the query operators in the background. And then you can do things like, for instance, filtering images based on what is shown in the image, or filtering
Sound files based on what people can hear in those sound files. So all of this back then was essentially only possible if you had some humans in the loop, but it is not very scalable and it is very costly to do. So since we have those large language models of let's say GPT three level language models that are really good at zero shot learning so they can process a new task.
Without task specific training or fine-tuning. and since we have those language models, it is suddenly possible to build systems which do something like that without any humans in the loop anymore. So there have been quite a few systems like that released by actually various major database companies, Google, Databricks, Snowflake, MotherDuck.
And so on and so forth. Basically, everyone is currently switching to that. And those systems they really expand a lot what you can do with a query processing. They expand in terms of the data that you can process from just tables to images, sound, videos, and so on and so forth. And they are expanding how you can process that data from relational operators.
Immanuel (40:17.679)
To operators that require some semantic understanding of the data. So that is a very cool topic that I'm excited about. It is very challenging to do, mainly because applying large language models directly to data leads to some significant costs. If you look at the pricing pages of large language model providers like Anthropic or OpenEye and so forth, you will see that the
Per input byte data processing overheads are by many orders of magnitude more expensive compared to traditional relational data processing. So I think in general, the big challenge here is just how to make this more efficient and scale this up to large data sets. So we have multiple projects in which we are working on that. I just actually gave a keynote about it here at a workshop at BLDB. And that is, I think, one of the
most significant impact of large language models that it can really change what we are able to do with the data.
Nitay (41:25.431)
That that sounds really fascinating. So I'll give give the audience maybe a bit of a a teaser towards the the lecture you gave at at BLDB. What what does it take to build a system that can have those operators, like you said, these kind of external AI, async, UDFs, whatever whatever form like you kind of might put it in?
but do it in a systematic way that is, you know, to the points that we discussed before, like kind of meets the latency and the system requirements and the kind of user interactiveness while also, you know, getting a handle on the cost and kind of the the broader kind of system approaches. So how how do you do that? What's the approaches you take?
Immanuel (42:02.854)
Okay, great questions. So, I mean, if you have AI operators in your queries at that point, essentially the entire bottleneck and query processing performance shifts to how you evaluate those AI operators efficiently. So you can essentially do two things, and that's also what I focused on in my keynote. You can do two things. The first one being you can reduce the size of the models that you're using. The second thing, what you can do.
is to reduce the number of tokens that you consume and generate. So those are the things that you can do. And in the keynote, I presented essentially one representative project for each of those approaches. For instance, if you look at the language models even like from the same provider and in the same category, then you have like different model sizes that differ by at least a factor of a hundred in terms of the processing costs per amount of input data.
So you really want to choose like the smallest model you can, but choosing the smallest model always is probably a bad idea because you might not get the output quality that you need anymore. So you basically want to choose the cheapest model, which still satisfies your personal requirements on the quality of the output. So the system that was presenting there, Spare LLM, it's a recent paper from us, it essentially helps to systematically
Select the cheapest model that satisfies user-defined quality constraints. So it is doing that by using profiling. So in those semantic SQL scenarios, you always have to perform the same task, whether that is like a filter operation or a semantic map or whatever, you have to perform the same task on a lot of data basically. So you can afford to
Do a little bit of profiling on some sample of the data in order to get an impression for how well different candidate models work. So you do the profiling, you compare the output of candidate models to the reference model, and then based on what you see, you select the cheapest candidate models that is still sufficiently equivalent to the reference model. There's a couple of interesting questions there. For instance, how much profiling should you do?
Immanuel (44:25.209)
Which candidate models should you profile? All of this costs money, so you want to be smart about like restricting profiling really to finding out the things that can help you ultimately save money when processing the rest of the data. The thing that I the project that I presented when it comes to saving tokens, that is a project that proposes a new implementation for a classical.
For a semantic version of a classical relational operator, the semantic join operator. And what I'm doing there basically is to propose an like to get across like to transfer some ideas from classical join algorithms. So I mean typically those database management systems, they always have multiple different implementations for the same relational operator, like a join. And I'm essentially bringing some of those ideas.
To the domain of semantic query processing and adapting them. So in classical database management systems, there's something called a blockness that loops join, which essentially loads batches of items into main memory in order to become more efficient. And I'm essentially doing some version of that for the language model, where it turns out that if we load batches of the items for which we want to perform the semantic join into the prompt at the same time.
Then we can get significantly more efficient. And maybe I should actually take a step back and explain what is a semantic join in general. So the classical database join that is generally have two input tables, and I want to find all the pairs of rows, one from each of those tables, that satisfy some joint condition together. The semantic version of that is basically the same, with the only difference that the joint condition now is formulated in natural language.
And has to be evaluated via a language model. So many of the existing systems they actually implement this join in a fairly like straightforward manner. They essentially take each possible pair of rows from the two input tables and then ask the language model, does this satisfy the join condition? Yes or no. But when doing that, you essentially have a number of language model invocations that is quadratic in the size of the input data.
Immanuel (46:48.303)
So instead of that, it is much more efficient if you take batches of data from both input tables, put that into the context window of the language model, and ask it to retrieve all the matching pairs that satisfy the joint condition together. You have to be smart about like how you divide the given context window, which limits the total number of input and output tokens that a language model can process across
Tokens used to represent a batch of input rows from the first table, tokens used to represent a batch of input rows from the second table, and tokens reserved for writing the output to the join into the context window. And the paper essentially proposes a couple of formulas that you can use in order to find optimal solutions. That also works very well in practice. So those are two representative projects which essentially focus on each of the two possibilities that you have in this context, either
using smaller models or reducing the amount of data.
Kostas (47:50.544)
Emmanuel, I have a question on that. this is an interesting topic that I guess for some reasons like I've been following like experimenting myself for like for a while. So and I'll take I I think like the the the semantic join is a very good example to to use it like as like the substrate to talk about. So if we take like a classical join, right, like at the end of the day, what we are doing is that
We put the machine to go and do let's say like lexical matching. Like if I'm joining like let's say on the user IDs, I need to find like an exact match between the user IDs between these two like columns, right? in the semi but in a way we push the semantics to the human person, right? Like the human has to know like
what the meaning of these columns are and why they should be joined, right? and either this is like encoded in the schema or it might not even be encoded in the schema but just leaves like in the head of like the the person who writes like the the query, right? Now with LLMs we can bypass that, right? Like in a way we can go and say, hey, like I want to go and find, let's say, all the let's say products
That these people of like this cohort of age or whatever, like they like and be like as generic as that, right? And then we expect that okay, this semantic join is actually going to find like somehow that the descriptions of the products and by filtering also like the users and blah blah blah, like all these things, and like we'll get get you the data back. Now instead of doing that.
Kostas (49:47.23)
Would it be meaningful to say, hey, like I understand the intent here, right, by using LLMs? Use LLMs to extract structure from the data set. So change the schema to represent, let's say, what the data should be joined on, and then go and rely on classical joins to do to do the the join at the end, right?
And there are like two reasons why I'm I I'm saying that, because okay, one is let's say probably like the performance, but there is another thing that I think is like really really important when we are putting LLMs into like SQL, and that's how easy it is to verify what happened, right? And if something doesn't feel right, how I can go and trace that something is wrong.
And actually there is a third one, which is kind of cussing in a way, because if the extraction of the information is like good enough, we don't need like to repeat that. We can just keep doing like natural joints, right? Like and we'll be we'll be fine. so yeah, anyway, that's what came in my mind while you're like describing that, and I think like would be would be interesting to hear like your thoughts on this.
Immanuel (51:12.911)
Those are some great points, and I absolutely agree that if you have a case where, for instance, you could actually revite the semantic joint condition into a traditional formula that can be evaluated automatically on the existing columns, you absolutely want to do that. You might also be able to extract things that can be useful for multiple further queries, for instance, and if that is possible, you should absolutely do that.
you can also sometimes work with embedding vectors, for instance, in certain cases. That can also work for very well for certain scenarios. In particular, if you have something that resembles a traditional quality joint, basically, where you want to find, for instance, documents or even images that are closely related. let me maybe give you some concrete example for where we might at least in the first in the first run need some semantic a joint operator.
So for instance, I was recently talking with some furniture companies, and I mean one of the things that I was discussing with them was actually a system that does the following. So imagine you as a user, you are taking a picture of your unfurnished living room, and then you have a database with pictures of furniture items and some other information such as maybe the price, the size, and so on and so forth.
Now, maybe you want to find a combination of furniture items that together would match well into your living room. Maybe things that are currently not in your living room, so you want to buy something else. And you want to find combinations of furniture items that match well together in terms of the colors, in terms of the materials, in terms of the style that might also match well together with your with your living room picture. So all of this essentially corresponds to one or multiple semantic.
join operations. Of course, you could extract certain types of information from the furniture items. In fact, if you go to a furniture database, then you will already have certain types of data about the furniture. For instance, you would typically have one color associated with the furniture item. But that is very coarse grained because they're different shades of the same color. And typically one piece of furniture would not only have one color but actually multiple different colors. And in order to be sure that you are
Immanuel (53:35.087)
Furniture items match together and that they match your living room, you would really have to actually see it. And that is what the language model can do in the semantic joint. Of course, if you can translate some of those conditions into into predicates that can be evaluated using traditional means, right? Then you should absolutely do that. And we are currently looking, for instance, at rewriting strategies where in cases where users tr are trying.
To use some semantic joint for things where don't need it, where we actually automatically rewrite that into something that can be evaluated much more efficiently. And then, of course, also for embedding vectors, they can be very useful in order to find rated items. You cannot always rely on them because it always depends on what kind of joint condition you have, right? So there are lots of possibilities in order to make this more efficient, and we are working on multiple different implementations.
But this is just to say that there are at least some scenarios where you couldn't get everything you need just from translating this semantic join predicate into a traditional joint predicate. But if you can do it, I absolutely agree with you, you should do it for performance reasons.
Kostas (54:51.047)
And what about being able to verify and debug like the the queries at the end of the day, right? Like I would say like things I I have a feeling that like things that have been like keywords like used a lot like in in research more than in industry actually.
Like provenance, for example, right? Or like doing row-based lineage instead of column-based lineage. Like things that let's say they were always in the mind of like researchers or like people were trying to like to do, but there wasn't enough need in the industry, let's say, or let's say the effort to implement these things wasn't justified for let's say the what we could get out of it.
But I have a feeling that like with LLMs and their nature, we might have to be pushed into like figure out new ways and new tools that have to be part of like our like database management systems to actually go and verify and debug let's say like what whatever analysis we are doing. Especially when it comes like to semantics, because at the end of the day, and I think
If you go like to in a you know like in a in a company out there like doing just boring BI stuff, right? Like still like any mature company will probably have like multiple different definitions of what like revenue is. And the reality is that they are all correct in the right context. That's why they have them. It's not that there's like one way that you can like calculate like revenue, right? And many problems that the industry has, like the market has out there.
Is okay, now a new person is asking a question about revenue, they live in their own org and they don't understand that there might be many different other versions of there, and they just have like a hard time like figuring out which one like to use, or even like use one and it comes out that it's like wrong, right? and someone has to go and debug that and figure out why and understand like all these things. And it's really hard. It's really hard. Like it's it's it's not easy.
Kostas (57:14.493)
And that's without putting anything non deterministic that LLMs can bring, right? so what are your thoughts on that? Like what do we need and also what we can reuse and what we can learn? Because I think like I don't know, database systems have been around I don't know, like since computers were a thing, like maybe even before that, because we always needed like to deal with like data, right? Like one way or another.
And I feel like we keep relearning things that we've learned in a different iteration of the industry out there. So I'm sure there's a lot of things that we can learn today in using LLMs and AI by looking into how like the database systems like deal with problems like in the past. so yeah, like love to hear your thoughts on that.
Immanuel (58:08.451)
Yes, correctness is of course a significant problem, especially when bringing large language models into the mix. I mean, in traditional database management systems, there was some expectation that there's one correct ground truth result, and for a reasonable database management system you should be able to get that ground truth result. there's there has been some work, for instance, on like partially
Giving up those correctness guarantees in favor of better performance. There has been all this work on approximate query processing in general, where you are not processing the entire database, but you're only processing a sample of the database in order to get some approximate results, for instance, some confidence bounds for certain query aggregates. I think it hasn't taken off that well.
Overall in in industry it it seems that the users do have a strong preference for actually getting precise results. But having said that now that we have those language models around and that they are actually applied directly on the data, things are becoming more expensive by many orders of magnitude. I mean, there are like rumors at least of like queries where people have paid like
hundred thousand dollars for one specific query even and so it's really like a very different it's very a very different budget. So it feels that in this context in general people might be more amenable again to methods like approximate query processing because the the trade offs have changed very significantly so as soon as you are
Processing unstructured data like images, for instance, it is very hard to get any formal guarantees on a ground trace. I mean, for some images, it would be even hard for human beings to conclusively say what you're seeing on those pictures. So you can probably not expect to get a formal proof at the end of query processing that the result is precisely correct. I mean.
Immanuel (01:00:25.593)
What is typically done and something that we and others are trying to do is to provide some statistical guarantees that what you're getting is at least equivalent to the best thing you could get using large language models. So some guarantee of the type that if you had used the latest GPT model or the latest anthropic model, and you had put in like
Each data item separately, like you have minimize the amount of data in the prompt, which often improves performance, then you would get a result that is very similar to what you're getting now, even though we have decreased processing overheads compared to that naive approach by a factor of a thousand, for instance. So guarantees that essentially bound how much you're losing compared to the most high quality and
By far the most expensive way of processing things. That is something that we believe can be useful for users. But as long as we are talking about structured data processing, I mean you will not have any formal guarantees. And I mean you that's basically the best that it seems that you can do. Now on the other side, when talking about traditional relational data processing, where you can
Also bring language models into the mix. I just actually had one conversation with an industry representative here at the conference, and he was actually very concerned about like just having agents working on the data that might do things to the data that you don't actually want to happen. And I mean, similar things, I mean, like whether agents are writing SQL queries or whether they're writing codes directly.
It is essentially hard to guarantee that what happens is exactly what you need to happen. So one idea in general in that space, that's something that like some people are currently looking at, including us, is like, I mean, if you can at least within certain boundaries provide some formal proofs to users that what's happening is within certain bounds, what you want to happen, basically. That would make them feel much better. It's
Immanuel (01:02:47.321)
Hard to achieve, but maybe not impossible. But that seems to be one of the next steps on that path. But that is only for structured data processing where you do have ground truths. And at least in principle, you should be able to create a formal proof which tells you that what you wanted to happen with the data has actually happened. But I mean, for unstructured data, I guess we just have to accept whether we apply human beings actually or whether we apply launched language models.
You will not get any formal guarantees.
Nitay (01:03:21.169)
I wanna perhaps close this out because we're we're actually running short on time here. but this one's really fascinating. I wanna tie together a few of the things you said here, because you mentioned and some of the research that you guys were doing, the user has to come and produce, provide some kind of user-defined e-value DF, if I understood it correctly, right? Where you were using that to then kind of run an an internal benchmark to decide you can where you can use a a smaller model. And then here you're kind of tying that, you're you're you're kind of also mentioning you know, kind of
of the the non-determinism of the semantic joints and these other semantic operators and how do we help users get the best results even though they're doing these non-deterministic functions ultimately. And so what's been perhaps in order to kind of close this out what's been some of the best practices you've seen in people using utilizing these systems and writing these sorts of evaluation functions and how to best you know kind of integrate that with the database.
Immanuel (01:04:19.693)
Right. So you are basically asking about like how people get optimal trade-offs between processing costs and result quality for those semantic operators. Right. Okay, great question. so there's quite a few different methods. It depends on the type of operator that we are investigating and what kind of function what what kind of data we are processing.
Nitay (01:04:31.754)
Yeah.
Immanuel (01:04:49.567)
so I mean, in principle, you can again use those two methods. You can either try to go to significantly smaller models, and there's a couple of companies that are doing versions of that. One approach, for instance, is to essentially calculate embedding vectors on the data once and then train fairly small models that are trying to do things like classification or filtering.
Based on those embedding vectors alone. And when you're doing that, you want to have some kind of sense for how you're doing compared to the pseudo-ground truth, which you are getting by applying the largest language model on the data directly. And then of course you can do things like reducing the number of rows, the amount of data in general that you are looking at. So here again.
Come those approximate query processing ideas where you are trying to get some kind of confidence bounds or the number of items that satisfy certain conditions, even if you don't get the the precise number. there are other things where you can, for instance, change the prompt templates in order to get different trade-offs between processing costs and result quality.
For instance, we have actually experimented with that. We have a recent paper in which, among other things, we are pointing at cost quality trade-offs that you can achieve by changing the prompt templates and semantic filter operations. If you use fairly long prompts, where you have a couple of examples of how the filter should work, for instance, then you're increasing your costs, but at the same time you're improving the result quality.
If you use much more concise prompts that just contain one short instruction, check whether that applies, give me a yes or no answer. And then you can reduce the the costs because you are reducing the number of tokens, but on the other side, the quality that you're getting is potentially not satisfactory depending on the specifics. So there's lots and lots of tuning knobs that you can change in order to get different costs between
Immanuel (01:07:07.905)
different trade-offs between cost and result quality and then you can do some kind of profiling in order to get a sense for how far you are from the actual pseudo ground truths that you would get when applying the most expensive model to all the data items in the like with the most detailed prompts that you that you have so
Nitay (01:07:35.329)
Right, good. Thank you for some great advice and great tuning. okay, so with that, we'll unfortunately have to stop it there, but I think we we've all seen that there's clearly lots more we could dive into. So we'd be happy to have you on again. definitely we'll keep track of some of the the great research that you mentioned. Sounds like some very, very innovative and cool projects. And yeah, this has been an absolute pleasure.
Immanuel (01:07:58.723)
Awesome. Thanks so much for having me.
Nitay (01:08:01.367)
Thank you.
Emmanuel, great to have you today on the podcast here with us with with Costas. Thanks for joining. Why don't we start with give the audience a bit of your background and kind of some of the things you've worked on in the past and what you're focused on now?
Immanuel (00:18.945)
Yes, hello, Nitai and Kostas. thanks for the invitation. Super happy to be here. And yes, so my name is Immanuel Trummer. I am a professor at Cornell University, and I'm generally working in the research area of database systems and data analysis. I imagine most people here in the community would probably know me for work that in particular
Connects databases with novel technologies. In particular, I'm we are working a lot on connecting database systems with large language models. I've been working in that space since quite a while. I still remember when I was doing this work before ChatGPT came out, I always had this like five to ten minute introduction in my talks about what is a language model, why is it cool, what can we do with it, and so on.
And then over the last couple of years that essentially part of the talk has become shorter and shorter. And now I'm basically just assuming that everyone knows about a language model and it's really taking off in the database community. And there's lots of things that you can do with large language models. For instance, we are working on projects that use large language models for database tuning, for essentially
Writing a code for data processing, replacing some of the components in a traditional database management system, and also a work that essentially uses launch language models directly for data processing, which means that you can essentially expand the scope in data processing from mostly structured and relational data to many unstructured types of data as well.
if you do that, then of course things become quickly expensive because large language models. if you look at the processing overheads per amount of input data, for instance, then it's just many orders of magnitude more expensive compared to traditional relational data processing. So a lot of our work is focused on how we can scale up data processing with language models to large data sets and make it more efficient in terms of time and monetary execution fees.
Immanuel (02:33.399)
Also recently working on a start up company in that space which will focus on making data analysis with language models more efficient on large data sets.
Nitay (02:45.389)
Amazing. Sounds like a lot of very cool stuff. Why why don't we go back to the beginning a little bit? You said, you know, you guys were looking at incorporating language models before it became the obvious thing. I I remember those days of people talking about I can predict the next word and everybody around them being like, Why why is that useful? Well, what are you gonna do with that? and so back in those kind of early days, what is it that you saw about language models and and their interaction with data the with the database world and the potentials and the research you guys started on there?
Immanuel (03:12.889)
I mean I generally got got hooked to language models. I think like then playing with it a little bit and then trying it out is just an amazing an amazing feeling. I still remember how I started working. In particular, I think like starting from GPT three onwards it really became magical and it really I had a feeling that there's a just a tremendous amount of cool things that you can do with it.
So, I mean, some of the first work that I actually did in that space was focused on problems around database tuning. I mean, those database management systems, they just have hundreds to thousands of tuning parameters that you can use in order to influence the system behavior in all kinds of tasks. So for instance, like tuning how much buffer to allocate, tuning how many cores to spend in certain tasks, and so on and so forth.
And I mean the settings for those parameters can have just tremendous amount of impact on the execution performance for certain workloads. The default settings are generally suboptimal for any given workload. There was just a lot of interest in the database community about how to find good parameter settings to optimize a given performance objective, such as maximizing throughput or minimizing latency for.
A given workload. And this a really, really hard problem just already because there's just a lot of parameters and a lot of possible value combinations. So you're dealing with this huge search space. And traditionally there are no really good models that enable you to predict which settings work well for a given scenario. So you often have to essentially try out different parameter values and
The thing that I was working on in that space, which turned out very well actually, is that a human being, when faced with the same problem, wouldn't just try out random parameter settings and see how they work. But that person would hopefully first have a look at the database manual, at tutorials about database tuning, then and maybe discussions on forums like DBA Stack Exchange and so on and so forth.
Immanuel (05:28.341)
There's just a tremendous amount of useful information about database tuning contained in those text documents. So the idea was basically that now that large language models at that point, a couple of years ago, they just had essentially crossed a boundary where they got really good at understanding natural language text and extracting information from it. So what I was doing in that space was essentially to use those language models in order to extract useful information from database tuning from the entirety.
Of the web, basically. And then you can use that as a starting point for automated database tuning. So that was a project a couple of years ago. And nowadays there's actually lots of work that follows the same direction that uses large language models in order to find good parameter settings for database systems. It is extremely useful because you just narrow down this huge search space to something much more reasonable. And that is
One way in which you can use language models. After that, we have been working a lot on, for instance, new user interfaces for data analysis that are based on data analysis, also on approaches that use language models for code generation to replace certain parts of database management systems. But yes, the very first project that I worked on in the context of language models and database systems, that was really about like database tuning.
And it turned out that language models are very good at like essentially providing you with a shortcut to cut down this huge space of possible parameter configurations.
Nitay (07:07.383)
And it's interesting. So you want the approach of like because traditionally the approach I imagine others would have taken would be, okay, it's a combinatorial explosion problem. There's too many options, like you said. Let's do some clever AI search with some Monte Carlo, some whatever, like try a bunch of things and hope to get to come some good result. And you took it the next step and said, wait a minute, there's a whole bunch of kind of DVA lore, knowledge base, et cetera, that we can pull in.
And so then we're gonna build system that's gonna take that and optimize. It seems like there's a next wave, I imagine. I'm curious if like maybe this takes us to the future of then bringing in some like reinforcement learning and having like, why is it not just like, hey, here's my goal? I want this to run as fast as possible. Why do I why do I even need to care about the language model or the search or the this? Why is it not just like here you go, do it?
Immanuel (07:55.472)
That is a very good question. And in fact, before I was doing this work, the primary method for database tuning was indeed based on reinforcement learning. So the reward function that you're optimizing for, that is essentially the performance gains that you're getting compared to the default parameter configuration on a given workload. And that works also, but it just takes a very long time to converge because in order to
Get a reward value from this reward function, you essentially have to execute at least part of your representative workload with a given parameter configuration. So I should clarify that this first work that was doing in the space of language models and database systems, it actually also used reinforcement learning, but it used reinforcement learning within a much more narrow search space, basically, which enabled it to convert much faster.
So you essentially use the language borders in order to get a search space of reasonable parameter settings and then you use reinforcement learning within that space in order to find the actually best configurations.
Nitay (09:03.847)
Makes a lot of sense. very cool. And what was the most interesting or maybe surprising finding and what was kind of the biggest challenge of of doing this research? Getting to behave well.
Immanuel (09:16.495)
Well, I mean, I think that at when starting this, it actually seemed like a fairly crazy, maybe crazy is too strong, but it feel it felt like an approach that might not work. And so I think just like the general outcome that those language models have really become good enough at understanding text that this approach is valid and is working quite well. I think that was the the primary outcome.
One thing that was interesting to see is that when doing database tuning, I mean, you might sometimes think that just reading the database manual gives you pretty much all the information that you need in order to tune well. And that turned out not to be the case. I was surprised to see that in some cases, the best configuration that the system finds is based on some little snippets of information.
That it extracts somewhere from a forum like Stack Overflow or DBA Stack Exchange, for instance, some discussion about some very specific workload and some very specific scenario. And there's this tuning option that generally might not even be a good idea, but every once in a while it actually makes a huge difference. And the way in which the system often beats those more traditional database tuning tools is often when it finds one of those pieces of information somewhere on the web.
that it can use in order to get better for a very specific work.
Kostas (10:47.765)
okay, that's very interesting. And I want to I I want to ask a question about something that might be like a little bit of like a step ahead of what you are describing, Emmanuel. So and let me do like a little bit of introduction here for my for our for our audience. So traditionally database systems have been like one of like let's say the most complex systems that like we
are engineering, right? Like databases, compilers, and probably also like operating systems. anyone who has gone through like a CS degree, like they know that these are, you know, they're really, really hard like classes to go and take and probably you won't even build the whole thing like at the end, right? Because of like the complexity there. and and I think a lot of like that complexity drives, let's say, some
in the industry at least like some decisions around how the systems are built, right? So obviously we can't write a specialized database for each workload that we have out there. Like it's prohibitively expensive, like or it was prohibited expensive just to start doing like a little bit of introduction here, like to my question. so we have like we needed to have like these let's say categories. Okay, we have transactional, we have all up, let's say we have time series,
Maybe streaming. And then within these, we have a lot of knobs that we use to go and customize and update the system as the workloads change. Now, LLMs, obviously, you said one of the things that they can do is like help figure out these knobs and come up like with good configurations. But another
Interesting side effect that all of them have is that now code is pretty cheap to generate, right? And pretty complex also code. And I've seen a few papers coming up. I think in the LDB there are like a I think like at least two of them where the approach is more of like why not just go and build from scratch a dedicated database specifically for like the workload that we have, right? Which obviously a few years ago that would be a crazy, crazy idea.
Kostas (13:10.837)
tell us a little bit about that and how does this relate with what you are talking about, the optimization of like the configuration of the systems and in a way where do you see things going? Because things like with LLMs like move like extremely, extremely fast. and I think that's like part of like the the what is like really fascinating with them. Like so yeah, tell us tell us a little bit about like about this.
Immanuel (13:39.686)
So that's an absolutely a great point. And I think in certain ways this is the extended version of what I was describing. I mean, if we want to adapt a database systems to get really good at a specific workload or in a specific scenario, we can only do so much by changing the tuning parameter configurations because we already need just the right tuning parameter to do what we wanna do. So in order to have the ultimate freedom.
In order to customize a database management system, in order to make it efficient for a specific scenario, in the end, what you have to do is basically to generate code that does the data processing for you. And we do actually have two papers here at this conference about that specific topic using launch language models in order to generate code for data processing. So that's a topic that we have been working at, I would say basically since it became
Technically somewhat feasible. I did some work a while back in twenty two, twenty-three, about a system called Codex DB, which uses language models in order to generate code for data processing from scratch for specific queries and a follow-up work on a system called GenesisDB, which implements custom operator implementations for a database management system for those traditional relational operators like joints, filters, and so on and so forth.
And that worked somewhat well. I mean like for this early work, we actually typically generated code in the Python programming language. So here the focus was a little bit less on efficiency, as you can infer from the language, and more about like making it generally possible in principle. So there it was actually very challenging to generate code.
That even works correctly. And for many queries back at that time, it was essentially not possible. Even if you gave it multiple retries and if you helped it quite a bit along the way, it was really, really challenging to generate a correct code for complex queries, even in a relatively concise high-level language such as Python. Now, this has recently changed actually. So the two papers that we are presenting here at this conference, they are actually
Immanuel (16:00.01)
using like the latest generation of launch language models in particular from anthropic and open AI, together with an agentic framework in order to generate code for data processing. And now we are generating C code. Typically, even for complex queries, you get the right code and you actually get code that is very, very efficient. And that was a nice outcome. It turns out that if we give those agents a lot of freedom about like what kind of code they're writing
They're often able to write like code that outperforms traditional database management systems. So we actually we have like two flavors of this here at the conference. One project called GenDB. It essentially uses an agentic framework based on large language models in order to generate code that processes specific queries on a specific hardware on specific data.
So this project is really about specializing the generated code as much as possible in order to make it perform really, really well for specific queries. And you actually do get some significant speed ups of a factor of a multiple compared to highly optimized traditional database management systems. And it's actually really amazing like the kind of optimizations that the system considers. We check the code that it is generating. It is a, I should probably say it's an iterative process.
So the system can essentially generate code, then benchmark that code, analyze the code, and then generate better code and try to improve over the original version. And that actually works quite well in practice. And it is doing some very fine brain optimization, for instance, taking into account the sizes of the cache hierarchy on your CPU in relation to the data.
and the query properties in order to synthesize like highly specialized data structures that work best for specific queries and for specific types of hardware. And yes, so this is things that you could not achieve anymore just by changing the settings for tuning parameters in an existing system, because this is really changing like fundamentally how the system represents data and how it processes it.
Immanuel (18:21.615)
So that is one of our systems, GenDB. We have another system called JPreg, which does a little bit of a different approach. So I mean, the former system generates highly specialized code for processing specific queries on specific hardware. The latter system tries to essentially generate code that will be reused across different queries. So you're trying to essentially reply some part of the database management system that accesses database.
And again, I mean it generates very efficient code. I mean the advantage if you generate code that is reused across different queries is that for instance, you can have somebody who verifies that code once and then you can use it in the system going forward. If you generate code for specific queries, then I mean if you say I don't really trust the language model, then you would essentially have to do something for each of those queries.
or query templates in order to verify the generated code. Having said that, we have lots of mechanisms that essentially increase the chances that the code you are generating using those agents is is correct. For instance, we have agents that are tasked with generating automatically test cases that are then used in order to evaluate and test their generated code. And if there's some some if there's some this
difference to the ground truths, a result for a a specific query on a data sample, and we would regenerate the code basically. But definitely I would say that we have been working on that topic since a while, and the focus has definitely changed from just making, just trying to get correct code in a large percentage of cases to really trying to generate code that outperforms existing methods in terms of a data processing time.
Kostas (19:52.574)
Mm-hmm.
Immanuel (20:15.26)
for instance.
Kostas (20:16.607)
Yeah. okay, I have like a a question for you. what's I'm I'm personally like curious about it, but I I I think like it will be interesting also like for the audience. So traditionally and correct me if I'm wrong by the way, traditionally in the database systems there was always like these let's say like two different approaches on how let's say we take like
A query in SQL, right? Like we create like our logical plans, then we get into like the physical plans. And then we either let's say how we how do we actually like implement and execute these physical plans, right? Like there is one approach that says, okay, let's go and generate like code that can be compiled and like we can run it, right? And the other one is we have like these operators, we combine the operators together and we go like and and execute. The reason I
I'm asking that is because my understanding is that generating code for our queries always like felt that can improve a lot like the performance, because obviously like we can go and compile code that you know like the compiler can do also like really great like optimizations based on the hardware you're running, blah blah blah, like all that stuff. But my feeling was also that it never became, let's say.
As adopted as someone would imagine, right? Based on like what it promises in terms of the performance. Part of my understanding, at least, the reason behind that, two things that I I was kind of like identifying. One is the complexity, right? It's like much harder like to debug a system like that, like from the engineers who are building these things. and the other one was like.
query latency, right? Like when you have like to compile, it is like a process that takes time. So if you are doing like online analytics, like for example, okay, you can't add like that much like latency there. So let's say if if it was like a batch process, yeah, like that can potentially like work. Like it's worth taking like the penalty there. But if it is like let's say somewhere where like a human is waiting like to answer like a query, then that starts becoming like a problem. What
Kostas (22:40.761)
LLMs do right that couldn't be done using let's say the previous approaches of like generating code for like the the plans that our like databases were generating. And that's one thing. And the other thing is what about the latency again, right? Like LLMs are slow. Like if it is hard like to compile something, like in terms of how fast you can compile something using let's say like
modern compilers, I assume that waiting for an a jetting system to come up like with the code and make sure that the code is correct and all these things and like goes around is like even like slower, right? So how how what are the differences there and like also how do we fix some of these issues that traditionally we had like with generating like and compiling code for database systems.
Immanuel (23:37.102)
Okay, great questions. Okay, so I mean in general, code compilation has been around since a while in database management systems in general. The short answer about like what language models are doing differently compared to traditional code generation is that you suddenly have a lot of freedom in terms of the code you are generating. Previously you would generate the code based on code templates, basically.
That restrict the like what kind of code it is, which data structures it is using in order to represent results, for instance. So now if you would generate code from scratch using a language model, there are many more optimization opportunities that you can exploit when doing so. About your comment in terms of latency of code compilation and even more so latency of using agentic frameworks in order to generate code.
That is absolutely correct and that restricts to some extent in for which queries you want to use those approaches. If you have short-running ad-hook queries, then you definitely don't want to generate fresh code using multi-agent systems because it would just take way too long generating the code in comparison to the query execution time. But on the other side, if you have queries that have to be executed repeatedly.
Maybe on changing data. Then you have to do this only once and then you can reuse those queries going forward. Going beyond single specific queries, when looking at the workloads from industry, it turns out that actually a large percentage of the queries that you're seeing there, they are not ad hoc queries, but they comply with query templates. And those query templates are known beforehand.
And if you know those query templates beforehand, then you can generate code that applies to a specific query template, which is also something that we have been doing. In those cases, again, you can afford to spend more time in generating the code because the result will be reused across many different instances of the same query template. If you have a lot of ad hoc queries in your specific workload, then what would apply most are approaches.
Immanuel (25:51.972)
which generate code that substitutes some components in the database system and don't have to be regenerated for each new query, but you generate them once for a specific hardware, for instance, or for a specific database, and then you reuse them across different different queries.
Kostas (26:16.309)
Okay, that's that's great. now we covered already some of the parts of like a DB system. There is another one that is related, and I think it's also like very interesting with the LLMs, and I'd love to hear from you the effect that LLMs have on that, and that's the optimizer. and the reason I find it interesting is because writing like
query optimizers, I don't know, I always felt to me like some kind of
science but also art. You have like these people that they come up like with these like these rules that because at the end of the day like optimizers are like a bunch of rules that we codify there and we try like to figure out the right order like to execute them and apply them or have them let's say find all the edge cases and all these things and it's all like one of these things that
Yes, you can reason about it and like s have a systemic approach, but at the end a human will figure out that like, like when we have this kind of distribution of data there and have it might make sense like to put our cost to look like this and like put it there and like all that stuff. so how how do you see the optimizers actually being affected by the use of LLMs?
Immanuel (27:53.102)
Alright, that's a great question. And I mean there are a couple of ways in which LLMs can be used in order to improve query optimization as well. There has been quite a bit of work in our community that tries to leverage LLMs in one way or another one for query optimization. So I mean, similar for code generation, also for query optimization, you might not want to use the LLM directly on the critical path for ad hoc.
Short-running queries just because it takes a little while to use the LLM to, for instance, rewrite the query, for instance. there's also some correctness issues. So, what you can do, for instance, we have been doing some work a while back on using LLMs in order to predict the data properties from the schema. for instance, I mean one.
Problem in general in query optimization is that it is often very hard to estimate the execution costs of plan candidates. And if those execution cost estimates are incorrect, then the query optimizer will select a suboptimal plan. Now, in practice, the primary reason why those cost estimates for candidate plans are incorrect is that the query optimizer has troubles estimating.
The sizes of intermediate results that get generated during query processing. And the primary reasons why it is hard to estimate the sizes of those intermediate results is that first of all, the data might have a skewed distribution. If you use a very simple size estimation model, you might assume that if there are 10 different values in a column and I have an equality predicate selecting one of them.
Then 10% of the data will satisfy that equality predicate. But in reality, it might be that you have certain values that are much more frequent than others in the data, and that can throw off those estimation models. The somewhat harder problem to fix in that context is that different predicates in the same query might have some correlation between them. So if a row in your data satisfies one of those conditions,
Immanuel (30:12.109)
It is much more likely or much less likely to satisfy the second condition. Classical query optimizers typically assume by default that different query predicates are independent, statistically independent from each other. So they will estimate a probability to satisfy a combination of predicates that is far off the true probability, and then they get those size estimates wrong.
Now, the way language models can be helpful in that context is that certain properties of the data, they can be predicted by people actually just from the semantics of the data. If you have the schema of the database and perhaps some accompanying documentation that tells you what is the semantics of the different column names in your data, you might be able to infer some properties of the data without even doing any in-depth analysis.
For instance, if the economist called ID column, then most likely you will find unique values in that column. Or if you have certain things, for instance, you have the column about the car maker and about the car model, you will probably have a correlation between those two. And a human being would know that just from looking at the semantics of the data without even having to do some expensive data.
Profiling. And language models can do that to some extent as well. So we have shown essentially that if you apply language models in order to understand the semantics of the data and predict some properties of the data, it often turns out to be correct. And those are this is information that a query optimizer can directly use in order to in order to counter some of those common problems, such as failing to
Detect that certain predicates are correlated, failing to detect the skew in certain columns. So, in that context, language models can be helpful. And that can be applied even before processing specific queries in a pre-processing step, basically. Then you can feed that to the query optimizer to get more reliable optimization. So that is one way in which you can.
Immanuel (32:32.289)
apply language model in the context of query optimization, but people have been trying out many different ways of doing that, and that's certainly an interesting space.
Nitay (32:44.363)
I'm curious, Emmanuel, through all these kind of research and explorations you guys have done, have you had any findings of things? You know, like I I always think back to kind of the the the AlphaGo. It was like move whatever, 30, 24, I forget the number, but you know, where it did this like surprising move that everybody what thought was, you know, the most terrible move ever and you should never do it. And we've been taught for, you know, decades, centuries probably more that's terrible. And it turns out actually in very certain cases, it's the exact right move.
So I'm curious through all the like kind of explorations you guys have done that you just discussed here, what has been some of the findings or things or tunings that you later go back in the system and you say, huh, it chose to do this or that. Like why why did it choose this? And then you investigate and what's been the learnings there.
Immanuel (33:33.027)
That's a very good question. So I think that I mean, in general, I mean, in particular when looking at those projects where the language model is used to generate code, for instance, I think we had quite a few aha moments when looking at the code that is generating. And I think in in in many cases, those are things that we wouldn't have thought about of doing in order to make query processing more efficient. So I think that I mean, probably most in that project that was actually
surprising in some cases how it was able to for instance like to use unusual data structures in order to represent data and get some additional performance gains that we might have overlooked. And I think in in general, I mean like when referring to Alpha Go, I mean I I think that I mean there I mean
The kind of performance gains that we are generally able to get out of those launch language models when applying them in different scenarios has actually been very positive. I mean, I would not always I mean, in some cases, I think it's like things that a good database administrator would be able to do. But I mean, that's something that requires a lot of database expertise and lay users who
want to use database management systems but are not necessarily experts in terms of the internals about how they work, they would probably not find that. So I mean, even for us, in some cases a surprising surprising moves, so to speak. But certainly for people who are not database experts explicitly, there's a lot of useful hints that they can get from those models for database tuning.
Nitay (35:17.165)
Very interesting. what's been by the way, we we didn't actually go into it, but tell us about some of your kind of you know, some of the research that you're most excited about or the most kind of landmark, what were the actual outcomes? What was the the delta in performance that you reached with the system or the amount of I don't know, less code that that the human has to write? Like how however you were measuring kind of the the value of it. Tell us about some of the results.
Immanuel (35:38.98)
Right. Okay, right. Good question. I mean, what we have been discussing so far is actually mostly about how to make classical database management systems more efficient, systems that process relational data, tabular data, using something like SQL queries. I think one of the most significant impacts of language models in the context of data management.
Is actually really that they dramatically expand the scope of what we can do in data processing in general. I mean, traditionally with SQL, you're essentially processing fairly simple operators on tabular data. So you have for instance, an operator that would enable you to count the number of product reviews in your database that contain a certain keyword, but you couldn't process something.
That counts the number of product reviews in your database that are generally positive about the product. So you cannot process operators that require semantic understanding of the data. And then on the other side, it is traditionally restricted to processing data tables, tables where the cells contain essentially the SQL data types, text, the numbers, and so on.
But you couldn't process something like images or sound files, videos, and so on and so forth. And one way in which large language models really advance data processing is that they expand the scope of what we can do with the data and the types of data that we can process. So one domain that I'm particularly excited about in the domain of database systems, and something where
We have been doing quite a lot of work here in the past couple of years. That is what is often called semantic SQL processing. So you essentially have SQL, which is expanded with AI operators that are evaluated by large language models in the background. So you essentially would be able now to write a query that counts the number of product reviews in your database that are positive towards the product.
Immanuel (37:56.806)
You would also be able to process tables where the cells not only contain text and numbers, but also images, sound files, videos, and so on and so forth. So this is something that has become reasonably possible only since quite recently. But actually people have been trying to do something like that since quite a long time. If you go back something around 15 years ago.
In the database community, you actually had a whole line of research on so-called crowdsourced database management systems. And those are database management systems, which essentially use human crowd workers taken from platforms like Amazon Mechanical Turk, for instance, in order to evaluate some of the query operators in the background. And then you can do things like, for instance, filtering images based on what is shown in the image, or filtering
Sound files based on what people can hear in those sound files. So all of this back then was essentially only possible if you had some humans in the loop, but it is not very scalable and it is very costly to do. So since we have those large language models of let's say GPT three level language models that are really good at zero shot learning so they can process a new task.
Without task specific training or fine-tuning. and since we have those language models, it is suddenly possible to build systems which do something like that without any humans in the loop anymore. So there have been quite a few systems like that released by actually various major database companies, Google, Databricks, Snowflake, MotherDuck.
And so on and so forth. Basically, everyone is currently switching to that. And those systems they really expand a lot what you can do with a query processing. They expand in terms of the data that you can process from just tables to images, sound, videos, and so on and so forth. And they are expanding how you can process that data from relational operators.
Immanuel (40:17.679)
To operators that require some semantic understanding of the data. So that is a very cool topic that I'm excited about. It is very challenging to do, mainly because applying large language models directly to data leads to some significant costs. If you look at the pricing pages of large language model providers like Anthropic or OpenEye and so forth, you will see that the
Per input byte data processing overheads are by many orders of magnitude more expensive compared to traditional relational data processing. So I think in general, the big challenge here is just how to make this more efficient and scale this up to large data sets. So we have multiple projects in which we are working on that. I just actually gave a keynote about it here at a workshop at BLDB. And that is, I think, one of the
most significant impact of large language models that it can really change what we are able to do with the data.
Nitay (41:25.431)
That that sounds really fascinating. So I'll give give the audience maybe a bit of a a teaser towards the the lecture you gave at at BLDB. What what does it take to build a system that can have those operators, like you said, these kind of external AI, async, UDFs, whatever whatever form like you kind of might put it in?
but do it in a systematic way that is, you know, to the points that we discussed before, like kind of meets the latency and the system requirements and the kind of user interactiveness while also, you know, getting a handle on the cost and kind of the the broader kind of system approaches. So how how do you do that? What's the approaches you take?
Immanuel (42:02.854)
Okay, great questions. So, I mean, if you have AI operators in your queries at that point, essentially the entire bottleneck and query processing performance shifts to how you evaluate those AI operators efficiently. So you can essentially do two things, and that's also what I focused on in my keynote. You can do two things. The first one being you can reduce the size of the models that you're using. The second thing, what you can do.
is to reduce the number of tokens that you consume and generate. So those are the things that you can do. And in the keynote, I presented essentially one representative project for each of those approaches. For instance, if you look at the language models even like from the same provider and in the same category, then you have like different model sizes that differ by at least a factor of a hundred in terms of the processing costs per amount of input data.
So you really want to choose like the smallest model you can, but choosing the smallest model always is probably a bad idea because you might not get the output quality that you need anymore. So you basically want to choose the cheapest model, which still satisfies your personal requirements on the quality of the output. So the system that was presenting there, Spare LLM, it's a recent paper from us, it essentially helps to systematically
Select the cheapest model that satisfies user-defined quality constraints. So it is doing that by using profiling. So in those semantic SQL scenarios, you always have to perform the same task, whether that is like a filter operation or a semantic map or whatever, you have to perform the same task on a lot of data basically. So you can afford to
Do a little bit of profiling on some sample of the data in order to get an impression for how well different candidate models work. So you do the profiling, you compare the output of candidate models to the reference model, and then based on what you see, you select the cheapest candidate models that is still sufficiently equivalent to the reference model. There's a couple of interesting questions there. For instance, how much profiling should you do?
Immanuel (44:25.209)
Which candidate models should you profile? All of this costs money, so you want to be smart about like restricting profiling really to finding out the things that can help you ultimately save money when processing the rest of the data. The thing that I the project that I presented when it comes to saving tokens, that is a project that proposes a new implementation for a classical.
For a semantic version of a classical relational operator, the semantic join operator. And what I'm doing there basically is to propose an like to get across like to transfer some ideas from classical join algorithms. So I mean typically those database management systems, they always have multiple different implementations for the same relational operator, like a join. And I'm essentially bringing some of those ideas.
To the domain of semantic query processing and adapting them. So in classical database management systems, there's something called a blockness that loops join, which essentially loads batches of items into main memory in order to become more efficient. And I'm essentially doing some version of that for the language model, where it turns out that if we load batches of the items for which we want to perform the semantic join into the prompt at the same time.
Then we can get significantly more efficient. And maybe I should actually take a step back and explain what is a semantic join in general. So the classical database join that is generally have two input tables, and I want to find all the pairs of rows, one from each of those tables, that satisfy some joint condition together. The semantic version of that is basically the same, with the only difference that the joint condition now is formulated in natural language.
And has to be evaluated via a language model. So many of the existing systems they actually implement this join in a fairly like straightforward manner. They essentially take each possible pair of rows from the two input tables and then ask the language model, does this satisfy the join condition? Yes or no. But when doing that, you essentially have a number of language model invocations that is quadratic in the size of the input data.
Immanuel (46:48.303)
So instead of that, it is much more efficient if you take batches of data from both input tables, put that into the context window of the language model, and ask it to retrieve all the matching pairs that satisfy the joint condition together. You have to be smart about like how you divide the given context window, which limits the total number of input and output tokens that a language model can process across
Tokens used to represent a batch of input rows from the first table, tokens used to represent a batch of input rows from the second table, and tokens reserved for writing the output to the join into the context window. And the paper essentially proposes a couple of formulas that you can use in order to find optimal solutions. That also works very well in practice. So those are two representative projects which essentially focus on each of the two possibilities that you have in this context, either
using smaller models or reducing the amount of data.
Kostas (47:50.544)
Emmanuel, I have a question on that. this is an interesting topic that I guess for some reasons like I've been following like experimenting myself for like for a while. So and I'll take I I think like the the the semantic join is a very good example to to use it like as like the substrate to talk about. So if we take like a classical join, right, like at the end of the day, what we are doing is that
We put the machine to go and do let's say like lexical matching. Like if I'm joining like let's say on the user IDs, I need to find like an exact match between the user IDs between these two like columns, right? in the semi but in a way we push the semantics to the human person, right? Like the human has to know like
what the meaning of these columns are and why they should be joined, right? and either this is like encoded in the schema or it might not even be encoded in the schema but just leaves like in the head of like the the person who writes like the the query, right? Now with LLMs we can bypass that, right? Like in a way we can go and say, hey, like I want to go and find, let's say, all the let's say products
That these people of like this cohort of age or whatever, like they like and be like as generic as that, right? And then we expect that okay, this semantic join is actually going to find like somehow that the descriptions of the products and by filtering also like the users and blah blah blah, like all these things, and like we'll get get you the data back. Now instead of doing that.
Kostas (49:47.23)
Would it be meaningful to say, hey, like I understand the intent here, right, by using LLMs? Use LLMs to extract structure from the data set. So change the schema to represent, let's say, what the data should be joined on, and then go and rely on classical joins to do to do the the join at the end, right?
And there are like two reasons why I'm I I'm saying that, because okay, one is let's say probably like the performance, but there is another thing that I think is like really really important when we are putting LLMs into like SQL, and that's how easy it is to verify what happened, right? And if something doesn't feel right, how I can go and trace that something is wrong.
And actually there is a third one, which is kind of cussing in a way, because if the extraction of the information is like good enough, we don't need like to repeat that. We can just keep doing like natural joints, right? Like and we'll be we'll be fine. so yeah, anyway, that's what came in my mind while you're like describing that, and I think like would be would be interesting to hear like your thoughts on this.
Immanuel (51:12.911)
Those are some great points, and I absolutely agree that if you have a case where, for instance, you could actually revite the semantic joint condition into a traditional formula that can be evaluated automatically on the existing columns, you absolutely want to do that. You might also be able to extract things that can be useful for multiple further queries, for instance, and if that is possible, you should absolutely do that.
you can also sometimes work with embedding vectors, for instance, in certain cases. That can also work for very well for certain scenarios. In particular, if you have something that resembles a traditional quality joint, basically, where you want to find, for instance, documents or even images that are closely related. let me maybe give you some concrete example for where we might at least in the first in the first run need some semantic a joint operator.
So for instance, I was recently talking with some furniture companies, and I mean one of the things that I was discussing with them was actually a system that does the following. So imagine you as a user, you are taking a picture of your unfurnished living room, and then you have a database with pictures of furniture items and some other information such as maybe the price, the size, and so on and so forth.
Now, maybe you want to find a combination of furniture items that together would match well into your living room. Maybe things that are currently not in your living room, so you want to buy something else. And you want to find combinations of furniture items that match well together in terms of the colors, in terms of the materials, in terms of the style that might also match well together with your with your living room picture. So all of this essentially corresponds to one or multiple semantic.
join operations. Of course, you could extract certain types of information from the furniture items. In fact, if you go to a furniture database, then you will already have certain types of data about the furniture. For instance, you would typically have one color associated with the furniture item. But that is very coarse grained because they're different shades of the same color. And typically one piece of furniture would not only have one color but actually multiple different colors. And in order to be sure that you are
Immanuel (53:35.087)
Furniture items match together and that they match your living room, you would really have to actually see it. And that is what the language model can do in the semantic joint. Of course, if you can translate some of those conditions into into predicates that can be evaluated using traditional means, right? Then you should absolutely do that. And we are currently looking, for instance, at rewriting strategies where in cases where users tr are trying.
To use some semantic joint for things where don't need it, where we actually automatically rewrite that into something that can be evaluated much more efficiently. And then, of course, also for embedding vectors, they can be very useful in order to find rated items. You cannot always rely on them because it always depends on what kind of joint condition you have, right? So there are lots of possibilities in order to make this more efficient, and we are working on multiple different implementations.
But this is just to say that there are at least some scenarios where you couldn't get everything you need just from translating this semantic join predicate into a traditional joint predicate. But if you can do it, I absolutely agree with you, you should do it for performance reasons.
Kostas (54:51.047)
And what about being able to verify and debug like the the queries at the end of the day, right? Like I would say like things I I have a feeling that like things that have been like keywords like used a lot like in in research more than in industry actually.
Like provenance, for example, right? Or like doing row-based lineage instead of column-based lineage. Like things that let's say they were always in the mind of like researchers or like people were trying to like to do, but there wasn't enough need in the industry, let's say, or let's say the effort to implement these things wasn't justified for let's say the what we could get out of it.
But I have a feeling that like with LLMs and their nature, we might have to be pushed into like figure out new ways and new tools that have to be part of like our like database management systems to actually go and verify and debug let's say like what whatever analysis we are doing. Especially when it comes like to semantics, because at the end of the day, and I think
If you go like to in a you know like in a in a company out there like doing just boring BI stuff, right? Like still like any mature company will probably have like multiple different definitions of what like revenue is. And the reality is that they are all correct in the right context. That's why they have them. It's not that there's like one way that you can like calculate like revenue, right? And many problems that the industry has, like the market has out there.
Is okay, now a new person is asking a question about revenue, they live in their own org and they don't understand that there might be many different other versions of there, and they just have like a hard time like figuring out which one like to use, or even like use one and it comes out that it's like wrong, right? and someone has to go and debug that and figure out why and understand like all these things. And it's really hard. It's really hard. Like it's it's it's not easy.
Kostas (57:14.493)
And that's without putting anything non deterministic that LLMs can bring, right? so what are your thoughts on that? Like what do we need and also what we can reuse and what we can learn? Because I think like I don't know, database systems have been around I don't know, like since computers were a thing, like maybe even before that, because we always needed like to deal with like data, right? Like one way or another.
And I feel like we keep relearning things that we've learned in a different iteration of the industry out there. So I'm sure there's a lot of things that we can learn today in using LLMs and AI by looking into how like the database systems like deal with problems like in the past. so yeah, like love to hear your thoughts on that.
Immanuel (58:08.451)
Yes, correctness is of course a significant problem, especially when bringing large language models into the mix. I mean, in traditional database management systems, there was some expectation that there's one correct ground truth result, and for a reasonable database management system you should be able to get that ground truth result. there's there has been some work, for instance, on like partially
Giving up those correctness guarantees in favor of better performance. There has been all this work on approximate query processing in general, where you are not processing the entire database, but you're only processing a sample of the database in order to get some approximate results, for instance, some confidence bounds for certain query aggregates. I think it hasn't taken off that well.
Overall in in industry it it seems that the users do have a strong preference for actually getting precise results. But having said that now that we have those language models around and that they are actually applied directly on the data, things are becoming more expensive by many orders of magnitude. I mean, there are like rumors at least of like queries where people have paid like
hundred thousand dollars for one specific query even and so it's really like a very different it's very a very different budget. So it feels that in this context in general people might be more amenable again to methods like approximate query processing because the the trade offs have changed very significantly so as soon as you are
Processing unstructured data like images, for instance, it is very hard to get any formal guarantees on a ground trace. I mean, for some images, it would be even hard for human beings to conclusively say what you're seeing on those pictures. So you can probably not expect to get a formal proof at the end of query processing that the result is precisely correct. I mean.
Immanuel (01:00:25.593)
What is typically done and something that we and others are trying to do is to provide some statistical guarantees that what you're getting is at least equivalent to the best thing you could get using large language models. So some guarantee of the type that if you had used the latest GPT model or the latest anthropic model, and you had put in like
Each data item separately, like you have minimize the amount of data in the prompt, which often improves performance, then you would get a result that is very similar to what you're getting now, even though we have decreased processing overheads compared to that naive approach by a factor of a thousand, for instance. So guarantees that essentially bound how much you're losing compared to the most high quality and
By far the most expensive way of processing things. That is something that we believe can be useful for users. But as long as we are talking about structured data processing, I mean you will not have any formal guarantees. And I mean you that's basically the best that it seems that you can do. Now on the other side, when talking about traditional relational data processing, where you can
Also bring language models into the mix. I just actually had one conversation with an industry representative here at the conference, and he was actually very concerned about like just having agents working on the data that might do things to the data that you don't actually want to happen. And I mean, similar things, I mean, like whether agents are writing SQL queries or whether they're writing codes directly.
It is essentially hard to guarantee that what happens is exactly what you need to happen. So one idea in general in that space, that's something that like some people are currently looking at, including us, is like, I mean, if you can at least within certain boundaries provide some formal proofs to users that what's happening is within certain bounds, what you want to happen, basically. That would make them feel much better. It's
Immanuel (01:02:47.321)
Hard to achieve, but maybe not impossible. But that seems to be one of the next steps on that path. But that is only for structured data processing where you do have ground truths. And at least in principle, you should be able to create a formal proof which tells you that what you wanted to happen with the data has actually happened. But I mean, for unstructured data, I guess we just have to accept whether we apply human beings actually or whether we apply launched language models.
You will not get any formal guarantees.
Nitay (01:03:21.169)
I wanna perhaps close this out because we're we're actually running short on time here. but this one's really fascinating. I wanna tie together a few of the things you said here, because you mentioned and some of the research that you guys were doing, the user has to come and produce, provide some kind of user-defined e-value DF, if I understood it correctly, right? Where you were using that to then kind of run an an internal benchmark to decide you can where you can use a a smaller model. And then here you're kind of tying that, you're you're you're kind of also mentioning you know, kind of
of the the non-determinism of the semantic joints and these other semantic operators and how do we help users get the best results even though they're doing these non-deterministic functions ultimately. And so what's been perhaps in order to kind of close this out what's been some of the best practices you've seen in people using utilizing these systems and writing these sorts of evaluation functions and how to best you know kind of integrate that with the database.
Immanuel (01:04:19.693)
Right. So you are basically asking about like how people get optimal trade-offs between processing costs and result quality for those semantic operators. Right. Okay, great question. so there's quite a few different methods. It depends on the type of operator that we are investigating and what kind of function what what kind of data we are processing.
Nitay (01:04:31.754)
Yeah.
Immanuel (01:04:49.567)
so I mean, in principle, you can again use those two methods. You can either try to go to significantly smaller models, and there's a couple of companies that are doing versions of that. One approach, for instance, is to essentially calculate embedding vectors on the data once and then train fairly small models that are trying to do things like classification or filtering.
Based on those embedding vectors alone. And when you're doing that, you want to have some kind of sense for how you're doing compared to the pseudo-ground truth, which you are getting by applying the largest language model on the data directly. And then of course you can do things like reducing the number of rows, the amount of data in general that you are looking at. So here again.
Come those approximate query processing ideas where you are trying to get some kind of confidence bounds or the number of items that satisfy certain conditions, even if you don't get the the precise number. there are other things where you can, for instance, change the prompt templates in order to get different trade-offs between processing costs and result quality.
For instance, we have actually experimented with that. We have a recent paper in which, among other things, we are pointing at cost quality trade-offs that you can achieve by changing the prompt templates and semantic filter operations. If you use fairly long prompts, where you have a couple of examples of how the filter should work, for instance, then you're increasing your costs, but at the same time you're improving the result quality.
If you use much more concise prompts that just contain one short instruction, check whether that applies, give me a yes or no answer. And then you can reduce the the costs because you are reducing the number of tokens, but on the other side, the quality that you're getting is potentially not satisfactory depending on the specifics. So there's lots and lots of tuning knobs that you can change in order to get different costs between
Immanuel (01:07:07.905)
different trade-offs between cost and result quality and then you can do some kind of profiling in order to get a sense for how far you are from the actual pseudo ground truths that you would get when applying the most expensive model to all the data items in the like with the most detailed prompts that you that you have so
Nitay (01:07:35.329)
Right, good. Thank you for some great advice and great tuning. okay, so with that, we'll unfortunately have to stop it there, but I think we we've all seen that there's clearly lots more we could dive into. So we'd be happy to have you on again. definitely we'll keep track of some of the the great research that you mentioned. Sounds like some very, very innovative and cool projects. And yeah, this has been an absolute pleasure.
Immanuel (01:07:58.723)
Awesome. Thanks so much for having me.
Nitay (01:08:01.367)
Thank you.
