Why validity beats scale when building multi‑step AI systems

WEBVTT
00:00:02.879 --> 00:00:14.960
The AI Fundamentalists, a podcast about the fundamentals of safe and resilient modeling systems behind the AI that impacts our lives and our businesses.
00:00:15.599 --> 00:00:20.239
Here are your hosts, Andrew Clark and Sid Mongalik.
00:00:21.120 --> 00:00:24.320
Welcome to today's episode of the AI Fundamentalists.
00:00:24.480 --> 00:00:27.359
Today we're going to be talking about validity.
00:00:27.760 --> 00:00:29.920
And to help us, we have today's guest.
00:00:30.160 --> 00:00:32.960
We're excited to welcome back Sebastian Benthal.
00:00:33.439 --> 00:00:37.679
Or as you're going to hear referenced throughout this podcast, we call him Seb.
00:00:38.159 --> 00:00:47.759
He's a senior research director at the Information Law Institute from NYU School of Law and a research scientist at the International Computer Science Institute.
00:00:48.079 --> 00:00:49.679
Seb, welcome back.
00:00:50.159 --> 00:00:51.359
Thanks for having me.
00:00:53.200 --> 00:00:59.759
So, you know, we were reading through this work recently, and we see that's still in submission and obviously, you know, subject to change.
00:00:59.840 --> 00:01:02.960
But I think we really liked a lot of what's happening in this work.
00:01:03.200 --> 00:01:14.799
And we were interested if you could help us break down a lot of the concepts in here, since I think there's a lot that we can learn from an academic standpoint and bring over to our worlds either in industry or in other research fields.
00:01:16.319 --> 00:01:35.040
This paper was really born out of conversations with Andrew about frustrations with kind of the way uh industry has been framing AI, and it was kind of meant to be an intervention in um a broader conversation, which is part of the role of academic work.
00:01:35.599 --> 00:01:42.719
Um we're seeing now a kind of wave of agentic AI hype.
00:01:43.359 --> 00:01:56.000
And whenever there's these sort of hype waves, it's useful to unpack the history of it, um, try to figure out what concepts have been feeding into the current moment and see what what's on the other side of it.
00:01:56.560 --> 00:02:06.799
Um and so we we really had to wrestle with um, among other things, the the notion of AI and agency.
00:02:07.040 --> 00:02:18.319
Agentic AI is this kind of new term that sounds like it's the same thing as an AI agent, um, but it flips it on its head in interesting ways.
00:02:18.560 --> 00:02:26.159
So um an agent in sort of classically in AI theory is anything with a sensory motor loop.
00:02:26.319 --> 00:02:30.159
You know, it senses the world and then it acts on the world.
00:02:30.319 --> 00:02:47.680
And if it's intelligent, then according to Stuart Russell, one of the modern godfathers of AI, practicing professor, textbook author, you know, then that agent is going to be acting to achieve goals, typically defined by a utility function.
00:02:48.080 --> 00:03:01.759
And if it's uh solving these problems over time, then it will be a kind of intertemporal utility function, uh called a Bellman equation that talks about future value and present reward.
00:03:02.240 --> 00:03:15.280
So um when people were trained in AI for the past several decades and they learned about reinforcement learning, this is what they hear about AI agents operating in this way.
00:03:15.759 --> 00:03:22.879
Um there's another sense of agency which is kind of more from law and economics, but I think is relevant.
00:03:23.039 --> 00:03:26.400
People think about like having an agent, your agent.
00:03:26.639 --> 00:03:31.439
Okay, and that's the idea that that legally or in economics, you have someone working for you.
00:03:31.599 --> 00:03:36.960
They're your employee, um, they're your doctor, they're your lawyer.
00:03:37.280 --> 00:03:49.360
And um legally, there's something called agency law, which means that there's duties that these agents have to the principals in order to settle what's called the principal agent problem.
00:03:49.520 --> 00:03:53.599
How do you trust your agent to really work in your interest?
00:03:54.159 --> 00:04:08.479
Um with agentic AI, I think that there's a blending of these ideas because there's almost a promise that industry is making that your agentic AI will be your agent.
00:04:09.680 --> 00:04:12.479
Maybe not quite in a legal sense, but almost.
00:04:12.960 --> 00:04:16.560
But it's also an AI agent in the sense that it's achieving goals.
00:04:16.639 --> 00:04:17.839
But whose goals?
00:04:18.560 --> 00:04:19.279
Its own?
00:04:19.600 --> 00:04:20.160
Yours?
00:04:20.959 --> 00:04:27.360
So already there's this ambiguity which which blurs some of the questions of trust.
00:04:28.160 --> 00:04:34.720
Um which I think are actually essential when when we're talking about these problems.
00:04:34.959 --> 00:04:49.600
Um other aspects of agentic AI that are sort of essential in the contemporary definition are um they have to do with the problem complexity and sort of using multiple steps to accomplish a problem.
00:04:49.839 --> 00:04:59.199
This seems to really be like a reaction to the the limitations of the large language model um tool, which is fueling everything.
00:04:59.360 --> 00:05:05.199
There is this promise um that large language models would be generally intelligent.
00:05:05.360 --> 00:05:17.040
You know, a trillion-dollar bet has been placed on a particular architecture of uh neural networks applied to natural language as accomplishing general intelligence, right?
00:05:17.680 --> 00:05:22.319
And it hasn't succeeded in many complex tasks.
00:05:22.399 --> 00:05:30.879
So the the idea that, well, if you start layering multiple queries to an LLM, you can accomplish something more complex.
00:05:31.040 --> 00:05:34.000
Well, that's multi-step reasoning.
00:05:34.240 --> 00:05:38.560
Um and there have been some limited successes in this area.
00:05:38.720 --> 00:05:43.759
There's also, I think, a really important element um, which is external tool use.
00:05:44.399 --> 00:05:50.639
So people are saying, you know, rather than you know, you your your browser could be smarter, it could go shopping for you.
00:05:50.720 --> 00:05:51.839
That's the holy grail.
00:05:51.920 --> 00:05:59.040
You know, if they could find some way to intermediate consumer spending, um, all these companies would be very, very happy, right?
00:05:59.199 --> 00:06:09.600
So um how much can this thing buy for you or operate other aspects of your your work life, your productivity suite, etc.
00:06:11.199 --> 00:06:36.879
So um the thing is when you start implicating um potentially like multiple stakeholders that care about the use of this system, uh, the user's data, which might be private data that might be leaked or not to an external tool, you're not talking about like the simple abstraction of the utility maximizing agent anymore.
00:06:36.959 --> 00:06:42.399
You're talking about a real embedded system with a social context.
00:06:43.120 --> 00:06:52.079
And um, there is a lot of discipline to systems engineering, security engineering, um, et cetera, et cetera.
00:06:52.240 --> 00:06:55.839
But these are not kind of native to the AI conversation.
00:06:55.920 --> 00:07:00.959
And they're starting to leak in in the in these discussions about agentich AI.
00:07:01.040 --> 00:07:14.240
There are recent kind of NURIPS papers that are really systems engineering papers, but it's um it's late comers to this larger section because it's really compensating for what LLMs cannot do themselves.
00:07:15.600 --> 00:07:16.560
Sub, thanks for that.
00:07:16.720 --> 00:07:18.399
We've got great uh setup.
00:07:18.480 --> 00:07:26.959
And that's what I think where you and I really were connected in talking about this is like from our background from economics and things in systems engineering, is like that's kind of this missing link here.
00:07:27.040 --> 00:07:32.000
Is to as you pointed out, like the, you know, you have this monolithic LLM, it's gonna be intelligent and can do everything.
00:07:32.079 --> 00:07:34.240
And then now we're gonna make it do multi-step things.
00:07:34.319 --> 00:07:41.120
This is a little bit living in dream world right now, and where, as we talked about on our uh last podcast, you know, the scaling is kind of showing cracks.
00:07:41.199 --> 00:07:51.680
Like you can't, the infinitely scaling, you know, we're talking about moving data centers in space, under the water, all these things, and we're also seeing that the return on on you know more scaling is is kind of falling apart on these on these systems.
00:07:51.759 --> 00:07:53.680
So that whole promise is being tested.
00:07:53.839 --> 00:07:59.360
But where it really comes down to now with a GenTech is to really be using them for the multi-step world.
00:07:59.519 --> 00:08:04.800
I don't see how you like even you talk about Bellman and like the uh like the optimization there, but that's still an actual optimization.
00:08:04.879 --> 00:08:08.959
You know, any of these algorithms, even if it's an uh an LLM, it optimizes on something.
00:08:09.120 --> 00:08:11.920
They're optimizing on sounding human, predicting the next word.
00:08:12.079 --> 00:08:15.920
So we're really running into like this uh I'm starting to call it like a flying car.
00:08:16.000 --> 00:08:18.399
Like a genetic on how it's being talked about today.
00:08:18.560 --> 00:08:23.120
What's really crazy is like the beginning of 2025, people are talking about it like it's here already.
00:08:23.199 --> 00:08:28.639
And you're gonna be replacing people and which is a whole other can of worms, but you're gonna be using it in the enterprise setting.
00:08:28.800 --> 00:08:37.679
But it's like we all know quantum computing isn't here yet, fusion power is not here yet, self-flying uh self-driving cars is not really even here yet, but flying cars is definitely not here yet.
00:08:37.759 --> 00:08:40.080
But somehow a Gentec was already sold that it's here.
00:08:40.240 --> 00:08:42.639
So like what we're kind of having this now crisis here.
00:08:42.879 --> 00:08:45.519
This is actually a great uh uh article I was reading last night.
00:08:45.679 --> 00:08:47.679
Wall Street Journal came out with something really interesting.
00:08:47.840 --> 00:08:51.120
They had we let an AI run our office vending machine, it lost hundreds of dollars.
00:08:51.279 --> 00:08:57.039
Like it was one of the examples I use instead of what we're talking about, like a travel agent, and that's kind of the example we've been talking through.
00:08:57.200 --> 00:09:03.039
One of the ones I've actually used is kind of a in conversations like uh unless you're viewing vending machines, like that's a low-risk system.
00:09:03.120 --> 00:09:04.000
It's kind of almost that.
00:09:04.080 --> 00:09:12.320
Well, it turns out this low-risk vending machine that Anthropic did with the Wall Street Journal turned out to be buying PlayStations, losing a ton of money.
00:09:12.399 --> 00:09:21.039
Like the whole idea was just make money make up make enough money to survive or just stay break-even and like be an office uh autonomous agent that is running a vending machine.
00:09:21.200 --> 00:09:25.600
And it found ways to lose hundreds of dollars and giving people PlayStations as a vending machine.
00:09:25.759 --> 00:09:46.399
Like basically issues with it without the systems engineering approach, and I think this is what what Seb and I are really trying to like set the stage of the methodology around, is like you're really to actually have successful multi-step systems, and this has been done in pockets of some of Seb's research and things in the agents for a long time, but like to really do it, you have to do systems engineered individual components are thoroughly validated and then they're engineered together.
00:09:46.559 --> 00:09:52.799
That's why I know I talk about it way too much, it's probably on the podcast here, but like the Apollo program and how impressive that was.
00:09:52.879 --> 00:09:58.000
And like NASA is still leading on like systems engineering, and very few people understand those areas, and that's what's great.
00:09:58.159 --> 00:10:01.519
That's bleeding into AI world of historically just scaling.
00:10:01.600 --> 00:10:05.919
We don't think we scale is kind of in uh now it's overgeneralization and unfair generalization.
00:10:06.000 --> 00:10:14.559
But that sometimes the computer science industry kind of operates that way versus like the we sent people to the moon using something much less powerful than our smartphone, right?
00:10:14.639 --> 00:10:31.679
And like buying how do we design and validate and fit together these systems, we could build robust multi-step agentic systems, but we're gonna have to bring back in systems engineering, control theory, like like different optimization techniques for even the say the thing we've talked about in this podcast a lot of like the vending machine example.
00:10:31.759 --> 00:10:41.679
If we've used utility theory and basic linear programming and just basic ROI maximization calculations, you would have made money, not bought people PlayStations, which was not on the SKU, right?
00:10:41.759 --> 00:10:54.480
So, like that it this is like the I think there's a major reset needed, and it's what like it's a methodologically framing, is what uh what Sev and I are trying to accomplish with with the setup, and I think he's done a great job with this initial draft.
00:10:57.279 --> 00:11:11.519
And what I'd really like to dig in is if we have these existing issues like compounding errors in multi-step processes, difficulties in applying these into complex systems, potential misalignments, and maintaining governability.
00:11:11.840 --> 00:11:13.600
Uh I guess it's right in the title.
00:11:14.240 --> 00:11:17.679
But I, you know, I want to ask the question why is validity what I need?
00:11:18.240 --> 00:11:38.960
You know, how does validity basically help us address these concerns and potentially resolve them one of the hardest things about what's implied by agenda is um is how many different sort of actors there are.
00:11:39.360 --> 00:11:45.039
Um if you if agentic AI involves tool use, you have a third party.
00:11:45.200 --> 00:11:56.799
That's um you also have your your end user that might be one of several stakeholders involved in the procurement decision for the agentic AI system.
00:11:57.120 --> 00:12:09.679
You've got the developer of the agentic AI system itself, who may or may not control the large language model, or certainly might not be able to trust or evaluate everything that the large language model can do.
00:12:10.000 --> 00:12:15.440
So right now you've got four different actors, all potentially misaligned.
00:12:16.799 --> 00:12:30.799
And even if each of those actors are able to per perform their job perfectly, um you might find that the end user is not satisfied or some of the stakeholders are not.
00:12:31.039 --> 00:12:36.559
So the the the point of the title of the paper was just to say keep your eye on the prize.
00:12:37.120 --> 00:12:40.960
What is it that is actually trying being accomplished by this thing?
00:12:41.120 --> 00:12:49.679
People marketing foundation technologies are going to try to say, you know, all you need is this foundation technology.
00:12:49.840 --> 00:13:00.480
Uh it's a play on attention is all you need argument that you know to solve all AI problems, you just need this particular transformer technique.
00:13:01.279 --> 00:13:05.360
Incredibly ambitious, perhaps hubristic statement.
00:13:05.840 --> 00:13:14.639
What an end user actually needs is something that validly accomplishes their goals.
00:13:14.879 --> 00:13:16.639
And this is just definitional.
00:13:16.720 --> 00:13:22.159
Um trying to uh trying to propose a tautology to answer what might be something overhyped.
00:13:22.240 --> 00:13:24.000
But working backwards from that.
00:13:24.960 --> 00:13:31.279
Well, how do you show validity to the end user of the system or the stakeholders of the system?
00:13:31.440 --> 00:13:38.080
I'd say the principal stakeholders of the system, that is the principles to whom the AI system is the agent.
00:13:38.399 --> 00:13:52.080
Well, that requires a lot of testing, a lot of demonstration, involves giving that principal the information that they need to trust the system, given that the system might be changing in time.
00:13:53.440 --> 00:14:01.120
If every you know six months there's a new LLM model release, you know, that we've had we've seen this before.
00:14:01.279 --> 00:14:13.039
You know, uh you end users that love the the chumminess of one version of the uh open AI model that then are disappointed because the personality changes later on.
00:14:13.279 --> 00:14:25.039
If you have a software pipeline, what tests are you running when you change the LLM model in the back end to ensure that it achieves the same level of quality that you come to expect of it?
00:14:25.279 --> 00:14:32.159
That's that needs to be communicated back up to the user from the agentic AI system designer.
00:14:32.639 --> 00:14:42.639
So um I think these are active are challenges that the LLM engine is not going to address.
00:14:43.440 --> 00:14:47.759
So uh the question is how do we shift focus to these problems?
00:14:47.919 --> 00:14:51.919
What is the scientific, rigorous method of of solving them?
00:14:52.159 --> 00:14:56.559
And uh how do we focus on real human needs?
00:14:58.159 --> 00:15:08.320
I think this is the to the more we talk about this, and I always like I so enjoy our conversations of that like we we get like really abstract and like the the kind of like the metaphysics we've been talking about on the podcast.
00:15:08.399 --> 00:15:16.399
Well, a metaphysics series, uh stuff we uh we're doing when we're kind of going through like metaphysics of thinking of like what does actually thinking mean and kind of like going back to Aristotle and Plato and stuff like that.
00:15:16.799 --> 00:15:17.440
So cool.
00:15:17.679 --> 00:15:20.240
But like we you and I do that kind of on AI when we talk.
00:15:20.320 --> 00:15:23.919
So like, yeah, actually, we might want to have you come in on one of the metaphysics specific episodes.
00:15:24.320 --> 00:15:26.000
Let's do yeah, like definitely.
00:15:26.080 --> 00:15:29.600
Let's uh we'll be brief after the podcast recording and get you up to speed on that.
00:15:29.759 --> 00:15:34.399
But in any case, um, I think this is really the heart of where I even see what is AI governance even.
00:15:34.639 --> 00:15:40.000
AI governance is not a compliance task, it's not a like a paperwork, paper pushing thing.
00:15:40.080 --> 00:15:42.159
It's like the people like put all this baggage on it.
00:15:42.240 --> 00:15:51.679
To me, the root of it is understand it's a principal agent problem, and how do you understand like that doesn't mean it's not too agentic, even like principal agent problem of what is your system doing?
00:15:51.919 --> 00:15:52.799
How is it performing?
00:15:52.879 --> 00:15:53.679
Who is checking it?
00:15:53.759 --> 00:15:55.600
How are you making sure that it's fit for your purpose?
00:15:55.919 --> 00:16:01.840
This is where it comes back to like early models, or even like cruise control, or like testing and validation, or like why do we have engineering?
00:16:01.919 --> 00:16:04.000
We really need to get to the part of AI engineering.
00:16:04.080 --> 00:16:06.720
Like there's software engineering, there's civil engineering, there's other things.
00:16:06.799 --> 00:16:08.159
We need to get to AI engineering.
00:16:08.240 --> 00:16:10.080
And that we're not, we're not there yet right now, right?
00:16:10.159 --> 00:16:13.200
We have hackers, we have programmers, we're not really gotten to AI engineering.
00:16:13.279 --> 00:16:15.840
And I think that's the key of like what even is governance?
00:16:16.000 --> 00:16:19.679
Governance is making sure that you have that alignment of what are you trying to accomplish.
00:16:19.840 --> 00:16:24.639
And any documentation just helps to be like, I document it first off from my own ability.
00:16:24.720 --> 00:16:29.279
So to it's a it's a prop, like writing something down is just your own thinking process and internalizing it.
00:16:29.360 --> 00:16:29.919
So there's that.
00:16:30.080 --> 00:16:34.960
There's also if I if I want to be able to show Seb and have him understand it, it needs to be written, like those sorts of things.
00:16:35.120 --> 00:16:39.440
But it really gets down to like the, and this is where validity and validation is the heart of it.
00:16:39.519 --> 00:16:41.919
So there's really the two parts of like, what is the objective?
00:16:42.000 --> 00:16:45.919
You can't get away, you can't, um no LLM will ever think for you.
00:16:46.000 --> 00:16:51.200
And this is the thing that's different of like, even if so, we do get super intelligent LLMs that are going to replace accountants and things.
00:16:51.279 --> 00:16:54.000
I don't see that happening anytime soon, but hypothetically, let's say that is.
00:16:54.159 --> 00:17:05.680
As we've seen with any other technological um innovation, the part that makes us human, the thinking is the the actual thinking, the reasoning, the human aspects, the abstract thinking, that's what we'd be good at.
00:17:05.759 --> 00:17:08.079
So you'd just be leveraging the tool for those capacities.
00:17:08.240 --> 00:17:19.119
But in any case, whenever you're leveraging a tool as an agent, you're the principal and you're levering an agent, which can be a one-step agent, a generative AI system prompting, is still an agent by that macro definition.
00:17:19.839 --> 00:17:22.240
You are the one that are defining what is it that you want.
00:17:22.319 --> 00:17:26.720
And like the siren song that it seems that we're trying to get with AI is I don't want to think.
00:17:27.039 --> 00:17:39.599
And as a company, like you have to be validations and validity of any AI system, specifically now that we're going from like single-step models to potential multi-step models, just becomes so much more critically important.
00:17:39.759 --> 00:17:48.799
And the only way you could really even get these in the world you talked about, Seb, of like agents interacting with different agents, like over the MCP protocol and things, is to have them independently validated.
00:17:48.880 --> 00:17:53.519
And then there's an orchestration layer of like the systems engineering of how you put the pieces together.
00:17:53.599 --> 00:17:58.400
Like you did not build a rock, sorry, for the Apollo again, but you didn't build a rocket in one factory.
00:17:58.559 --> 00:18:08.000
It was all over the United States, which tons of different factories of their components are rigorously validated and stress tested for their specs, then they're assembled at some point.
00:18:08.160 --> 00:18:16.880
A gentic to actually be successful, and again, we're nowhere near here, but it doesn't have to be as far as a flying car, but we have to be like this validity is all you need.
00:18:16.960 --> 00:18:20.559
I think it's just really the underpinning of the two big issues of what am I doing and why.
00:18:20.720 --> 00:18:24.319
We're not tackling you need to tackle that, but that's the definition of principal agent problem.
00:18:24.400 --> 00:18:30.240
But then the validity of the system, how are we validating that these individual components are operating as intended?
00:18:30.319 --> 00:18:34.079
And then we can know, like, if I'm chaining things together that they're working.
00:18:34.240 --> 00:18:39.680
But uh I really think this is the crux of what is even AI governance or what what is any of it we're trying to accomplish here.
00:18:39.759 --> 00:18:40.960
And we can't really get away from this.
00:18:41.039 --> 00:18:43.119
And this is where we've kind of tried to skip steps.
00:18:43.279 --> 00:18:48.400
As you know, like you can't if you're a you know it's couch to 5k, not couch to marathon as an example, right?
00:18:48.480 --> 00:18:49.359
You can't skip steps.
00:18:49.440 --> 00:19:06.559
And I think we've in 2025, the public narrative, enterprise at times, like tried to skip steps of like we're gonna go ahead and we're gonna just we're gonna be replacing humans, we're gonna be doing all this great stuff because Anthropic says it can happen, and we're skipping why we're even doing it, what we're trying to accomplish, and how do we validate that it's actually ready to be used.
00:19:07.599 --> 00:19:08.480
I agree.
00:19:09.200 --> 00:19:17.279
It seems like you know, uh a couple decades ago, uh people would try AI felt still seemed like a research field.
00:19:17.359 --> 00:19:30.000
There was some use of expert systems more broadly, but in terms of AI research, it was a lot of it was you would you would develop an AI system to solve some kind of problem, uh some discrete problem.
00:19:30.319 --> 00:19:40.319
And I think where things are at now with Gen AI systems is they're trying to get these things just to accomplish tasks.
00:19:41.039 --> 00:19:54.880
The the logic is if we're gonna have these things automate labor, we need to figure out what the tasks are that are part of that labor and automate those tasks.
00:19:55.440 --> 00:19:59.680
And for kind of productivity software sort of stuff, like writing.
00:20:00.160 --> 00:20:08.799
Writing documents, coding, which are significant significant uh forms of labor, right?
00:20:08.960 --> 00:20:15.119
But they're they're trying to figure out how to automate the tasks involved with that work.
00:20:18.799 --> 00:20:20.799
There's a lot that that leaves out.
00:20:21.200 --> 00:20:34.559
Um and when things go beyond being just something that's accomplishing tasks to someone that's trusted to perform a role in a long term.
00:20:34.640 --> 00:20:42.240
Like think of think of um hiring a contractor for something versus having an employee, right?
00:20:42.480 --> 00:20:56.799
There's uh a lot that's wrapped up in the ongoing memory and development of that person that um an agentic AI system might need to evolve to be able to accomplish.
00:20:57.039 --> 00:21:06.160
There's also um the issue of thinking about long-term interests and goals as opposed to short-term tasks.
00:21:06.640 --> 00:21:13.359
So you can scope in the classical AI agent definition a problem down so that it's trivial.
00:21:13.599 --> 00:21:16.960
But what the uh Yonadab Shavid et al.
00:21:17.200 --> 00:21:27.039
open AI paper, that's one of the definitional ones for a genetic AI today, is about is about um task and environment and goal complexity.
00:21:27.279 --> 00:21:37.119
And it seems like a lot of that complexity comes from longer-term stake and uh sort of multi-agent interaction in the environment.
00:21:37.359 --> 00:21:41.839
You know, we talk about sort of impact on social life.
00:21:42.400 --> 00:21:58.960
Um, so uh if you think about a system that um advises a person, say a clinical health related chatbot, um what kind of information does that need to know about the patient?
00:21:59.200 --> 00:22:03.759
And how does it maintain that information over time in a trustworthy way?
00:22:03.920 --> 00:22:17.039
These are a bunch of challenges that are not part of the kind of like the basic um assembly of an agentic AI system, but there are people working on this problem now, um, because they see it as as the frontier.
00:22:17.200 --> 00:22:33.920
So I think it's it's um some extent these are worthy problems, um, but it does seem like a shift in focus is necessary in order to really uh realize the value that's been promised.
00:22:36.880 --> 00:22:54.079
I think that that's really poignant, and I think that it it gets me to this idea that we've been talking about this inherent mechanism design that's built into these agentic AI systems, wherein they understand the world around them and they respond accordingly to that world and within that world context.
00:22:54.559 --> 00:23:10.880
And one thing we talk about in the paper is this idea that you can model an enterprise context as a multi-agent socio-technical system, or this idea wherein agents don't just operate on their own goals, they operate within an environment and within feedback within that environment.
00:23:11.200 --> 00:23:23.039
Can we talk a little bit more about how we instill these types of contexts into agents and how we see agents properly respond back to feedback that they get in that environment?
00:23:24.640 --> 00:23:30.960
Yeah, I think that's uh that's uh I've seen this mentioned as sort of an emerging issue.
00:23:31.119 --> 00:23:50.640
Um in like there's there are people uh you know at Google, at uh OpenAI, etc., are working on kind of individual-facing assistance, but this idea of like the enterprise assistant seems to open up a whole other can of worms.
00:23:51.039 --> 00:24:00.880
Um because uh in in even like the economic theory, we don't have a very good way of comparing what people want.
00:24:01.680 --> 00:24:07.839
Um, you know, it's not like the utility of one person adds up easily with the utility of another.
00:24:08.000 --> 00:24:15.119
There's ways of fudging it with dollar values, and maybe that is the right way to do it, but that that requires building in some assumptions.
00:24:15.279 --> 00:24:19.920
So aggregating preferences over many people is a known challenge.
00:24:20.319 --> 00:24:32.000
And um understanding what's in, say, a company's best interest when that can be very multifaceted and there can be multiple bottom lines is is challenging also.
00:24:33.200 --> 00:24:48.319
And if you talk about the AI alignment problem in that setting, what does it mean for an agentic AI system to be aligned with the interests of the corporate body that is its its principle?
00:24:49.759 --> 00:24:59.359
It's a fascinating problem, but I don't think that anybody's come up with a clean answer in social theory, computer science theory, economics.
00:25:00.240 --> 00:25:07.519
Um but that one of the closest things to it would be something like mechanism design, which is saying, like, well, okay, suppose you're designing an auction.
00:25:07.599 --> 00:25:11.039
You know you've got a lot of people bidding in the auction.
00:25:11.359 --> 00:25:21.359
How do you design the auction in order to get people to, for example, reveal their preferences honestly or maximize the revenue of the auctioneer?
00:25:21.920 --> 00:25:26.000
So these are well studied in operations research and economics.
00:25:26.160 --> 00:25:38.240
Um, applying these, that kind of thinking to the design of a in-house agentic AI in an enterprise setting is extremely ambitious because that's a complex problem.
00:25:38.400 --> 00:25:48.880
There's inherent computational complexities to solving this, you know, multiple optimization problems wrapped in other optimization problems.
00:25:48.960 --> 00:25:57.599
And so it's uh there's real questions about how do you make that into some a problem that decomposes nicely into something that can be solved.
00:25:58.400 --> 00:26:00.240
But that's a great research frontier.
00:26:00.559 --> 00:26:06.960
And I think this is what we we talked about in our wrap-up uh as well of like, I really think this is, and I there's other folks saying this as well.
00:26:07.039 --> 00:26:10.720
It's like we're really turning back into scalability is not going to get us to where we want.
00:26:10.799 --> 00:26:12.240
Like, we've skipped steps.
00:26:12.480 --> 00:26:23.920
I think what we're really seeing is like there there is a uh such a future here, and I'm excited about the ability of AI to make our lives better, and there's a lot of different avenues here, but I think we're really realizing that we can't skip steps.
00:26:24.079 --> 00:26:35.680
Research has to come next, and also this like this universal function idea that one like that you can make an a copilot that does everything for your enterprise at the same level of your highly skilled workers.
00:26:35.920 --> 00:26:40.559
That idea is starting to get a little long in the tooth, and you're realizing the complexity here.
00:26:40.720 --> 00:26:48.240
However, like just be the going back to there is so much deep research and like the decomposing or the auction design and the mechanism design, there's so much here.
00:26:48.319 --> 00:26:57.680
Like we can be building these really expertise systems, but it's gonna be a research-led thing, not a we're just gonna throw computer the problem and hope for the best type thing.
00:26:57.839 --> 00:26:59.200
And that's what the paradigm is shifting.
00:26:59.279 --> 00:27:09.119
And this is where what we're also trying to set up with the paper is how software as a service has really transformed the enterprise landscape of like you can buy specialized tools for specialized tasks.
00:27:09.279 --> 00:27:12.880
The V2 of that is agents, right?
00:27:12.960 --> 00:27:21.039
Like that, and that's what we're trying to set up as the the paradigm of the in the validity is all you need is agentic AI, meaning multi-step AI as a paradigm.
00:27:21.119 --> 00:27:26.799
And again, AI can have many, whatever the solver is, and AI methods' definition is just doing things humans used to do.
00:27:26.880 --> 00:27:28.960
So the solver could just be a bunch of if statements.
00:27:29.039 --> 00:27:30.319
It's whatever it is, it doesn't matter.
00:27:30.400 --> 00:27:40.799
But the paradigm of multi-step AI that is now built for specific business purposes, you've seen a little bit of people talking about like small language models, but they're still missing the point of like a small fine-tuned language model.
00:27:40.960 --> 00:27:43.359
It might language model might not be the best solution.
00:27:43.440 --> 00:27:52.079
But getting back to you can build these very targeted expertise systems that might use mechanism of design, they might use this, they might use an LLM, they might use only linear programming.
00:27:52.240 --> 00:27:53.359
We don't know, it doesn't matter.
00:27:53.519 --> 00:28:00.960
Uh, but they're there's very optimized that can solve these enterprise problems and they can be offered as SAS or something like that.
00:28:01.119 --> 00:28:09.759
However, it takes a research focused driven, let's take a step back, let's let's understand the socio-technical dynamics or or what have you of the specific problem we're looking at.
00:28:09.920 --> 00:28:11.519
And then we can build those systems.
00:28:11.599 --> 00:28:17.599
And that's where I think it we're, I don't, I think it's still probably gonna take six months to a year until that we fully transition that way.
00:28:17.759 --> 00:28:25.599
But I am already seeing the folks talk about like the need for like the research or research led or back to the labs and that kind of conversation point.
00:28:25.759 --> 00:28:32.480
And I think uh Steph's on the forefront of that and seeing a lot of that develop as well of like people realizing that to what what is the goal we're trying to accomplish?
00:28:32.559 --> 00:28:35.759
The goal we're trying to accomplish, we can accomplish, but we've been going about it the wrong way.
00:28:35.839 --> 00:28:40.799
And we're gonna have to do it the the the sequential order and have research embedded into it.
00:28:40.880 --> 00:28:51.039
And it might and this the universal approximation function might not be viable, at least in the short term, versus optimized systems that are built.
00:28:51.200 --> 00:28:54.480
Like eBay is a case in point of mechanism design and auction theory.
00:28:54.559 --> 00:28:58.240
It's beautiful, it's a fantastically built company, and they did this.
00:28:58.559 --> 00:29:04.400
We could be doing this in actual sys AI systems, not just a marketplace, but you have to do it in the right way.
00:29:06.240 --> 00:29:19.359
One thing I've heard um is uh a frustration with the recklessness of open AI in kind of pushing pushing the boundary of LLMs with their focus.
00:29:20.000 --> 00:29:47.279
Um and and uh but also as a as a part of that, as kind of the research community starts to kind of backfill and respond, um there's a role for a lot of other kinds of expertise that have been marginalized by the generative AI thrust.
00:29:47.920 --> 00:30:07.119
Um and um as those gen AI tools become more commodified, right, they're gonna be a component of these AI agentic systems in the future because people really like natural language interfaces.
00:30:07.440 --> 00:30:12.240
But um they're gonna be easier and easier to use.
00:30:12.319 --> 00:30:18.640
And so the value add on top of it is going to depend on things from other fields.
00:30:18.880 --> 00:30:26.400
There's people uh now engaged in conversation with sort of formal methods in security or software validation.
00:30:26.720 --> 00:30:50.079
Um, there's people working on multi-agent systems, there's people working on um other kinds of computer science and economics, crossover theory that are seeing the way that they can engage with these systems and add a lot of value to it by providing that systems engineering scaffolding.
00:30:52.079 --> 00:30:53.759
So I think that's that you're right.
00:30:53.839 --> 00:30:56.160
That's I think where things are going to have to go.
00:30:56.480 --> 00:31:05.119
And where it seems that they are going because uh you know the the steam is running out of of you know the current system.
00:31:05.200 --> 00:31:07.359
So the adjustment.
00:31:07.519 --> 00:31:18.559
But with all this additional like compute capacity, uh it's amazing the it's amazing the ambition that's been unlocked by this sort of agenda era.
00:31:18.640 --> 00:31:35.440
I mean, it would be uh 10 years ago, if you were to say, oh, we want to like automate this whole aspect of your business and you trust us, we're gonna accomplish it, it would have been hard to make the argument that that would be feasible or worth the effort.
00:31:36.000 --> 00:31:50.240
Um but now I think what's happening is it's actually becoming lower, it's becoming cheaper to do many things, uh, partly because of Gen AI advances in kind of coding tooling.
00:31:50.480 --> 00:31:58.799
So it the cheap the cheaper it is to build, the more valuable it is to know exactly what it is you're trying to build.
00:31:59.279 --> 00:32:02.160
Because uh that's really the challenge.
00:32:02.640 --> 00:32:06.480
You can build anything, so what do you do?
00:32:08.720 --> 00:32:36.960
And I guess that kind of lines me up into asking if we see this as basically maybe a temporary dead end, or we're temporarily blocked, and we're still figuring out how to make these you know universal approximation functions work in the ways that we want them to work, and to understand and quantify our needs and our goals, what should we be looking forward to as the future of AgenTech AI?
00:32:37.200 --> 00:32:43.279
You know, what what do you see as like the next evolution of it that we're gonna see in the next coming years?
00:32:43.440 --> 00:32:48.559
And you know, where is the research pointed in to address some of these shortcomings?
00:32:51.039 --> 00:33:02.960
My gut says it's in requirements engineering, you know, something so basic as uh I mean, the people talk about the specification problem for AI alignment.
00:33:03.039 --> 00:33:14.720
They think about it in terms of how do you specify the utility function that really reflects the interests of the, you know, the principles of the humans involved?
00:33:14.880 --> 00:33:17.279
That's essentially requirements engineering.
00:33:17.440 --> 00:33:23.039
But there might be better, more efficient ways of going about doing it in the future.
00:33:23.200 --> 00:33:33.519
Like how do you efficiently collect requirements, figure out what pick people actually need, um, not just in the short term, but in the long term.
00:33:33.680 --> 00:33:39.759
How do you effectively balance the interests of multiple stakeholders within an enterprise setting?
00:33:40.559 --> 00:33:43.039
These these I think are are important problems.
00:33:43.119 --> 00:33:52.079
And once once they are solved, if they can be solved, then everything else becomes an engineering problem in service to those goals.
00:33:52.240 --> 00:33:55.839
It's a bit waterfally, that way of thinking.
00:33:56.000 --> 00:34:03.039
Um, there's probably a more agile, dynamic way of accomplishing a similar thing, but that that I think is the crux.
00:34:04.160 --> 00:34:04.480
Yeah.
00:34:04.640 --> 00:34:28.559
There was something that you were saying earlier that was it was almost alluding to it, but there's a psychology of like explorers, experimenters often don't like being backed into a hard goal or outcome because there's a there's a feeling that they're going to be blocked from doing the exploration that might find them something else or find the next biggest innovation.
00:34:28.719 --> 00:34:34.800
So when you start getting into like the thing that's going to make it better is like like the requirements management.
00:34:34.960 --> 00:34:36.000
Where are we going with this?
00:34:36.159 --> 00:34:37.440
What are the outcomes?
00:34:37.679 --> 00:34:41.039
I think you were almost saying it was you were getting to a big part of that.
00:34:41.199 --> 00:34:46.800
Is that does that psychology resonate with maybe what some reluctance is?
00:34:49.840 --> 00:34:55.119
I guess I I I sympathize with the explore experiment or mindset.
00:34:55.599 --> 00:35:15.679
Um but um you know I think there's an element of experimentation which we've seen, which is um what happens if we blow this up really big.
00:35:16.159 --> 00:35:27.679
You know, uh it's a kind of Manhattan project approach to experimentation, which um okay is one way to go about science.
00:35:28.639 --> 00:35:53.360
But um maybe because I'm uh you know I've got a foot in social science, I think that there's there's um there are plenty of of interesting mysteries to be solved um at the intersection of humans interacting with each other and with machines, and there are people that have been working in that field of human-computer interaction for some time.
00:35:53.920 --> 00:35:58.400
Um and there are still problems that are not solved.
00:35:58.559 --> 00:36:14.000
And what talking about sort of requirements engineering and how to back that into a complex systems design when you've got multiple differently interested agents operating, you know, sociotechnically around each other.
00:36:14.159 --> 00:36:17.119
Um, there are many, many open problems there.
00:36:17.280 --> 00:36:25.440
So that's the scientists, I think, will not get bored if we can drive the field to take these problems seriously.
00:36:25.679 --> 00:36:34.800
Um and then you know, if you talk about from kind of an industrial standpoint, you know, I think industry wants to deliver something of value.
00:36:35.199 --> 00:36:40.559
So um, and they have to innovate on how to deliver that value, how to create that value.
00:36:40.719 --> 00:37:00.559
So um say I think figuring out how to turn these really challenging conceptual problems into ones that are well formed enough to solve and deliver, I think that's also a contribution for for you know for the engineers who are an important piece of the puzzle.
00:37:02.480 --> 00:37:23.360
And I guess just to like, you know, take a moment for us to synthesize our thoughts here today, I think that a lot of what we discussed today kind of gets at the heart of this problem where we're currently being sold technologies which solve a problem, but we're not being sold mechanisms that understand our needs and actually resolve those needs.
00:37:23.519 --> 00:37:34.960
And this is causing some almost inherent disconnect where we're being sold the foundational model rather than being sold uh the application of the of the problem that we actually need to resolve.
00:37:37.119 --> 00:37:39.199
Yeah, we're we're skip we're skipping steps.
00:37:39.280 --> 00:37:50.719
I think is the kind of the what we've said here is like there's ways we can accomplish what are we trying to accomplish, but defining what are we trying to accomplish versus right now, FOMO, let's all use AI, quote unquote AI, meaning LLMs.
00:37:50.880 --> 00:37:56.000
Let's all use the LLMs because everybody else is using them and they'll somehow magically get do what we want them to do.
00:37:56.079 --> 00:37:58.079
But we still haven't decided what we even want them to do yet.
00:37:58.159 --> 00:38:04.239
We have to decide what we want them to do, get really good at that, and then designing what's the best way to solve that problem.
00:38:04.480 --> 00:38:24.800
And the paradigm, and that's where kind of what we put in the paper, of the idea that multi-step AI-based systems is a thing, I think is kind of the biggest takeaway of 2025 and what we can leverage really well to make very solid systems if we relax what that means to means what's actually engineered systems and what's the best solver for the job.
00:38:24.880 --> 00:38:33.360
As we've talked about, you can interact in natural language, that's the neurosymbolic, and there's different ways we can do that of like you can be talking in natural language, but you don't have to be solving in extra language.
00:38:33.519 --> 00:38:38.400
But we've been convinced that you need to be doing everything with a natural language machine, right?
00:38:38.559 --> 00:38:40.400
So it's like, but we can district those.
00:38:40.800 --> 00:38:44.880
And I think the biggest takeaway is so we know that multi-step can be a thing.
00:38:45.039 --> 00:38:47.920
We know we have to figure out the principal agent problem, and that's never gonna go away.
00:38:48.000 --> 00:38:50.800
There's not like a let's build an AI system that can take away that.
00:38:50.960 --> 00:38:52.239
That's part of us being human.
00:38:52.400 --> 00:38:55.199
We're gonna have to just embrace that that's never going away.
00:38:55.599 --> 00:39:16.800
Own that, get good at that, and then once we've decided what those do we want to get good at, then we focus on it's gonna be a little bit harder yards potentially, but we can make really performant systems that are multi-step, and then just uh marrying those together as like the the V2 of what SaaS can be for an enterprise is multi-step AI systems that need to be designed and validate and validate it.
00:39:16.880 --> 00:39:20.000
And validate it is the key part of like you design it, but how do you make sure it's fit for your purpose?
00:39:20.079 --> 00:39:22.400
And that's where the validity comes in.
00:39:24.800 --> 00:39:25.440
Perfect.
00:39:25.679 --> 00:39:30.719
Well, Seb, it has been a pleasure to have you back to the show to discuss your research.
00:39:30.880 --> 00:39:34.639
Uh, we really appreciate your insights and everything that you've had to share.
00:39:35.280 --> 00:39:35.760
Thank you.
00:39:36.000 --> 00:39:37.360
My pleasure as well.
00:39:38.159 --> 00:39:48.639
And for our listeners, if you enjoyed today's episode, you will also want to check out our episode with Seb, where we discuss differential privacy and contextual integrity.
00:39:49.360 --> 00:39:59.199
As always, if you have any questions about the content on our show, please reach out to us at theaifundamentalists at monotar.ai.
00:40:00.079 --> 00:40:01.280
Until next time.