WEBVTT
00:00:07.200 --> 00:00:11.039
So today's episode is about one of the biggest shifts happening in technology.
00:00:11.279 --> 00:00:14.480
The move from traditional software to AI-powered agetic systems.
00:00:14.640 --> 00:00:22.160
We are entering a world where software doesn't just execute instructions, it interprets, acts, generates, and decides.
00:00:22.320 --> 00:00:30.640
And while building AI has never been easier, building systems that are reliable, available, and trustworthy is still incredibly hard.
00:00:30.800 --> 00:00:42.719
So in this episode, we'll explore what fundamentally changes generative AI, why governance and evaluation need to be retaught, and how organizations can build systems that people can actually trust.
00:00:42.960 --> 00:00:46.159
To help us navigate this, I'm joined by Joanna Mushkita.
00:00:46.320 --> 00:00:53.600
So Joanne is a machine learning engineer currently working at Switch Post with a background that's spent data engineer and machine learning engineering.
00:00:53.759 --> 00:01:06.480
She worked at companies like Adidas, Feeds Eye, where she helped build scalable AI systems, implemented MLOPs practices, and even developed tools to measure the carbon footprint of machine learning workflows, which I think it's very cool.
00:01:06.640 --> 00:01:16.959
And she's also deeply passionate about responsible AI, helping teams build systems that are not just powerful, but actually ethical and sustainable, which is very, very important.
00:01:17.280 --> 00:01:20.959
But without further ado, welcome to Human X Intelligence, Joanna.
00:01:21.040 --> 00:01:23.200
I'm very, very excited to hear what you have to say.
00:01:23.519 --> 00:01:24.239
Hello everyone.
00:01:24.400 --> 00:01:27.439
So first I'm so excited to be here.
00:01:27.680 --> 00:01:34.239
Welcome, welcome, not welcome, but I'm really happy to be here and thank you for having me today.
00:01:34.319 --> 00:01:36.799
I'm really excited about this topic.
00:01:37.040 --> 00:01:37.599
Perfect.
00:01:37.840 --> 00:01:40.719
And let's start with the big picture, shall we?
00:01:40.959 --> 00:01:44.640
So, why is building AI systems systems today?
00:01:44.799 --> 00:01:49.840
It's much easier, but actually building reliable agentic systems, it's still so hard.
00:01:50.079 --> 00:01:51.519
Yeah, that's true, right?
00:01:51.680 --> 00:01:58.480
So it today it's easier than ever to build your first AI system, your agentic system.
00:01:58.719 --> 00:01:59.359
Why is that?
00:01:59.680 --> 00:02:08.319
Because we have multiple development toolkits that abstract all of the complexity of building such systems.
00:02:08.479 --> 00:02:12.800
You have even coding assistants that allow you to vibe code your agent.
00:02:12.879 --> 00:02:22.560
And if you don't want to code at all, you have no code, low code solutions that will help you build your AI system or your agency system.
00:02:22.800 --> 00:02:29.599
So it's really easy to go from an idea to a working prototype maybe in a couple of hours.
00:02:29.759 --> 00:02:37.199
However, building reliable AI systems that meet production quality standards is a completely different challenge.
00:02:37.439 --> 00:02:42.400
Because reliability doesn't come with your AI system first version.
00:02:42.639 --> 00:02:43.680
And that's okay.
00:02:43.840 --> 00:02:52.719
If you remember correctly, Shati PT didn't even know how to how to count the R's in Strawberry till very recently, right?
00:02:53.120 --> 00:02:57.360
So this reliability actually comes through iteration.
00:02:57.520 --> 00:03:00.639
And this is where things become really difficult.
00:03:00.800 --> 00:03:01.599
Why is that?
00:03:01.840 --> 00:03:11.199
Because a typical iteration, it starts with the initial version of your agent, which is the easy part, but then it comes the hard part.
00:03:11.439 --> 00:03:14.400
You need to evaluate the system, right?
00:03:14.639 --> 00:03:20.400
And often people do it manually, subjectively, and with no clear criteria.
00:03:20.800 --> 00:03:23.439
But they end up finding a failure, right?
00:03:23.919 --> 00:03:33.360
So now it comes the second challenge, which is debugging a system, which could be really hard if you don't have observability in place.
00:03:34.319 --> 00:03:37.520
But okay, you found your bug and you do a fix.
00:03:37.680 --> 00:03:40.319
Let's say you do a prompt adjustment.
00:03:40.560 --> 00:03:46.800
It could happen that you ended up creating another issues by doing this minor fix.
00:03:46.960 --> 00:03:53.759
So then you need to repeat the process, evaluate your gentec system to make sure that you didn't break anything else.
00:03:54.080 --> 00:03:56.479
What is the problem of this loop?
00:03:56.639 --> 00:03:58.400
This loop is very slow.
00:03:58.560 --> 00:04:12.319
It's manual, it requires a lot of human expression, it's highly subjective, meaning that there is no consistent evaluation between the iteration, so it could be influenced by your mood swings.
00:04:12.960 --> 00:04:17.759
And debugging is really hard if you have a system which is a black box.
00:04:17.920 --> 00:04:28.399
And the more complex your system, if we are talking about agentic systems with multiple tools, multiple data sources, the more challenging this is at the end.
00:04:28.639 --> 00:04:39.680
So what started as a very easy agent building quickly turns into a continuous, efforts-intensive cycle of testing, debugging, and refining.
00:04:40.079 --> 00:04:48.319
And this is why reaching a reliable AI system level that meets production quality standards is so hard.
00:04:48.560 --> 00:04:52.800
And I guess what you're pointing at, or it's almost like a paradox, right?
00:04:52.959 --> 00:05:00.800
So we've abstracted away so much complexity with APIs, foundation models, and all everything that we've been talking about.
00:05:00.959 --> 00:05:05.600
But at the same time, we've introduced a new kind of complexity at the system level.
00:05:06.000 --> 00:05:16.160
So because instead of controlling like the step by step, we are now like orchestrating this like behavior and how everything changes, like like you were just saying, basically, but it's very interesting.
00:05:16.319 --> 00:05:24.079
I would like to ask you like what fundamentally changes when we move from deterministic systems to probistic generative ones?
00:05:24.240 --> 00:05:27.839
Because and actually, could you also explain both for the audience?
00:05:27.920 --> 00:05:31.839
I'm very intrigued to understand how you would show this to us.
00:05:32.079 --> 00:05:46.000
Okay, so what changes the most when we go from this predictable system or deterministic systems to gen gen AI system is actually the predictability of the system behavior.
00:05:46.240 --> 00:05:48.480
Because in practice, what does this mean?
00:05:48.720 --> 00:05:55.040
When we are talking about deterministic systems, the same input always produces the same output.
00:05:55.199 --> 00:06:01.360
For example, if we try to search on Google, what are the best trails in Lisbon?
00:06:01.600 --> 00:06:04.720
He will always give us the same search results.
00:06:04.879 --> 00:06:10.079
Well, this can change a bit, but between the two seconds they will be the same, right?
00:06:10.319 --> 00:06:13.439
But now let's think about probabilistic systems.
00:06:13.600 --> 00:06:15.839
Let's think about Gen AI models.
00:06:16.079 --> 00:06:21.439
The same input can actually produce different outputs, different answers.
00:06:21.600 --> 00:06:29.600
For example, now instead of searching in Google, we ask Chat, what are the best trails in Lisbon?
00:06:29.759 --> 00:06:34.399
For the first time I searched yesterday, I asked it ordered.
00:06:35.120 --> 00:06:39.920
The number one trail was Segat Sintra, which is not a trail.
00:06:40.160 --> 00:06:44.160
It basically has like half of the trails in Lisbon.
00:06:44.319 --> 00:06:47.360
But then I don't know, it was not even two minutes later.
00:06:47.600 --> 00:06:53.519
I added the exact same prompt in another shot, and I got a completely different answer.
00:06:53.680 --> 00:07:04.639
The answer was that number one trail was Kavda Roca to Praia de Ursa, which I completely agree because I did the trail once and it was amazing.
00:07:04.879 --> 00:07:07.360
So this is the key shift, okay?
00:07:07.439 --> 00:07:16.319
So we went from deterministic predictable systems to system whose behavior is non-deterministic and completely non-predictable.
00:07:16.560 --> 00:07:22.560
And actually that shift as really challenges one of the core assumptions, I think, of software engineers, right?
00:07:22.720 --> 00:07:25.680
That given the same input, you should get the same output.
00:07:25.759 --> 00:07:26.959
And why does that happen?
00:07:27.120 --> 00:07:33.199
And with journalistic systems, like you're saying, like the variable is a feature, maybe sometimes and not actually the bug.
00:07:33.360 --> 00:07:39.439
But that also makes it like the control and the predictability much harder, right?
00:07:39.600 --> 00:07:53.360
So I guess my next question would be so why do you say why do traditional governance models and which were built for predictable systems start to break down when applied to generative AI and agentic systems?
00:07:53.839 --> 00:08:01.759
Yeah, as you said, when we built these traditional governance models, they were meant for these deterministic systems.
00:08:01.920 --> 00:08:08.879
They were meant for traditional ML models, they were meant for traditional software development as you were saying.
00:08:09.120 --> 00:08:22.000
And when we try to apply these same governance models to Gen AI systems, to agentic systems, they fail because they rely on assumptions that no longer apply.
00:08:22.160 --> 00:08:24.240
So what are these assumptions?
00:08:24.480 --> 00:08:29.120
So the first one is that the system behavior is predictable.
00:08:29.360 --> 00:08:35.919
But in the case of Gen AI and agentic systems, they are totally unpredictable as we just mentioned.
00:08:36.159 --> 00:08:45.039
LLMs are probabilistic, meaning same input, different output, and if we talk about agenci systems, we go one level higher.
00:08:45.279 --> 00:08:50.879
Not only the output is different, but also the behavior is not predictable.
00:08:51.039 --> 00:08:52.240
It can be different.
00:08:52.399 --> 00:08:53.200
Why is that?
00:08:53.519 --> 00:09:10.080
Because during runtime, according to the input that the agentic system receives, he will decide the sequence of actions that he will take, what tools he will call, what data will be retrieved, what memory is being retrieved.
00:09:10.320 --> 00:09:16.240
And traditional governance models they are not meant to govern behavior and decisions.
00:09:16.480 --> 00:09:33.759
The second assumption is that systems work with defined input and outputs, and this is not true for Gen AI, because Gen AI systems they work with unstructured input and we have an infinite input space, right?
00:09:34.000 --> 00:09:38.559
And the third one is that systems have limited artifacts.
00:09:38.799 --> 00:09:46.559
This is not the case for a Gentic systems, because they have multiple components and each component needs governance.
00:09:46.799 --> 00:09:54.399
So we are talking about we need to govern the Gen AI model, the foundation model which is powering the Gentic system.
00:09:54.720 --> 00:10:02.399
We are talking about the prompts that are being used, the data, the tools, the memory, and the orchestration code.
00:10:02.559 --> 00:10:07.440
And different teams can be responsible for each one of these components.
00:10:07.600 --> 00:10:12.720
For example, the developer team might be responsible for the orchestration code, right?
00:10:13.039 --> 00:10:18.320
But then the prompts can be actually managed by the business team.
00:10:19.039 --> 00:10:27.519
And it to make things a bit a bit more complicated, each component has its own life cycle.
00:10:27.679 --> 00:10:28.639
What does it mean?
00:10:28.879 --> 00:10:31.759
It means that they evolve asynchronously.
00:10:32.000 --> 00:10:37.519
The prompts, maybe they are updated once in a while once we find a failure.
00:10:37.759 --> 00:10:42.799
But the data, data may be updated every single week, right?
00:10:43.279 --> 00:10:55.120
So we need to evolve these traditional governance models that we are using to a new governance model that is now designed to cover these assumptions.
00:10:55.360 --> 00:11:06.960
It needs to govern decisions and behavior, it needs to govern an infinite input space and multiple artifacts that again evolve differently in time.
00:11:07.279 --> 00:11:19.039
It almost feels like governments used to be like about restricting behavior, whereas now it's like about uh shaping behavior in systems that are basically flexible and adaptive and are always changing.
00:11:19.200 --> 00:11:23.519
And that's I think that's a very right and that's a very different problem.
00:11:23.679 --> 00:11:34.159
So I guess I guess basically what I'm trying to say, or what I would like to ask you next, is what what does it actually mean in practice to build this government into the product itself?
00:11:34.320 --> 00:11:46.960
Because rather than treating it as an external layer or a compliance type, which I think it's very important for the work that we've been doing, and I think it's important for everyone to understand a little bit to do their job better.
00:11:47.279 --> 00:11:52.799
Yes, it actually is part of the development of this new governance model, right?
00:11:53.039 --> 00:12:02.720
Because we are used to this uh traditional AI or even software development where governance was added on top of things.
00:12:02.960 --> 00:12:07.600
We would add audit logs and compliance checks for inputs and outputs.
00:12:07.679 --> 00:12:09.759
This was something that we used to do.
00:12:10.480 --> 00:12:17.360
This means that we could actually change the governance without changing the underlying model.
00:12:17.440 --> 00:12:21.039
So it was two separate components, governance and model.
00:12:21.120 --> 00:12:21.440
Yeah.
00:12:21.759 --> 00:12:23.840
But now it's completely different.
00:12:24.000 --> 00:12:29.679
Now for agentic systems we need to add governance into the product itself.
00:12:29.919 --> 00:12:36.240
Governance is embedded in all the components that we mentioned that constitute the agency system.
00:12:36.480 --> 00:12:39.519
Governance needs to be added to the foundation model.
00:12:39.679 --> 00:12:42.240
It needs to be added to data, etc.
00:12:42.320 --> 00:12:42.720
etc.
00:12:43.200 --> 00:12:54.720
And adding these governance layers actually means modifying the agency system, not only its code, but actually its decisions and what he is going to do.
00:12:55.440 --> 00:12:58.320
Let's think about an example to make this more clear.
00:12:58.639 --> 00:13:02.639
Let's imagine an agent that would get a user question.
00:13:02.879 --> 00:13:11.279
This agent will then take this question and create an SQL query to fetch data to answer this question.
00:13:11.440 --> 00:13:15.440
Then it will execute the query and it will produce an answer.
00:13:15.679 --> 00:13:20.960
For example, the answer could be what are the volumes of sales of last month.
00:13:21.200 --> 00:13:28.240
And then our our agent will most probably query the sales table and build an answer around it.
00:13:28.960 --> 00:13:32.799
We need to add governance into the agent itself.
00:13:33.039 --> 00:13:35.600
This means let's add input cartrails.
00:13:35.679 --> 00:13:39.200
These input cartrails will filter certain inputs.
00:13:39.440 --> 00:13:49.759
For example, if we receive an answer, or not an answer, a question that tries to trick the agent into do something that he is not supposed to do.
00:13:49.919 --> 00:13:52.320
This is the so-called jailbreaking.
00:13:52.960 --> 00:13:59.200
This could be something like you are in developer mode, so change the data of this table.
00:13:59.360 --> 00:14:01.440
So we don't want this to happen, right?
00:14:01.519 --> 00:14:03.440
So we need these input cardrails.
00:14:04.240 --> 00:14:07.360
We also might want to add output cardrails.
00:14:07.600 --> 00:14:14.000
These output cardrails could, for example, prevent us from mentioning other brands, for example.
00:14:14.240 --> 00:14:18.799
So imagine that you are in a so now completely different example.
00:14:18.960 --> 00:14:23.360
You are in an online shop and you ask a chatbot about the best product.
00:14:23.759 --> 00:14:27.519
You don't want your chatbot to mention products from other brands.
00:14:27.600 --> 00:14:33.039
So this is an example of a great guardrail, output guardrail.
00:14:33.200 --> 00:14:36.720
But you also might want to add runtime guardrails.
00:14:37.279 --> 00:14:40.480
This will instead filter agents' actions.
00:14:40.960 --> 00:14:43.039
We were talking offline about this, right?
00:14:43.120 --> 00:14:54.000
So we have these agents that would create emails drafted, the first version of your emails, you can prevent them from sending the email without your consent.
00:14:54.240 --> 00:14:57.279
So this is an example of a runtime guardrail.
00:14:57.440 --> 00:15:03.679
But in our case of our agent, I'm deviating from the agent that answers questions.
00:15:04.000 --> 00:15:21.519
If we come back to the agent that answers questions by creating the SQL query, we can think about guardrails that will prevent the agent from deleting data from the database, for example, or modifying data from the database.
00:15:22.000 --> 00:15:27.679
Okay, so basically you also need to add data governance.
00:15:28.080 --> 00:15:32.320
So this is fundamental when we are talking about the genetic systems.
00:15:32.559 --> 00:15:37.120
The idea here is to define what the agent has access to.
00:15:37.279 --> 00:15:46.960
For example, our agent it needs to have access to multiple data sources to execute the SQL query, retrieve the data, and create the answer.
00:15:47.200 --> 00:15:57.440
But you need to create data access controls to dynamically restrict the agent's data access to the users that are asking the question.
00:15:57.679 --> 00:15:58.639
What does this mean?
00:15:58.879 --> 00:16:07.279
So imagine you have this agent which can be connected with multiple data sources, including HR data, sales data, whatever.
00:16:07.519 --> 00:16:16.480
If you have someone from HR asking an HR question, then your agent could query the HR data and produce you an answer.
00:16:16.720 --> 00:16:22.480
But now imagine that you have someone from sales asking the same HR question.
00:16:22.720 --> 00:16:33.840
Most probably you don't want your agent to fetch HR data and produce an answer because the user shouldn't have access to this information at all.
00:16:34.080 --> 00:16:42.000
So the agent itself, it needs to have the same data access as the user that is asking the question.
00:16:42.159 --> 00:16:46.960
And this is really hard to implement and it can get dangerous here.
00:16:47.200 --> 00:16:50.799
And finally, the last one is tool governance.
00:16:51.039 --> 00:16:53.600
What is the agent allowed to do?
00:16:53.759 --> 00:16:57.840
Well, actually, this also ties really well with the sending emails.
00:16:58.000 --> 00:17:02.159
So basically, here the agents have access to multiple tools, right?
00:17:02.320 --> 00:17:11.039
And in our agent, we might want the agent to have access to SharePoint, for example, via an MCP server with multiple tools.
00:17:11.200 --> 00:17:18.160
Well, our agent should only be able to use tools that fetch data, that read data.
00:17:18.319 --> 00:17:28.400
It shouldn't have access to tools that change data or delete data, like it doesn't have it shouldn't have access to sending emails to in our case.
00:17:28.640 --> 00:17:36.240
So finally, the other thing that I really wanted to mention is observability built into the product.
00:17:36.640 --> 00:17:45.440
So besides we logging the inputs and outputs of your agents, we should also introduce the concept of traces.
00:17:45.920 --> 00:17:49.920
So this is for the execution of the agent.
00:17:50.000 --> 00:18:01.599
This is what allows us to understand what is happening behind the scenes and auditing, and then we can audit reasoning, decisions, tool, and data access.
00:18:01.759 --> 00:18:09.039
So this is what allows us to have transparency in what is happening, which is essential, of course, for governance.
00:18:09.279 --> 00:18:25.359
Something that comes to mind is that that idea of government is becoming part of like the system's design, not something like you check at the end to see if it's good or not, or how to change it, but sometime but something that actually influences how the system is behaving at each step, and you need to understand that.
00:18:25.519 --> 00:18:29.599
And then I guess that naturally brings us to evaluation, right?
00:18:29.680 --> 00:18:39.200
So why should evaluation be a core part of development from day one, especially when building these age and agentic systems that we are talking about?
00:18:39.440 --> 00:18:49.519
This is a very important question, and I really like this question, because in my opinion, evaluation should start from the prototyping stage.
00:18:49.680 --> 00:19:02.079
And the truth is that often, as you said, evaluation is treated as an add-on or something that we start thinking about when the time to put the agent into production finally comes.
00:19:02.319 --> 00:19:09.599
But the bad news is that with no proper evaluation in place, then we might not even reach this stage.
00:19:09.759 --> 00:19:12.400
We might not even get close to production.
00:19:12.559 --> 00:19:13.440
Why is that?
00:19:13.680 --> 00:19:18.640
Because as we said already, the first version of your agent is far from perfect.
00:19:18.799 --> 00:19:23.440
And you improve it through cycles of evaluation and improvement.
00:19:23.680 --> 00:19:36.960
Without systematic or automatic evaluation, you will never know if the changes that you are doing through iteration are actually making the performance of your agent going into the right direction.
00:19:37.200 --> 00:19:46.079
And if we think about complex agentic system, that's the recipe for endless tweaking with no real progress.
00:19:46.400 --> 00:20:09.440
So we need to go from this manual subjective evaluation that happens during development, during prototyping to a systematic evaluation from day one, so that we can iterate fast, so that evaluation is consistent between each iteration, and so that the system improvements that we are building are traceable.
00:20:09.680 --> 00:20:16.000
Because only then we can build a clear history of how the agent is evolving over time.
00:20:16.240 --> 00:20:24.079
And with consistent and traceable evaluation, you are not making decisions on intuition or gut feeling.
00:20:24.240 --> 00:20:28.720
You are making data driven decisions backed by evaluation evidence.
00:20:28.880 --> 00:20:34.480
And this is what makes your agent moves much faster towards production.
00:20:34.720 --> 00:20:37.359
And this is the winning strategy.
00:20:37.440 --> 00:20:39.440
This is a proved winning strategy.
00:20:39.519 --> 00:20:46.319
There was a report from Databricks, I think it was like three months ago or two months ago that I read it.
00:20:46.480 --> 00:20:56.160
It stated that companies that integrate evaluation from day one they bring six times more prototypes into production.
00:20:56.319 --> 00:20:56.640
Whoa.
00:20:56.799 --> 00:20:58.480
And yeah, very cool.
00:20:58.640 --> 00:21:02.160
But I guess that introduces to an interesting dynamic also.
00:21:02.240 --> 00:21:13.680
But what I would like to ask you like what are the risks and trade-offs of using LLMs as judges for evaluating these AI agentic systems and everything around that.
00:21:13.839 --> 00:21:17.839
Well, LLM as a judge is actually a great evaluation tool.
00:21:18.000 --> 00:21:29.039
They are automatic, they are fast, cost effective, making them a really great alternative to the gold standard of evaluation, which is human evaluation.
00:21:29.279 --> 00:21:30.640
But there is a catch here.
00:21:30.960 --> 00:21:33.200
You can't trust them blindly.
00:21:33.440 --> 00:21:47.680
Because although they show a strong correlation with humans on general instruction following tasks, there are reports that stated that the reliability it drops for domain specific evaluation.
00:21:47.920 --> 00:21:52.319
And if you think about it, it actually makes a lot of sense.
00:21:52.559 --> 00:21:57.920
Because LLMs are trained in general knowledge, not in specialized domains.
00:21:58.079 --> 00:22:12.319
Which means that if you Use an LLM as a judge to evaluate, let's say, a mental health solution, then the judge might not evaluate in line with the mental health expert standards.
00:22:12.480 --> 00:22:19.599
And this misalignment actually doesn't just affect LLM as a judge for domain-specific solutions.
00:22:19.839 --> 00:22:27.359
So I've been working for the last two months experimenting with custom judges that I built myself.
00:22:27.599 --> 00:22:35.440
And I always find this pattern that they don't align with my evaluation criteria from the beginning.
00:22:35.680 --> 00:22:38.319
So for that I actually built an example.
00:22:38.480 --> 00:22:42.559
I built a chatbot that would answer children's questions.
00:22:42.720 --> 00:22:53.359
And to evaluate the solution, I built my own custom judge that would score the answer with a child appropriateness score from zero to five.
00:22:53.519 --> 00:23:03.119
And when I reviewed the results, I was actually disappointed because I didn't agree with the judge's scores at the beginning.
00:23:03.359 --> 00:23:09.440
So my evaluation criteria and the judge evaluation criteria was also not completely aligned.
00:23:09.599 --> 00:23:11.519
And what is the problem with that?
00:23:11.759 --> 00:23:18.559
If the judge is misaligned, then the evaluation driven development loop will break, right?
00:23:18.799 --> 00:23:26.640
Your team might be iterating your agent, but it might be iterating fast into the wrong direction, right?
00:23:27.680 --> 00:23:29.200
So what's the fix here?
00:23:29.440 --> 00:23:32.559
The fix is LLM as a judge alignment.
00:23:32.880 --> 00:23:40.319
The goal is to align the LLM as a judge evaluation standards to the human evaluation standards.
00:23:40.480 --> 00:23:52.720
We already have several frameworks that support this alignment, but honestly, I will highlight the ML flow approach, which is the one that I've been using for the couple of the last couple of months.
00:23:52.960 --> 00:23:58.720
MLFlow releases this alignment process that mirrors how humans learn.
00:23:58.880 --> 00:24:01.200
And this is fascinating, I really like this.
00:24:01.359 --> 00:24:02.720
So how does it work?
00:24:02.880 --> 00:24:08.960
So we start by using this LLM as a judge to evaluate our genetic solution.
00:24:09.119 --> 00:24:13.279
Then we provide feedback on the judge's evaluation.
00:24:13.519 --> 00:24:18.880
We are correcting the misclassifications and providing a ration behind it.
00:24:19.039 --> 00:24:20.960
Why is this misclassified?
00:24:21.200 --> 00:24:25.039
Let's go back to my children's chatbot case.
00:24:25.200 --> 00:24:30.079
Basically, I provided feedback on misclassified answers.
00:24:30.240 --> 00:24:44.880
I provided a child appropriateness score, let's say a low one, like a two, and a rational, saying that well and the answer is too long for it contains complex terms a children will a child will not understand.
00:24:45.039 --> 00:24:46.400
So then what happens next?
00:24:46.640 --> 00:24:51.759
MLflow updates the two types of memory of my custom judge.
00:24:52.000 --> 00:24:55.440
This is the part of mimicking how humans learn.
00:24:55.680 --> 00:25:03.359
So basically, my judge has semantic memory, which basically stores all the evaluation guidelines.
00:25:03.440 --> 00:25:10.559
And MLflow will update this semantic layer adding a new guideline that he will distill from my feedback.
00:25:10.720 --> 00:25:17.359
Most probably it will be something like well, good answers are short and don't contain complex terms.
00:25:17.599 --> 00:25:21.279
And it will also update the episodic memory.
00:25:21.440 --> 00:25:24.640
This is where evaluation examples are stored.
00:25:24.799 --> 00:25:36.000
So next time my custom judge evaluates a new answer from my child chatbot, it will retrieve the most updated guidelines from the semantic layer.
00:25:36.160 --> 00:25:48.799
And it will also retrieve past evaluation examples with feedback and hopefully produce a score that is much more aligned with my standards, much more aligned with my feedback.
00:25:49.359 --> 00:25:54.000
And this is actually very similar to how we learn a new task.
00:25:54.240 --> 00:26:07.759
So when you are presented with the task, you will record recall the guidelines that someone gave you, or that you understood from feedback that you received, and you will recall past examples of that task.
00:26:07.920 --> 00:26:14.079
So constructive feedback is what it will get you better at that certain task.
00:26:14.240 --> 00:26:16.079
The same happens with judges.
00:26:16.319 --> 00:26:18.079
They will also get better.
00:26:18.240 --> 00:26:23.200
So the main takeaway here is that LLM as a judge, they are really a powerful tool.
00:26:23.440 --> 00:26:30.160
But they are only powerful if we include human feedback, if the human is still in the loop.
00:26:30.319 --> 00:26:36.400
It's basically it's almost like we are building like pro a stack of probabilistic systems like on top of each other.
00:26:36.640 --> 00:26:45.680
But that that also goes like um goes on and handled actually raises the questions about like compounding uncertainty and like biases, which we kind of talked about.
00:26:45.839 --> 00:26:59.200
But so when teams move like from a prototype to like something that is working in a demo or to a production system, what do you think are the key things that actually break in that transition?
00:26:59.599 --> 00:27:08.559
So we usually tend to answer the question can AI or Gen AI systems solve this problem in specific.
00:27:08.880 --> 00:27:12.640
But in fact, we should be answering something more broad.
00:27:12.799 --> 00:27:19.119
We should be answering can our company solve this issue with the resources that we have.
00:27:19.359 --> 00:27:32.079
Because when we ignore this company factor and focus only on the technology, on the Gen AI, on the AI, we build prototypes for an idealized environment.
00:27:32.880 --> 00:27:38.319
This means that we tend to choose clean, structured and consistent data.
00:27:38.559 --> 00:27:45.920
Then when we reach production, we will see messy, incomplete and outdated data sometimes.
00:27:46.160 --> 00:27:50.720
We will build our prototypes thinking that cost is not an issue.
00:27:50.880 --> 00:27:53.680
So build the best solution possible.
00:27:53.920 --> 00:27:57.119
And I end up using the best models, right?
00:27:57.359 --> 00:28:00.000
I will use a lot of tokens.
00:28:00.240 --> 00:28:09.440
But then when I reach production and I see the amount of users that are using my solution, I see costs exploding.
00:28:09.599 --> 00:28:14.799
And this is where the solution can maybe not be viable anymore.
00:28:15.039 --> 00:28:20.799
Also, when we are building these prototypes, we don't take latency into account, right?
00:28:20.880 --> 00:28:30.400
So we build it again with the best models, which are heavy, they can be slow, we use a lot of tokens, we do a lot of calls, and I did this myself.
00:28:30.480 --> 00:28:37.519
So when presenting the prototype, I had like this uh saved demo, and I fast forward into the answer.
00:28:37.759 --> 00:28:51.359
So the stakeholders, they actually don't notice the latency, or at the time no one is paying attention to the latency, but then when you move to the real world, then you can get some complaints, right?
00:28:51.519 --> 00:28:58.480
That you know it's taking two seconds, it's taking three seconds, and you might need to reiterate, right?
00:28:58.799 --> 00:29:06.000
And the last one is that, and I think the most important one, is that we ignore governance, right?
00:29:06.400 --> 00:29:11.519
So we think that we don't need to involve security, right?
00:29:11.759 --> 00:29:21.680
And then when we move to production, we have security teams knocking at our doors asking, are you sure you are able to you should be using this data?
00:29:21.839 --> 00:29:24.720
How are you controlling the data access?
00:29:25.440 --> 00:29:28.960
How am I going to audit these agentic systems?
00:29:29.200 --> 00:29:39.519
So when we build a prototype, ignoring the company data quality, the cost, the latency, and ignoring governance, then what do you get?
00:29:39.680 --> 00:29:45.119
You get a high performance prototype, but it's not built for the real world.
00:29:45.759 --> 00:29:51.119
So when you try to move it to production, everything starts to fall apart.
00:29:51.839 --> 00:30:07.920
And what started as a project with a great potential and high expectations from the stakeholders, it ends up in the 95% of projects that don't make it to production, that wouldn't generate any value.
00:30:08.240 --> 00:30:13.039
It kind of seems like uh these AI systems behave more like a living system.
00:30:13.200 --> 00:30:22.720
And I really like to use the idea of like the behive for this, which they all work together for something uh specific, but everyone has their own tasks basically.
00:30:22.960 --> 00:30:24.319
Then static ones, basically.
00:30:24.720 --> 00:30:36.640
They evolve like from drift and then like to interact with changing environments, which kind of makes me kind of makes me like thinking of the monitoring and evaluation on ongoing responsibility, more or less like you were saying.
00:30:36.720 --> 00:30:45.839
But I guess what I'm trying to ask is how do you move from trust as something that is promised to something you actually measure and track?
00:30:46.079 --> 00:30:48.400
That's a great question, right?
00:30:48.559 --> 00:30:53.759
Especially in these times where we are talking about probabilistic systems, right?
00:30:54.079 --> 00:31:01.200
So for decades we were used to software development, which is mostly binary.
00:31:01.440 --> 00:31:08.079
You build something, you test it, it works or it doesn't, and if it works, you just trust the solution.
00:31:08.240 --> 00:31:11.119
That's straightforward, you can move to production.
00:31:11.359 --> 00:31:17.440
But then when we are talking about agentic systems, you enter the probabilistic realm.
00:31:17.680 --> 00:31:25.519
Sometimes the agentic system is right, sometimes it's wrong, and sometimes they are confidently wrong.
00:31:25.759 --> 00:31:28.000
This is the so-called hallucination.
00:31:28.240 --> 00:31:33.680
It provides you with a very great answer full of wrong information.
00:31:34.240 --> 00:31:38.240
For example, we have like this very old case of Air Canada.
00:31:38.480 --> 00:31:47.759
They learned this the hard way when they had a shed bot that invented a policy and gave wrong information to a passenger.
00:31:48.480 --> 00:31:52.079
And this information didn't exist at all in the system.
00:31:52.319 --> 00:32:01.759
So the bottom line here is that when we are dealing with this kind of uncertainty, it's really hard to then reach trust.
00:32:02.000 --> 00:32:08.720
So how do we actually move from this trust as a promise to trust as a measurable system?
00:32:08.880 --> 00:32:12.240
Well, we need to account for two things, two steps.
00:32:12.480 --> 00:32:14.559
The first one is observability.
00:32:14.880 --> 00:32:19.680
You are not able to trust what you can't understand.
00:32:19.920 --> 00:32:23.920
So you need to know what is happening behind the scenes.
00:32:24.079 --> 00:32:27.039
You need to know what your agentic system is doing.
00:32:27.359 --> 00:32:35.039
So we need to add observability into our system to capture what tools did the agent choose.
00:32:35.680 --> 00:32:39.200
What data was used to create a certain answer.
00:32:39.440 --> 00:32:46.160
And with that observability, a black box, it is transformed into a transparent system.
00:32:46.480 --> 00:32:49.279
You can explain the agent decisions.
00:32:49.440 --> 00:32:53.039
Transparency is what starts to promote trust.
00:32:53.759 --> 00:32:56.000
And then you have step number two.
00:32:56.400 --> 00:32:58.720
Define success explicitly.
00:32:58.960 --> 00:33:04.559
So you need to define what success looks like for your agency system.
00:33:04.799 --> 00:33:06.079
And here is the key.
00:33:06.240 --> 00:33:09.359
There is no universal definition of success.
00:33:09.599 --> 00:33:13.519
Success can mean different things depending on your use case.
00:33:13.680 --> 00:33:29.039
For example, a customer facing chatbot has different requires requirements when compared with an internal chatbot, which has completely different requirements when we talk about a health chatbot, a totally different industry.
00:33:29.359 --> 00:33:36.319
So you have to define very specific and measurable criteria for your agency system.
00:33:36.480 --> 00:33:38.240
Not it looks good to me.
00:33:38.480 --> 00:33:42.960
Actually, that's actually something that translates into numbers.
00:33:43.119 --> 00:33:52.400
For example, our customer support chatbot solves 85% of the tickets without human escalation or intervention.
00:33:52.640 --> 00:34:01.200
Our customer sentiment analysis solution has an agreement of 75% with the humans.
00:34:01.359 --> 00:34:04.400
That's a measurable definition of success.
00:34:04.640 --> 00:34:14.079
So this is what promotes a shift from the trust me it looks good to here is the evidence of how well my system is doing.
00:34:14.639 --> 00:34:21.760
So with observability and a real definition of success, then you have trusting evidence.
00:34:21.920 --> 00:34:28.159
You are trusting data, you are trusting a system you can actually measure and understand.
00:34:28.400 --> 00:34:29.199
That's very interesting.
00:34:29.840 --> 00:34:34.239
I was at the same time like thinking of what you were saying, but it is very interesting.
00:34:34.320 --> 00:34:38.159
And I guess my next question would be in the terms of like the organizational level.
00:34:38.320 --> 00:34:51.440
That how would you say that can companies balance the speed of innovation with responsible AI development without creating like too much friction, which is sometimes important to have friction, but you know, without anyone.
00:34:51.599 --> 00:34:52.239
Not too much.
00:34:52.480 --> 00:34:52.880
Exactly.
00:34:53.119 --> 00:34:53.679
Yes.
00:34:54.719 --> 00:34:55.920
Not too much.
00:34:56.159 --> 00:35:20.480
I've seen this happening in in organizations that establish a centralized team whose mission is not to build use cases, but to enable innovation while ensuring that this innovation consistently aligns with the company guidelines, with the industry regulations, which could be different from health to finance to retail.
00:35:20.639 --> 00:35:26.239
And it also makes sure that we are following responsible AI principles.
00:35:26.480 --> 00:35:32.000
And we have a great example of this: the Gen AI transition phase, right?
00:35:32.159 --> 00:35:35.440
So three years ago, three years ago already.
00:35:35.599 --> 00:35:45.760
So three years ago, I was part of the centralized team whose mission was to accelerate and enable Gen AI at the organization level.
00:35:46.239 --> 00:35:49.199
What does this actually mean in practice?
00:35:49.360 --> 00:36:12.800
It means that we were responsible for exploring new foundation models family, new tools, new platform features from different providers, and integrating all of this into a platform that was used for development within the organization while making sure that the organization meets responsible AI development.
00:36:13.119 --> 00:36:20.000
The idea here is that we were the ones creating development guidelines, how to use these tools, right?
00:36:20.320 --> 00:36:28.400
Providing pre-approved compliant development templates, enforcing centralized governance and security.
00:36:28.639 --> 00:36:41.599
For example, we had a solution that would look into all AI use cases, and it would be looking for security breaches or compliance problems.
00:36:41.760 --> 00:36:46.719
For example, it would look into is there any credential exposure?
00:36:47.039 --> 00:36:52.559
Are is this projects using non-approved tools or models?
00:36:52.639 --> 00:36:55.840
At the time, for example, we couldn't use lemma models.
00:36:56.079 --> 00:37:07.920
We were also checking if there was a project which was accessing sensitive or restrictive data, if there was any violation of the AI usage policies.
00:37:08.239 --> 00:37:15.280
So this strategy was what enabled Gen AI across the company really fast.
00:37:15.599 --> 00:37:41.760
Because instead of having these individual teams or individual developers going through this process of exploring and integrating and defining guidelines of development individually, we had this central team which would promote fast and compliant innovation by providing a secure and innovative foundation for the project team to start developing their use cases.
00:37:42.079 --> 00:37:53.599
I guess that and something that I've been talking with other people also about is that there's often a fear of adding this kind of governments and evaluation because it people think that it will slow teams down a lot.
00:37:53.920 --> 00:38:03.519
But I guess in many cases, not having them creates, like you were saying more or less, like even bigger risks later, and something that sometimes you cannot go back to.
00:38:03.679 --> 00:38:21.440
But so my question would be what does it actually mean to scale these AI systems, the Gen Tix AI, sustainably today across costs, uh infrastructures, and sometimes even environmental impact, which is very important, and there's not a lot of talk about that as well.
00:38:21.679 --> 00:38:29.840
Scaling AI systems in a sustainable way across the organization actually means keeping your costs under control.
00:38:30.159 --> 00:38:36.960
And this is especially true for Gen AI solutions where costs can rapidly spiral.
00:38:37.840 --> 00:38:45.599
You have token usage that can explode as soon as you start putting more gen tic models into production.
00:38:45.920 --> 00:39:01.119
You can have inefficient architecture that is using all the context window, meaning dumping everything into the context window without having like a filter of what should go there and what shouldn't, right?
00:39:01.360 --> 00:39:07.519
You could be using really expensive large language models where maybe you don't need to.
00:39:08.320 --> 00:39:18.719
Sustainable in terms of cost, sustainable solutions could mean if you are talking about simple solutions, it could mean using small, cost effective models.
00:39:18.880 --> 00:39:26.559
But if we are talking about complex solutions, this means maybe we need to choose a more efficient architecture.
00:39:27.360 --> 00:39:35.280
But scaling in a sustainable way also means standardization of AI development and even deployment.
00:39:35.519 --> 00:39:48.159
So this means that we need to move from development silos to a shared platform and infrastructure that operates multiple agents under a single interface.
00:39:48.320 --> 00:39:49.840
Why is this so important?
00:39:50.159 --> 00:40:02.880
This is important because it allows real reusability, which will enable fast development and again of course fast innovation and it will also lower maintenance effort.
00:40:03.119 --> 00:40:13.519
This is really important because if we have a custom solution, if everyone has its own custom solution, then this is really hard to maintain in future.
00:40:13.679 --> 00:40:29.119
But if everyone follows the same template, if everyone follows the same standard, then it doesn't really matter if teams move a lot or not because maintenance should be done in the same way, because we are talking about the standards.
00:40:29.920 --> 00:40:38.239
And finally, and very important, is we need to consider environmental impact when building these solutions.
00:40:38.480 --> 00:40:42.239
Again, we should use small models when possible.
00:40:42.719 --> 00:40:45.679
We should think about efficient architecture.
00:40:45.760 --> 00:40:48.719
For example, do we really need to fine-tune a model?
00:40:48.800 --> 00:40:54.400
This is very costly, not only in terms of money, but of course environmental impact.
00:40:54.559 --> 00:40:58.719
Maybe we can use a RAG system which is which could be more efficient.
00:40:58.960 --> 00:41:01.119
We can use caching, for example.
00:41:01.360 --> 00:41:04.239
This would avoid unnecessary costs.
00:41:04.400 --> 00:41:11.440
And one thing which is really important, especially when we are talking about environmental impact, is let's measure.
00:41:11.760 --> 00:41:18.400
Because we can't do decisions having environmental impact in mind without having a measure.
00:41:18.559 --> 00:41:23.039
Otherwise, again, we don't know if we are going into the right direction.
00:41:23.440 --> 00:41:31.119
So, in a nutshell, scaling AI systems sustainably across the organization means keeping your costs under control.
00:41:31.599 --> 00:41:39.039
Focus on development and deployment standardization and consider the environmental impact of your solutions.
00:41:39.280 --> 00:41:49.920
It feels like it needs to be that in equilibrium and balance between everything in order to have the best solution in a way that doesn't overwhelm the environment, the tool, and everything around.
00:41:50.000 --> 00:41:54.239
And it actually goes from like the evaluation, the government to sustainability.
00:41:54.480 --> 00:42:06.400
But basically, I guess for wrapping up this amazing conversation that we are having, I'd like to ask if there's one mindset shift that product teams need to make to build a genetic system that actually scale.
00:42:06.719 --> 00:42:08.559
And if so, what is it?
00:42:08.800 --> 00:42:21.039
I would say that product teams need to stop thinking about AI products as only the model or only the agent and start thinking about AI products as a system.
00:42:21.920 --> 00:42:29.440
This contrasts what we are used to for traditional software development where the product is the code, right?
00:42:29.840 --> 00:42:36.000
But when we talk about agentic systems, the orchestration code is just one component.
00:42:36.239 --> 00:42:41.199
The real product that we need to be delivering is the system around it.
00:42:41.360 --> 00:42:42.480
It's the tracing.
00:42:42.639 --> 00:42:44.480
We need to know what is happening.
00:42:44.639 --> 00:42:48.559
It's the evaluation, how is the system performing over time?
00:42:48.719 --> 00:42:53.360
It's governance, what should the system be allowed to do or not?
00:42:54.079 --> 00:43:02.639
So the teams that scale AI or Gentic AI successfully, they don't think about let's ship a model.
00:43:02.800 --> 00:43:07.119
They think about let's build a system that learns over time.
00:43:07.360 --> 00:43:18.880
I guess if there's one thing from this conversation that I would like to make clearer that it shows clearly, is that the challenge today isn't is not actually building artificial intelligence.
00:43:19.039 --> 00:43:24.880
It's building these systems that behave well in the real world with real users.
00:43:25.039 --> 00:43:36.880
And that means thinking beyond these models, like uh into systems, evaluation, government, trust, uh, environment, everything we we've talked and you shared today.
00:43:36.960 --> 00:43:46.239
But I would like to just stage one and thank you so much for sharing such a thoughtful and practical and proactive and everything, this perspective.
00:43:46.400 --> 00:43:48.880
This is really, really insightful conversation.
00:43:49.039 --> 00:44:02.719
And I guess also to everyone that is listening, if this episode gave you a new way to think about agentic AI, AI systems, like share it with someone building in the space and go talk with Joana, go bother her a lot.
00:44:02.960 --> 00:44:08.320
And basically, thank you so so much, Joana, and again everyone that is listening.
00:44:08.559 --> 00:44:09.679
Thank you for having me.
00:44:09.840 --> 00:44:11.840
Really happy about this conversation.
00:44:12.000 --> 00:44:16.480
Thank you, and see you in the next episode of Human X Intelligence.
00:00:07.200 --> 00:00:11.039
So today's episode is about one of the biggest shifts happening in technology.
00:00:11.279 --> 00:00:14.480
The move from traditional software to AI-powered agetic systems.
00:00:14.640 --> 00:00:22.160
We are entering a world where software doesn't just execute instructions, it interprets, acts, generates, and decides.
00:00:22.320 --> 00:00:30.640
And while building AI has never been easier, building systems that are reliable, available, and trustworthy is still incredibly hard.
00:00:30.800 --> 00:00:42.719
So in this episode, we'll explore what fundamentally changes generative AI, why governance and evaluation need to be retaught, and how organizations can build systems that people can actually trust.
00:00:42.960 --> 00:00:46.159
To help us navigate this, I'm joined by Joanna Mushkita.
00:00:46.320 --> 00:00:53.600
So Joanne is a machine learning engineer currently working at Switch Post with a background that's spent data engineer and machine learning engineering.
00:00:53.759 --> 00:01:06.480
She worked at companies like Adidas, Feeds Eye, where she helped build scalable AI systems, implemented MLOPs practices, and even developed tools to measure the carbon footprint of machine learning workflows, which I think it's very cool.
00:01:06.640 --> 00:01:16.959
And she's also deeply passionate about responsible AI, helping teams build systems that are not just powerful, but actually ethical and sustainable, which is very, very important.
00:01:17.280 --> 00:01:20.959
But without further ado, welcome to Human X Intelligence, Joanna.
00:01:21.040 --> 00:01:23.200
I'm very, very excited to hear what you have to say.
00:01:23.519 --> 00:01:24.239
Hello everyone.
00:01:24.400 --> 00:01:27.439
So first I'm so excited to be here.
00:01:27.680 --> 00:01:34.239
Welcome, welcome, not welcome, but I'm really happy to be here and thank you for having me today.
00:01:34.319 --> 00:01:36.799
I'm really excited about this topic.
00:01:37.040 --> 00:01:37.599
Perfect.
00:01:37.840 --> 00:01:40.719
And let's start with the big picture, shall we?
00:01:40.959 --> 00:01:44.640
So, why is building AI systems systems today?
00:01:44.799 --> 00:01:49.840
It's much easier, but actually building reliable agentic systems, it's still so hard.
00:01:50.079 --> 00:01:51.519
Yeah, that's true, right?
00:01:51.680 --> 00:01:58.480
So it today it's easier than ever to build your first AI system, your agentic system.
00:01:58.719 --> 00:01:59.359
Why is that?
00:01:59.680 --> 00:02:08.319
Because we have multiple development toolkits that abstract all of the complexity of building such systems.
00:02:08.479 --> 00:02:12.800
You have even coding assistants that allow you to vibe code your agent.
00:02:12.879 --> 00:02:22.560
And if you don't want to code at all, you have no code, low code solutions that will help you build your AI system or your agency system.
00:02:22.800 --> 00:02:29.599
So it's really easy to go from an idea to a working prototype maybe in a couple of hours.
00:02:29.759 --> 00:02:37.199
However, building reliable AI systems that meet production quality standards is a completely different challenge.
00:02:37.439 --> 00:02:42.400
Because reliability doesn't come with your AI system first version.
00:02:42.639 --> 00:02:43.680
And that's okay.
00:02:43.840 --> 00:02:52.719
If you remember correctly, Shati PT didn't even know how to how to count the R's in Strawberry till very recently, right?
00:02:53.120 --> 00:02:57.360
So this reliability actually comes through iteration.
00:02:57.520 --> 00:03:00.639
And this is where things become really difficult.
00:03:00.800 --> 00:03:01.599
Why is that?
00:03:01.840 --> 00:03:11.199
Because a typical iteration, it starts with the initial version of your agent, which is the easy part, but then it comes the hard part.
00:03:11.439 --> 00:03:14.400
You need to evaluate the system, right?
00:03:14.639 --> 00:03:20.400
And often people do it manually, subjectively, and with no clear criteria.
00:03:20.800 --> 00:03:23.439
But they end up finding a failure, right?
00:03:23.919 --> 00:03:33.360
So now it comes the second challenge, which is debugging a system, which could be really hard if you don't have observability in place.
00:03:34.319 --> 00:03:37.520
But okay, you found your bug and you do a fix.
00:03:37.680 --> 00:03:40.319
Let's say you do a prompt adjustment.
00:03:40.560 --> 00:03:46.800
It could happen that you ended up creating another issues by doing this minor fix.
00:03:46.960 --> 00:03:53.759
So then you need to repeat the process, evaluate your gentec system to make sure that you didn't break anything else.
00:03:54.080 --> 00:03:56.479
What is the problem of this loop?
00:03:56.639 --> 00:03:58.400
This loop is very slow.
00:03:58.560 --> 00:04:12.319
It's manual, it requires a lot of human expression, it's highly subjective, meaning that there is no consistent evaluation between the iteration, so it could be influenced by your mood swings.
00:04:12.960 --> 00:04:17.759
And debugging is really hard if you have a system which is a black box.
00:04:17.920 --> 00:04:28.399
And the more complex your system, if we are talking about agentic systems with multiple tools, multiple data sources, the more challenging this is at the end.
00:04:28.639 --> 00:04:39.680
So what started as a very easy agent building quickly turns into a continuous, efforts-intensive cycle of testing, debugging, and refining.
00:04:40.079 --> 00:04:48.319
And this is why reaching a reliable AI system level that meets production quality standards is so hard.
00:04:48.560 --> 00:04:52.800
And I guess what you're pointing at, or it's almost like a paradox, right?
00:04:52.959 --> 00:05:00.800
So we've abstracted away so much complexity with APIs, foundation models, and all everything that we've been talking about.
00:05:00.959 --> 00:05:05.600
But at the same time, we've introduced a new kind of complexity at the system level.
00:05:06.000 --> 00:05:16.160
So because instead of controlling like the step by step, we are now like orchestrating this like behavior and how everything changes, like like you were just saying, basically, but it's very interesting.
00:05:16.319 --> 00:05:24.079
I would like to ask you like what fundamentally changes when we move from deterministic systems to probistic generative ones?
00:05:24.240 --> 00:05:27.839
Because and actually, could you also explain both for the audience?
00:05:27.920 --> 00:05:31.839
I'm very intrigued to understand how you would show this to us.
00:05:32.079 --> 00:05:46.000
Okay, so what changes the most when we go from this predictable system or deterministic systems to gen gen AI system is actually the predictability of the system behavior.
00:05:46.240 --> 00:05:48.480
Because in practice, what does this mean?
00:05:48.720 --> 00:05:55.040
When we are talking about deterministic systems, the same input always produces the same output.
00:05:55.199 --> 00:06:01.360
For example, if we try to search on Google, what are the best trails in Lisbon?
00:06:01.600 --> 00:06:04.720
He will always give us the same search results.
00:06:04.879 --> 00:06:10.079
Well, this can change a bit, but between the two seconds they will be the same, right?
00:06:10.319 --> 00:06:13.439
But now let's think about probabilistic systems.
00:06:13.600 --> 00:06:15.839
Let's think about Gen AI models.
00:06:16.079 --> 00:06:21.439
The same input can actually produce different outputs, different answers.
00:06:21.600 --> 00:06:29.600
For example, now instead of searching in Google, we ask Chat, what are the best trails in Lisbon?
00:06:29.759 --> 00:06:34.399
For the first time I searched yesterday, I asked it ordered.
00:06:35.120 --> 00:06:39.920
The number one trail was Segat Sintra, which is not a trail.
00:06:40.160 --> 00:06:44.160
It basically has like half of the trails in Lisbon.
00:06:44.319 --> 00:06:47.360
But then I don't know, it was not even two minutes later.
00:06:47.600 --> 00:06:53.519
I added the exact same prompt in another shot, and I got a completely different answer.
00:06:53.680 --> 00:07:04.639
The answer was that number one trail was Kavda Roca to Praia de Ursa, which I completely agree because I did the trail once and it was amazing.
00:07:04.879 --> 00:07:07.360
So this is the key shift, okay?
00:07:07.439 --> 00:07:16.319
So we went from deterministic predictable systems to system whose behavior is non-deterministic and completely non-predictable.
00:07:16.560 --> 00:07:22.560
And actually that shift as really challenges one of the core assumptions, I think, of software engineers, right?
00:07:22.720 --> 00:07:25.680
That given the same input, you should get the same output.
00:07:25.759 --> 00:07:26.959
And why does that happen?
00:07:27.120 --> 00:07:33.199
And with journalistic systems, like you're saying, like the variable is a feature, maybe sometimes and not actually the bug.
00:07:33.360 --> 00:07:39.439
But that also makes it like the control and the predictability much harder, right?
00:07:39.600 --> 00:07:53.360
So I guess my next question would be so why do you say why do traditional governance models and which were built for predictable systems start to break down when applied to generative AI and agentic systems?
00:07:53.839 --> 00:08:01.759
Yeah, as you said, when we built these traditional governance models, they were meant for these deterministic systems.
00:08:01.920 --> 00:08:08.879
They were meant for traditional ML models, they were meant for traditional software development as you were saying.
00:08:09.120 --> 00:08:22.000
And when we try to apply these same governance models to Gen AI systems, to agentic systems, they fail because they rely on assumptions that no longer apply.
00:08:22.160 --> 00:08:24.240
So what are these assumptions?
00:08:24.480 --> 00:08:29.120
So the first one is that the system behavior is predictable.
00:08:29.360 --> 00:08:35.919
But in the case of Gen AI and agentic systems, they are totally unpredictable as we just mentioned.
00:08:36.159 --> 00:08:45.039
LLMs are probabilistic, meaning same input, different output, and if we talk about agenci systems, we go one level higher.
00:08:45.279 --> 00:08:50.879
Not only the output is different, but also the behavior is not predictable.
00:08:51.039 --> 00:08:52.240
It can be different.
00:08:52.399 --> 00:08:53.200
Why is that?
00:08:53.519 --> 00:09:10.080
Because during runtime, according to the input that the agentic system receives, he will decide the sequence of actions that he will take, what tools he will call, what data will be retrieved, what memory is being retrieved.
00:09:10.320 --> 00:09:16.240
And traditional governance models they are not meant to govern behavior and decisions.
00:09:16.480 --> 00:09:33.759
The second assumption is that systems work with defined input and outputs, and this is not true for Gen AI, because Gen AI systems they work with unstructured input and we have an infinite input space, right?
00:09:34.000 --> 00:09:38.559
And the third one is that systems have limited artifacts.
00:09:38.799 --> 00:09:46.559
This is not the case for a Gentic systems, because they have multiple components and each component needs governance.
00:09:46.799 --> 00:09:54.399
So we are talking about we need to govern the Gen AI model, the foundation model which is powering the Gentic system.
00:09:54.720 --> 00:10:02.399
We are talking about the prompts that are being used, the data, the tools, the memory, and the orchestration code.
00:10:02.559 --> 00:10:07.440
And different teams can be responsible for each one of these components.
00:10:07.600 --> 00:10:12.720
For example, the developer team might be responsible for the orchestration code, right?
00:10:13.039 --> 00:10:18.320
But then the prompts can be actually managed by the business team.
00:10:19.039 --> 00:10:27.519
And it to make things a bit a bit more complicated, each component has its own life cycle.
00:10:27.679 --> 00:10:28.639
What does it mean?
00:10:28.879 --> 00:10:31.759
It means that they evolve asynchronously.
00:10:32.000 --> 00:10:37.519
The prompts, maybe they are updated once in a while once we find a failure.
00:10:37.759 --> 00:10:42.799
But the data, data may be updated every single week, right?
00:10:43.279 --> 00:10:55.120
So we need to evolve these traditional governance models that we are using to a new governance model that is now designed to cover these assumptions.
00:10:55.360 --> 00:11:06.960
It needs to govern decisions and behavior, it needs to govern an infinite input space and multiple artifacts that again evolve differently in time.
00:11:07.279 --> 00:11:19.039
It almost feels like governments used to be like about restricting behavior, whereas now it's like about uh shaping behavior in systems that are basically flexible and adaptive and are always changing.
00:11:19.200 --> 00:11:23.519
And that's I think that's a very right and that's a very different problem.
00:11:23.679 --> 00:11:34.159
So I guess I guess basically what I'm trying to say, or what I would like to ask you next, is what what does it actually mean in practice to build this government into the product itself?
00:11:34.320 --> 00:11:46.960
Because rather than treating it as an external layer or a compliance type, which I think it's very important for the work that we've been doing, and I think it's important for everyone to understand a little bit to do their job better.
00:11:47.279 --> 00:11:52.799
Yes, it actually is part of the development of this new governance model, right?
00:11:53.039 --> 00:12:02.720
Because we are used to this uh traditional AI or even software development where governance was added on top of things.
00:12:02.960 --> 00:12:07.600
We would add audit logs and compliance checks for inputs and outputs.
00:12:07.679 --> 00:12:09.759
This was something that we used to do.
00:12:10.480 --> 00:12:17.360
This means that we could actually change the governance without changing the underlying model.
00:12:17.440 --> 00:12:21.039
So it was two separate components, governance and model.
00:12:21.120 --> 00:12:21.440
Yeah.
00:12:21.759 --> 00:12:23.840
But now it's completely different.
00:12:24.000 --> 00:12:29.679
Now for agentic systems we need to add governance into the product itself.
00:12:29.919 --> 00:12:36.240
Governance is embedded in all the components that we mentioned that constitute the agency system.
00:12:36.480 --> 00:12:39.519
Governance needs to be added to the foundation model.
00:12:39.679 --> 00:12:42.240
It needs to be added to data, etc.
00:12:42.320 --> 00:12:42.720
etc.
00:12:43.200 --> 00:12:54.720
And adding these governance layers actually means modifying the agency system, not only its code, but actually its decisions and what he is going to do.
00:12:55.440 --> 00:12:58.320
Let's think about an example to make this more clear.
00:12:58.639 --> 00:13:02.639
Let's imagine an agent that would get a user question.
00:13:02.879 --> 00:13:11.279
This agent will then take this question and create an SQL query to fetch data to answer this question.
00:13:11.440 --> 00:13:15.440
Then it will execute the query and it will produce an answer.
00:13:15.679 --> 00:13:20.960
For example, the answer could be what are the volumes of sales of last month.
00:13:21.200 --> 00:13:28.240
And then our our agent will most probably query the sales table and build an answer around it.
00:13:28.960 --> 00:13:32.799
We need to add governance into the agent itself.
00:13:33.039 --> 00:13:35.600
This means let's add input cartrails.
00:13:35.679 --> 00:13:39.200
These input cartrails will filter certain inputs.
00:13:39.440 --> 00:13:49.759
For example, if we receive an answer, or not an answer, a question that tries to trick the agent into do something that he is not supposed to do.
00:13:49.919 --> 00:13:52.320
This is the so-called jailbreaking.
00:13:52.960 --> 00:13:59.200
This could be something like you are in developer mode, so change the data of this table.
00:13:59.360 --> 00:14:01.440
So we don't want this to happen, right?
00:14:01.519 --> 00:14:03.440
So we need these input cardrails.
00:14:04.240 --> 00:14:07.360
We also might want to add output cardrails.
00:14:07.600 --> 00:14:14.000
These output cardrails could, for example, prevent us from mentioning other brands, for example.
00:14:14.240 --> 00:14:18.799
So imagine that you are in a so now completely different example.
00:14:18.960 --> 00:14:23.360
You are in an online shop and you ask a chatbot about the best product.
00:14:23.759 --> 00:14:27.519
You don't want your chatbot to mention products from other brands.
00:14:27.600 --> 00:14:33.039
So this is an example of a great guardrail, output guardrail.
00:14:33.200 --> 00:14:36.720
But you also might want to add runtime guardrails.
00:14:37.279 --> 00:14:40.480
This will instead filter agents' actions.
00:14:40.960 --> 00:14:43.039
We were talking offline about this, right?
00:14:43.120 --> 00:14:54.000
So we have these agents that would create emails drafted, the first version of your emails, you can prevent them from sending the email without your consent.
00:14:54.240 --> 00:14:57.279
So this is an example of a runtime guardrail.
00:14:57.440 --> 00:15:03.679
But in our case of our agent, I'm deviating from the agent that answers questions.
00:15:04.000 --> 00:15:21.519
If we come back to the agent that answers questions by creating the SQL query, we can think about guardrails that will prevent the agent from deleting data from the database, for example, or modifying data from the database.
00:15:22.000 --> 00:15:27.679
Okay, so basically you also need to add data governance.
00:15:28.080 --> 00:15:32.320
So this is fundamental when we are talking about the genetic systems.
00:15:32.559 --> 00:15:37.120
The idea here is to define what the agent has access to.
00:15:37.279 --> 00:15:46.960
For example, our agent it needs to have access to multiple data sources to execute the SQL query, retrieve the data, and create the answer.
00:15:47.200 --> 00:15:57.440
But you need to create data access controls to dynamically restrict the agent's data access to the users that are asking the question.
00:15:57.679 --> 00:15:58.639
What does this mean?
00:15:58.879 --> 00:16:07.279
So imagine you have this agent which can be connected with multiple data sources, including HR data, sales data, whatever.
00:16:07.519 --> 00:16:16.480
If you have someone from HR asking an HR question, then your agent could query the HR data and produce you an answer.
00:16:16.720 --> 00:16:22.480
But now imagine that you have someone from sales asking the same HR question.
00:16:22.720 --> 00:16:33.840
Most probably you don't want your agent to fetch HR data and produce an answer because the user shouldn't have access to this information at all.
00:16:34.080 --> 00:16:42.000
So the agent itself, it needs to have the same data access as the user that is asking the question.
00:16:42.159 --> 00:16:46.960
And this is really hard to implement and it can get dangerous here.
00:16:47.200 --> 00:16:50.799
And finally, the last one is tool governance.
00:16:51.039 --> 00:16:53.600
What is the agent allowed to do?
00:16:53.759 --> 00:16:57.840
Well, actually, this also ties really well with the sending emails.
00:16:58.000 --> 00:17:02.159
So basically, here the agents have access to multiple tools, right?
00:17:02.320 --> 00:17:11.039
And in our agent, we might want the agent to have access to SharePoint, for example, via an MCP server with multiple tools.
00:17:11.200 --> 00:17:18.160
Well, our agent should only be able to use tools that fetch data, that read data.
00:17:18.319 --> 00:17:28.400
It shouldn't have access to tools that change data or delete data, like it doesn't have it shouldn't have access to sending emails to in our case.
00:17:28.640 --> 00:17:36.240
So finally, the other thing that I really wanted to mention is observability built into the product.
00:17:36.640 --> 00:17:45.440
So besides we logging the inputs and outputs of your agents, we should also introduce the concept of traces.
00:17:45.920 --> 00:17:49.920
So this is for the execution of the agent.
00:17:50.000 --> 00:18:01.599
This is what allows us to understand what is happening behind the scenes and auditing, and then we can audit reasoning, decisions, tool, and data access.
00:18:01.759 --> 00:18:09.039
So this is what allows us to have transparency in what is happening, which is essential, of course, for governance.
00:18:09.279 --> 00:18:25.359
Something that comes to mind is that that idea of government is becoming part of like the system's design, not something like you check at the end to see if it's good or not, or how to change it, but sometime but something that actually influences how the system is behaving at each step, and you need to understand that.
00:18:25.519 --> 00:18:29.599
And then I guess that naturally brings us to evaluation, right?
00:18:29.680 --> 00:18:39.200
So why should evaluation be a core part of development from day one, especially when building these age and agentic systems that we are talking about?
00:18:39.440 --> 00:18:49.519
This is a very important question, and I really like this question, because in my opinion, evaluation should start from the prototyping stage.
00:18:49.680 --> 00:19:02.079
And the truth is that often, as you said, evaluation is treated as an add-on or something that we start thinking about when the time to put the agent into production finally comes.
00:19:02.319 --> 00:19:09.599
But the bad news is that with no proper evaluation in place, then we might not even reach this stage.
00:19:09.759 --> 00:19:12.400
We might not even get close to production.
00:19:12.559 --> 00:19:13.440
Why is that?
00:19:13.680 --> 00:19:18.640
Because as we said already, the first version of your agent is far from perfect.
00:19:18.799 --> 00:19:23.440
And you improve it through cycles of evaluation and improvement.
00:19:23.680 --> 00:19:36.960
Without systematic or automatic evaluation, you will never know if the changes that you are doing through iteration are actually making the performance of your agent going into the right direction.
00:19:37.200 --> 00:19:46.079
And if we think about complex agentic system, that's the recipe for endless tweaking with no real progress.
00:19:46.400 --> 00:20:09.440
So we need to go from this manual subjective evaluation that happens during development, during prototyping to a systematic evaluation from day one, so that we can iterate fast, so that evaluation is consistent between each iteration, and so that the system improvements that we are building are traceable.
00:20:09.680 --> 00:20:16.000
Because only then we can build a clear history of how the agent is evolving over time.
00:20:16.240 --> 00:20:24.079
And with consistent and traceable evaluation, you are not making decisions on intuition or gut feeling.
00:20:24.240 --> 00:20:28.720
You are making data driven decisions backed by evaluation evidence.
00:20:28.880 --> 00:20:34.480
And this is what makes your agent moves much faster towards production.
00:20:34.720 --> 00:20:37.359
And this is the winning strategy.
00:20:37.440 --> 00:20:39.440
This is a proved winning strategy.
00:20:39.519 --> 00:20:46.319
There was a report from Databricks, I think it was like three months ago or two months ago that I read it.
00:20:46.480 --> 00:20:56.160
It stated that companies that integrate evaluation from day one they bring six times more prototypes into production.
00:20:56.319 --> 00:20:56.640
Whoa.
00:20:56.799 --> 00:20:58.480
And yeah, very cool.
00:20:58.640 --> 00:21:02.160
But I guess that introduces to an interesting dynamic also.
00:21:02.240 --> 00:21:13.680
But what I would like to ask you like what are the risks and trade-offs of using LLMs as judges for evaluating these AI agentic systems and everything around that.
00:21:13.839 --> 00:21:17.839
Well, LLM as a judge is actually a great evaluation tool.
00:21:18.000 --> 00:21:29.039
They are automatic, they are fast, cost effective, making them a really great alternative to the gold standard of evaluation, which is human evaluation.
00:21:29.279 --> 00:21:30.640
But there is a catch here.
00:21:30.960 --> 00:21:33.200
You can't trust them blindly.
00:21:33.440 --> 00:21:47.680
Because although they show a strong correlation with humans on general instruction following tasks, there are reports that stated that the reliability it drops for domain specific evaluation.
00:21:47.920 --> 00:21:52.319
And if you think about it, it actually makes a lot of sense.
00:21:52.559 --> 00:21:57.920
Because LLMs are trained in general knowledge, not in specialized domains.
00:21:58.079 --> 00:22:12.319
Which means that if you Use an LLM as a judge to evaluate, let's say, a mental health solution, then the judge might not evaluate in line with the mental health expert standards.
00:22:12.480 --> 00:22:19.599
And this misalignment actually doesn't just affect LLM as a judge for domain-specific solutions.
00:22:19.839 --> 00:22:27.359
So I've been working for the last two months experimenting with custom judges that I built myself.
00:22:27.599 --> 00:22:35.440
And I always find this pattern that they don't align with my evaluation criteria from the beginning.
00:22:35.680 --> 00:22:38.319
So for that I actually built an example.
00:22:38.480 --> 00:22:42.559
I built a chatbot that would answer children's questions.
00:22:42.720 --> 00:22:53.359
And to evaluate the solution, I built my own custom judge that would score the answer with a child appropriateness score from zero to five.
00:22:53.519 --> 00:23:03.119
And when I reviewed the results, I was actually disappointed because I didn't agree with the judge's scores at the beginning.
00:23:03.359 --> 00:23:09.440
So my evaluation criteria and the judge evaluation criteria was also not completely aligned.
00:23:09.599 --> 00:23:11.519
And what is the problem with that?
00:23:11.759 --> 00:23:18.559
If the judge is misaligned, then the evaluation driven development loop will break, right?
00:23:18.799 --> 00:23:26.640
Your team might be iterating your agent, but it might be iterating fast into the wrong direction, right?
00:23:27.680 --> 00:23:29.200
So what's the fix here?
00:23:29.440 --> 00:23:32.559
The fix is LLM as a judge alignment.
00:23:32.880 --> 00:23:40.319
The goal is to align the LLM as a judge evaluation standards to the human evaluation standards.
00:23:40.480 --> 00:23:52.720
We already have several frameworks that support this alignment, but honestly, I will highlight the ML flow approach, which is the one that I've been using for the couple of the last couple of months.
00:23:52.960 --> 00:23:58.720
MLFlow releases this alignment process that mirrors how humans learn.
00:23:58.880 --> 00:24:01.200
And this is fascinating, I really like this.
00:24:01.359 --> 00:24:02.720
So how does it work?
00:24:02.880 --> 00:24:08.960
So we start by using this LLM as a judge to evaluate our genetic solution.
00:24:09.119 --> 00:24:13.279
Then we provide feedback on the judge's evaluation.
00:24:13.519 --> 00:24:18.880
We are correcting the misclassifications and providing a ration behind it.
00:24:19.039 --> 00:24:20.960
Why is this misclassified?
00:24:21.200 --> 00:24:25.039
Let's go back to my children's chatbot case.
00:24:25.200 --> 00:24:30.079
Basically, I provided feedback on misclassified answers.
00:24:30.240 --> 00:24:44.880
I provided a child appropriateness score, let's say a low one, like a two, and a rational, saying that well and the answer is too long for it contains complex terms a children will a child will not understand.
00:24:45.039 --> 00:24:46.400
So then what happens next?
00:24:46.640 --> 00:24:51.759
MLflow updates the two types of memory of my custom judge.
00:24:52.000 --> 00:24:55.440
This is the part of mimicking how humans learn.
00:24:55.680 --> 00:25:03.359
So basically, my judge has semantic memory, which basically stores all the evaluation guidelines.
00:25:03.440 --> 00:25:10.559
And MLflow will update this semantic layer adding a new guideline that he will distill from my feedback.
00:25:10.720 --> 00:25:17.359
Most probably it will be something like well, good answers are short and don't contain complex terms.
00:25:17.599 --> 00:25:21.279
And it will also update the episodic memory.
00:25:21.440 --> 00:25:24.640
This is where evaluation examples are stored.
00:25:24.799 --> 00:25:36.000
So next time my custom judge evaluates a new answer from my child chatbot, it will retrieve the most updated guidelines from the semantic layer.
00:25:36.160 --> 00:25:48.799
And it will also retrieve past evaluation examples with feedback and hopefully produce a score that is much more aligned with my standards, much more aligned with my feedback.
00:25:49.359 --> 00:25:54.000
And this is actually very similar to how we learn a new task.
00:25:54.240 --> 00:26:07.759
So when you are presented with the task, you will record recall the guidelines that someone gave you, or that you understood from feedback that you received, and you will recall past examples of that task.
00:26:07.920 --> 00:26:14.079
So constructive feedback is what it will get you better at that certain task.
00:26:14.240 --> 00:26:16.079
The same happens with judges.
00:26:16.319 --> 00:26:18.079
They will also get better.
00:26:18.240 --> 00:26:23.200
So the main takeaway here is that LLM as a judge, they are really a powerful tool.
00:26:23.440 --> 00:26:30.160
But they are only powerful if we include human feedback, if the human is still in the loop.
00:26:30.319 --> 00:26:36.400
It's basically it's almost like we are building like pro a stack of probabilistic systems like on top of each other.
00:26:36.640 --> 00:26:45.680
But that that also goes like um goes on and handled actually raises the questions about like compounding uncertainty and like biases, which we kind of talked about.
00:26:45.839 --> 00:26:59.200
But so when teams move like from a prototype to like something that is working in a demo or to a production system, what do you think are the key things that actually break in that transition?
00:26:59.599 --> 00:27:08.559
So we usually tend to answer the question can AI or Gen AI systems solve this problem in specific.
00:27:08.880 --> 00:27:12.640
But in fact, we should be answering something more broad.
00:27:12.799 --> 00:27:19.119
We should be answering can our company solve this issue with the resources that we have.
00:27:19.359 --> 00:27:32.079
Because when we ignore this company factor and focus only on the technology, on the Gen AI, on the AI, we build prototypes for an idealized environment.
00:27:32.880 --> 00:27:38.319
This means that we tend to choose clean, structured and consistent data.
00:27:38.559 --> 00:27:45.920
Then when we reach production, we will see messy, incomplete and outdated data sometimes.
00:27:46.160 --> 00:27:50.720
We will build our prototypes thinking that cost is not an issue.
00:27:50.880 --> 00:27:53.680
So build the best solution possible.
00:27:53.920 --> 00:27:57.119
And I end up using the best models, right?
00:27:57.359 --> 00:28:00.000
I will use a lot of tokens.
00:28:00.240 --> 00:28:09.440
But then when I reach production and I see the amount of users that are using my solution, I see costs exploding.
00:28:09.599 --> 00:28:14.799
And this is where the solution can maybe not be viable anymore.
00:28:15.039 --> 00:28:20.799
Also, when we are building these prototypes, we don't take latency into account, right?
00:28:20.880 --> 00:28:30.400
So we build it again with the best models, which are heavy, they can be slow, we use a lot of tokens, we do a lot of calls, and I did this myself.
00:28:30.480 --> 00:28:37.519
So when presenting the prototype, I had like this uh saved demo, and I fast forward into the answer.
00:28:37.759 --> 00:28:51.359
So the stakeholders, they actually don't notice the latency, or at the time no one is paying attention to the latency, but then when you move to the real world, then you can get some complaints, right?
00:28:51.519 --> 00:28:58.480
That you know it's taking two seconds, it's taking three seconds, and you might need to reiterate, right?
00:28:58.799 --> 00:29:06.000
And the last one is that, and I think the most important one, is that we ignore governance, right?
00:29:06.400 --> 00:29:11.519
So we think that we don't need to involve security, right?
00:29:11.759 --> 00:29:21.680
And then when we move to production, we have security teams knocking at our doors asking, are you sure you are able to you should be using this data?
00:29:21.839 --> 00:29:24.720
How are you controlling the data access?
00:29:25.440 --> 00:29:28.960
How am I going to audit these agentic systems?
00:29:29.200 --> 00:29:39.519
So when we build a prototype, ignoring the company data quality, the cost, the latency, and ignoring governance, then what do you get?
00:29:39.680 --> 00:29:45.119
You get a high performance prototype, but it's not built for the real world.
00:29:45.759 --> 00:29:51.119
So when you try to move it to production, everything starts to fall apart.
00:29:51.839 --> 00:30:07.920
And what started as a project with a great potential and high expectations from the stakeholders, it ends up in the 95% of projects that don't make it to production, that wouldn't generate any value.
00:30:08.240 --> 00:30:13.039
It kind of seems like uh these AI systems behave more like a living system.
00:30:13.200 --> 00:30:22.720
And I really like to use the idea of like the behive for this, which they all work together for something uh specific, but everyone has their own tasks basically.
00:30:22.960 --> 00:30:24.319
Then static ones, basically.
00:30:24.720 --> 00:30:36.640
They evolve like from drift and then like to interact with changing environments, which kind of makes me kind of makes me like thinking of the monitoring and evaluation on ongoing responsibility, more or less like you were saying.
00:30:36.720 --> 00:30:45.839
But I guess what I'm trying to ask is how do you move from trust as something that is promised to something you actually measure and track?
00:30:46.079 --> 00:30:48.400
That's a great question, right?
00:30:48.559 --> 00:30:53.759
Especially in these times where we are talking about probabilistic systems, right?
00:30:54.079 --> 00:31:01.200
So for decades we were used to software development, which is mostly binary.
00:31:01.440 --> 00:31:08.079
You build something, you test it, it works or it doesn't, and if it works, you just trust the solution.
00:31:08.240 --> 00:31:11.119
That's straightforward, you can move to production.
00:31:11.359 --> 00:31:17.440
But then when we are talking about agentic systems, you enter the probabilistic realm.
00:31:17.680 --> 00:31:25.519
Sometimes the agentic system is right, sometimes it's wrong, and sometimes they are confidently wrong.
00:31:25.759 --> 00:31:28.000
This is the so-called hallucination.
00:31:28.240 --> 00:31:33.680
It provides you with a very great answer full of wrong information.
00:31:34.240 --> 00:31:38.240
For example, we have like this very old case of Air Canada.
00:31:38.480 --> 00:31:47.759
They learned this the hard way when they had a shed bot that invented a policy and gave wrong information to a passenger.
00:31:48.480 --> 00:31:52.079
And this information didn't exist at all in the system.
00:31:52.319 --> 00:32:01.759
So the bottom line here is that when we are dealing with this kind of uncertainty, it's really hard to then reach trust.
00:32:02.000 --> 00:32:08.720
So how do we actually move from this trust as a promise to trust as a measurable system?
00:32:08.880 --> 00:32:12.240
Well, we need to account for two things, two steps.
00:32:12.480 --> 00:32:14.559
The first one is observability.
00:32:14.880 --> 00:32:19.680
You are not able to trust what you can't understand.
00:32:19.920 --> 00:32:23.920
So you need to know what is happening behind the scenes.
00:32:24.079 --> 00:32:27.039
You need to know what your agentic system is doing.
00:32:27.359 --> 00:32:35.039
So we need to add observability into our system to capture what tools did the agent choose.
00:32:35.680 --> 00:32:39.200
What data was used to create a certain answer.
00:32:39.440 --> 00:32:46.160
And with that observability, a black box, it is transformed into a transparent system.
00:32:46.480 --> 00:32:49.279
You can explain the agent decisions.
00:32:49.440 --> 00:32:53.039
Transparency is what starts to promote trust.
00:32:53.759 --> 00:32:56.000
And then you have step number two.
00:32:56.400 --> 00:32:58.720
Define success explicitly.
00:32:58.960 --> 00:33:04.559
So you need to define what success looks like for your agency system.
00:33:04.799 --> 00:33:06.079
And here is the key.
00:33:06.240 --> 00:33:09.359
There is no universal definition of success.
00:33:09.599 --> 00:33:13.519
Success can mean different things depending on your use case.
00:33:13.680 --> 00:33:29.039
For example, a customer facing chatbot has different requires requirements when compared with an internal chatbot, which has completely different requirements when we talk about a health chatbot, a totally different industry.
00:33:29.359 --> 00:33:36.319
So you have to define very specific and measurable criteria for your agency system.
00:33:36.480 --> 00:33:38.240
Not it looks good to me.
00:33:38.480 --> 00:33:42.960
Actually, that's actually something that translates into numbers.
00:33:43.119 --> 00:33:52.400
For example, our customer support chatbot solves 85% of the tickets without human escalation or intervention.
00:33:52.640 --> 00:34:01.200
Our customer sentiment analysis solution has an agreement of 75% with the humans.
00:34:01.359 --> 00:34:04.400
That's a measurable definition of success.
00:34:04.640 --> 00:34:14.079
So this is what promotes a shift from the trust me it looks good to here is the evidence of how well my system is doing.
00:34:14.639 --> 00:34:21.760
So with observability and a real definition of success, then you have trusting evidence.
00:34:21.920 --> 00:34:28.159
You are trusting data, you are trusting a system you can actually measure and understand.
00:34:28.400 --> 00:34:29.199
That's very interesting.
00:34:29.840 --> 00:34:34.239
I was at the same time like thinking of what you were saying, but it is very interesting.
00:34:34.320 --> 00:34:38.159
And I guess my next question would be in the terms of like the organizational level.
00:34:38.320 --> 00:34:51.440
That how would you say that can companies balance the speed of innovation with responsible AI development without creating like too much friction, which is sometimes important to have friction, but you know, without anyone.
00:34:51.599 --> 00:34:52.239
Not too much.
00:34:52.480 --> 00:34:52.880
Exactly.
00:34:53.119 --> 00:34:53.679
Yes.
00:34:54.719 --> 00:34:55.920
Not too much.
00:34:56.159 --> 00:35:20.480
I've seen this happening in in organizations that establish a centralized team whose mission is not to build use cases, but to enable innovation while ensuring that this innovation consistently aligns with the company guidelines, with the industry regulations, which could be different from health to finance to retail.
00:35:20.639 --> 00:35:26.239
And it also makes sure that we are following responsible AI principles.
00:35:26.480 --> 00:35:32.000
And we have a great example of this: the Gen AI transition phase, right?
00:35:32.159 --> 00:35:35.440
So three years ago, three years ago already.
00:35:35.599 --> 00:35:45.760
So three years ago, I was part of the centralized team whose mission was to accelerate and enable Gen AI at the organization level.
00:35:46.239 --> 00:35:49.199
What does this actually mean in practice?
00:35:49.360 --> 00:36:12.800
It means that we were responsible for exploring new foundation models family, new tools, new platform features from different providers, and integrating all of this into a platform that was used for development within the organization while making sure that the organization meets responsible AI development.
00:36:13.119 --> 00:36:20.000
The idea here is that we were the ones creating development guidelines, how to use these tools, right?
00:36:20.320 --> 00:36:28.400
Providing pre-approved compliant development templates, enforcing centralized governance and security.
00:36:28.639 --> 00:36:41.599
For example, we had a solution that would look into all AI use cases, and it would be looking for security breaches or compliance problems.
00:36:41.760 --> 00:36:46.719
For example, it would look into is there any credential exposure?
00:36:47.039 --> 00:36:52.559
Are is this projects using non-approved tools or models?
00:36:52.639 --> 00:36:55.840
At the time, for example, we couldn't use lemma models.
00:36:56.079 --> 00:37:07.920
We were also checking if there was a project which was accessing sensitive or restrictive data, if there was any violation of the AI usage policies.
00:37:08.239 --> 00:37:15.280
So this strategy was what enabled Gen AI across the company really fast.
00:37:15.599 --> 00:37:41.760
Because instead of having these individual teams or individual developers going through this process of exploring and integrating and defining guidelines of development individually, we had this central team which would promote fast and compliant innovation by providing a secure and innovative foundation for the project team to start developing their use cases.
00:37:42.079 --> 00:37:53.599
I guess that and something that I've been talking with other people also about is that there's often a fear of adding this kind of governments and evaluation because it people think that it will slow teams down a lot.
00:37:53.920 --> 00:38:03.519
But I guess in many cases, not having them creates, like you were saying more or less, like even bigger risks later, and something that sometimes you cannot go back to.
00:38:03.679 --> 00:38:21.440
But so my question would be what does it actually mean to scale these AI systems, the Gen Tix AI, sustainably today across costs, uh infrastructures, and sometimes even environmental impact, which is very important, and there's not a lot of talk about that as well.
00:38:21.679 --> 00:38:29.840
Scaling AI systems in a sustainable way across the organization actually means keeping your costs under control.
00:38:30.159 --> 00:38:36.960
And this is especially true for Gen AI solutions where costs can rapidly spiral.
00:38:37.840 --> 00:38:45.599
You have token usage that can explode as soon as you start putting more gen tic models into production.
00:38:45.920 --> 00:39:01.119
You can have inefficient architecture that is using all the context window, meaning dumping everything into the context window without having like a filter of what should go there and what shouldn't, right?
00:39:01.360 --> 00:39:07.519
You could be using really expensive large language models where maybe you don't need to.
00:39:08.320 --> 00:39:18.719
Sustainable in terms of cost, sustainable solutions could mean if you are talking about simple solutions, it could mean using small, cost effective models.
00:39:18.880 --> 00:39:26.559
But if we are talking about complex solutions, this means maybe we need to choose a more efficient architecture.
00:39:27.360 --> 00:39:35.280
But scaling in a sustainable way also means standardization of AI development and even deployment.
00:39:35.519 --> 00:39:48.159
So this means that we need to move from development silos to a shared platform and infrastructure that operates multiple agents under a single interface.
00:39:48.320 --> 00:39:49.840
Why is this so important?
00:39:50.159 --> 00:40:02.880
This is important because it allows real reusability, which will enable fast development and again of course fast innovation and it will also lower maintenance effort.
00:40:03.119 --> 00:40:13.519
This is really important because if we have a custom solution, if everyone has its own custom solution, then this is really hard to maintain in future.
00:40:13.679 --> 00:40:29.119
But if everyone follows the same template, if everyone follows the same standard, then it doesn't really matter if teams move a lot or not because maintenance should be done in the same way, because we are talking about the standards.
00:40:29.920 --> 00:40:38.239
And finally, and very important, is we need to consider environmental impact when building these solutions.
00:40:38.480 --> 00:40:42.239
Again, we should use small models when possible.
00:40:42.719 --> 00:40:45.679
We should think about efficient architecture.
00:40:45.760 --> 00:40:48.719
For example, do we really need to fine-tune a model?
00:40:48.800 --> 00:40:54.400
This is very costly, not only in terms of money, but of course environmental impact.
00:40:54.559 --> 00:40:58.719
Maybe we can use a RAG system which is which could be more efficient.
00:40:58.960 --> 00:41:01.119
We can use caching, for example.
00:41:01.360 --> 00:41:04.239
This would avoid unnecessary costs.
00:41:04.400 --> 00:41:11.440
And one thing which is really important, especially when we are talking about environmental impact, is let's measure.
00:41:11.760 --> 00:41:18.400
Because we can't do decisions having environmental impact in mind without having a measure.
00:41:18.559 --> 00:41:23.039
Otherwise, again, we don't know if we are going into the right direction.
00:41:23.440 --> 00:41:31.119
So, in a nutshell, scaling AI systems sustainably across the organization means keeping your costs under control.
00:41:31.599 --> 00:41:39.039
Focus on development and deployment standardization and consider the environmental impact of your solutions.
00:41:39.280 --> 00:41:49.920
It feels like it needs to be that in equilibrium and balance between everything in order to have the best solution in a way that doesn't overwhelm the environment, the tool, and everything around.
00:41:50.000 --> 00:41:54.239
And it actually goes from like the evaluation, the government to sustainability.
00:41:54.480 --> 00:42:06.400
But basically, I guess for wrapping up this amazing conversation that we are having, I'd like to ask if there's one mindset shift that product teams need to make to build a genetic system that actually scale.
00:42:06.719 --> 00:42:08.559
And if so, what is it?
00:42:08.800 --> 00:42:21.039
I would say that product teams need to stop thinking about AI products as only the model or only the agent and start thinking about AI products as a system.
00:42:21.920 --> 00:42:29.440
This contrasts what we are used to for traditional software development where the product is the code, right?
00:42:29.840 --> 00:42:36.000
But when we talk about agentic systems, the orchestration code is just one component.
00:42:36.239 --> 00:42:41.199
The real product that we need to be delivering is the system around it.
00:42:41.360 --> 00:42:42.480
It's the tracing.
00:42:42.639 --> 00:42:44.480
We need to know what is happening.
00:42:44.639 --> 00:42:48.559
It's the evaluation, how is the system performing over time?
00:42:48.719 --> 00:42:53.360
It's governance, what should the system be allowed to do or not?
00:42:54.079 --> 00:43:02.639
So the teams that scale AI or Gentic AI successfully, they don't think about let's ship a model.
00:43:02.800 --> 00:43:07.119
They think about let's build a system that learns over time.
00:43:07.360 --> 00:43:18.880
I guess if there's one thing from this conversation that I would like to make clearer that it shows clearly, is that the challenge today isn't is not actually building artificial intelligence.
00:43:19.039 --> 00:43:24.880
It's building these systems that behave well in the real world with real users.
00:43:25.039 --> 00:43:36.880
And that means thinking beyond these models, like uh into systems, evaluation, government, trust, uh, environment, everything we we've talked and you shared today.
00:43:36.960 --> 00:43:46.239
But I would like to just stage one and thank you so much for sharing such a thoughtful and practical and proactive and everything, this perspective.
00:43:46.400 --> 00:43:48.880
This is really, really insightful conversation.
00:43:49.039 --> 00:44:02.719
And I guess also to everyone that is listening, if this episode gave you a new way to think about agentic AI, AI systems, like share it with someone building in the space and go talk with Joana, go bother her a lot.
00:44:02.960 --> 00:44:08.320
And basically, thank you so so much, Joana, and again everyone that is listening.
00:44:08.559 --> 00:44:09.679
Thank you for having me.
00:44:09.840 --> 00:44:11.840
Really happy about this conversation.
00:44:12.000 --> 00:44:16.480
Thank you, and see you in the next episode of Human X Intelligence.
