TRASCRIZIONE 🔗
WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 07/14/2026 18:29:56
Duration: 2305.985
Channels: 1
1
00:00:06.319 --> 00:00:07.120
So
2
00:00:07.120 --> 00:00:08.080
welcome,
3
00:00:08.400 --> 00:00:12.160
Hina, and thank you very much for taking the time to talk with me today.
4
00:00:12.480 --> 00:00:17.735
Yeah. Thanks, Barry, for having me on your podcast. I'm very excited to be here. And,
5
00:00:18.135 --> 00:00:20.775
yeah, looking forward to it, to to our conversation.
6
00:00:21.255 --> 00:00:23.975
Yeah. I it's it's a good topic. Definitely.
7
00:00:25.300 --> 00:00:28.980
So perhaps you can introduce yourself to the listeners.
8
00:00:29.619 --> 00:00:34.980
Yes, of course. So hi everyone. I'm Hina, a Cloud Solution Architect
9
00:00:35.059 --> 00:00:38.579
at Microsoft focusing on AI and
10
00:00:38.820 --> 00:00:40.500
the use of AI, so applications.
11
00:00:42.175 --> 00:00:48.734
I've been with Microsoft for around two years now, and I've been working in IT for around thirteen plus years,
12
00:00:49.055 --> 00:00:51.695
all of it in the AI and machine learning space.
13
00:00:52.495 --> 00:00:55.055
So, yeah, I'm excited to be here and
14
00:00:55.430 --> 00:00:57.030
share my experience
15
00:00:57.910 --> 00:00:59.110
on this channel.
16
00:00:59.190 --> 00:01:07.030
Oh, excellent. So you've all you've been working in machine learning for a long time then already. What did that look like, let's call it,
17
00:01:07.270 --> 00:01:08.310
ten years ago?
18
00:01:08.870 --> 00:01:09.590
Well
19
00:01:10.235 --> 00:01:12.715
so I remember the time that was the whole
20
00:01:13.034 --> 00:01:18.795
Hadoop and big data boom at that time. Got it? So I remember being
21
00:01:19.275 --> 00:01:22.795
a big data engineer supporting a team of data scientists,
22
00:01:23.835 --> 00:01:25.195
building pipelines,
23
00:01:25.275 --> 00:01:29.100
know, doing also data engineering work, getting data ready
24
00:01:29.420 --> 00:01:33.420
so that they could build models on top of a big insights cluster,
25
00:01:33.500 --> 00:01:34.860
I remember at that time,
26
00:01:35.259 --> 00:01:44.555
with IBM. So then, yeah, so my role was, at that time, ML engineer was not a formal role, but I was doing somewhat a data engineer and a machine learning engineer,
27
00:01:44.795 --> 00:01:53.595
helping data scientists get the data, get the pipelines, do the CICD and DevOps so that the models, the feature engineering that they do, the models that they build,
28
00:01:54.640 --> 00:01:55.680
the training,
29
00:01:55.760 --> 00:01:57.920
the inferencing retraining happens
30
00:01:58.960 --> 00:02:07.200
in a standard way on the platform with the right processes. So that was what I was focusing on there. And it has evolved quite a lot since the onset of
31
00:02:07.945 --> 00:02:10.345
generative AI since the past few years.
32
00:02:10.905 --> 00:02:12.985
But traditional machine learning still
33
00:02:13.225 --> 00:02:16.185
is quite close
Load full transcript
to my heart because that's where I started.
34
00:02:16.505 --> 00:02:20.905
Yeah. It it's strange if you think back about it because it was the big data boom,
35
00:02:21.360 --> 00:02:29.040
and then, you know, the slogan was, there's like gold in your data and there's so much unused potential in your data. Go use your data.
36
00:02:29.280 --> 00:02:33.040
Where where did that go? Do what do you see now with customers?
37
00:02:33.694 --> 00:02:37.135
Have they embraced that boom, or is it just fizzled out?
38
00:02:37.375 --> 00:02:38.815
No. To be honest,
39
00:02:39.694 --> 00:02:42.975
somehow with this whole wave of AI,
40
00:02:44.575 --> 00:02:47.135
data has taken a backseat,
41
00:02:47.135 --> 00:02:47.534
and
42
00:02:48.110 --> 00:02:52.830
it's surprising to see at enterprises and customers because everyone is so
43
00:02:53.550 --> 00:02:59.070
caught up in the hype of AI. We wanna do something with AI. We want to embed it into our systems that they've forgotten to
44
00:02:59.790 --> 00:03:10.995
basically look at the underlying data that will support your AI. So the garbage in garbage out is becoming more and more prominent now because if your data is not
45
00:03:11.875 --> 00:03:17.795
if you don't have good data, then your AI systems are they're not magic. They're also gonna be as,
46
00:03:19.360 --> 00:03:28.560
yeah, as good as the data you feed into it. Right? So I do see a shift from we want to do something with AI, but not looking at if our underlying
47
00:03:29.680 --> 00:03:37.495
setup, our underlying basics and foundation is set for scaling the AI. So that's what I see at customers.
48
00:03:37.815 --> 00:03:38.775
Yeah. So
49
00:03:39.415 --> 00:03:41.495
they had that data scientists,
50
00:03:41.495 --> 00:03:45.015
ML engineers at some point, but they kind of never really
51
00:03:45.735 --> 00:03:48.855
did the thing with big data, right? They didn't really clean their data,
52
00:03:49.880 --> 00:03:58.280
a little bit, but not not really. Yeah. Yeah. Not really. Not really. I there were some very good use cases, though. Risk modeling.
53
00:03:58.360 --> 00:04:03.960
I think that is still a very important machine learning use case that is being implemented in production at many customers.
54
00:04:04.665 --> 00:04:05.545
That
55
00:04:05.545 --> 00:04:06.265
is still,
56
00:04:06.905 --> 00:04:09.865
like, quite there with machine learning. But now with
57
00:04:10.025 --> 00:04:11.225
generative AI,
58
00:04:11.465 --> 00:04:15.065
I think that traditional machine learning has taken a backseat, and it's more about
59
00:04:15.385 --> 00:04:23.950
finding the latest model on Hugging Face and see what we do with it rather than, you know, going back to the roots and finding the features and doing feature engineering,
60
00:04:24.670 --> 00:04:27.710
trying out different algorithms to come up with the specific
61
00:04:28.750 --> 00:04:31.870
machine learning model. Yep. That has taken a backseat, I would say.
62
00:04:32.625 --> 00:04:43.985
Yeah. That's I I've never really thought about that from this angle. So, yeah, we used to do machine learning, and then the basis was your own data to find patterns and to do things with that data, find gold,
63
00:04:44.465 --> 00:04:46.385
and that required lots of expertise.
64
00:04:46.740 --> 00:04:56.260
And then basically, you're training your own model. It's kind of what's happening there. And so I guess maybe now because those models from the hyperscalers are so extremely interesting.
65
00:04:56.340 --> 00:05:04.075
It's so fun to talk to them and you get all this magic stuff back. But they do it not on your data per se, or you can do that.
66
00:05:05.115 --> 00:05:11.115
But yet, why do you think that's fizzled out or or now shifted towards what we're doing now? It's kinda strange.
67
00:05:13.115 --> 00:05:17.035
I think it has a lot to do with the hype, but also because
68
00:05:17.960 --> 00:05:19.000
data science
69
00:05:19.160 --> 00:05:25.640
or being a data scientist, a good data scientist requires a certain set of skills, certain set of
70
00:05:26.040 --> 00:05:26.840
learnings,
71
00:05:27.160 --> 00:05:40.485
which is not very easy to find, that skill set. It has always been a challenge. I think a couple of years back, there was a saying, right, data scientist is the sexiest job of 2020 even in that period, 2019 to 2020, that big years. Right?
72
00:05:41.525 --> 00:05:42.325
And that
73
00:05:42.870 --> 00:05:46.870
led to organizations to think that we need to have the specialized
74
00:05:46.870 --> 00:05:47.510
people
75
00:05:48.070 --> 00:05:51.430
in order to build our own models and do machine learning.
76
00:05:51.590 --> 00:05:54.390
And IT not being the core business of
77
00:05:54.550 --> 00:05:56.710
many of the organizations,
78
00:05:57.485 --> 00:05:59.805
which are which are not prominently in IT,
79
00:06:00.285 --> 00:06:06.925
this was a a difficult task. They did not wanna invest so much in setting up a team and setting up the right foundations,
80
00:06:06.925 --> 00:06:10.125
ML ML's life cycle in setting up the entire,
81
00:06:11.580 --> 00:06:17.260
yeah, the entire skill set and the people that are needed to build machine learning models or to build
82
00:06:17.580 --> 00:06:19.740
the data science muscle in the organization.
83
00:06:20.140 --> 00:06:21.740
And with generative AI,
84
00:06:21.980 --> 00:06:29.005
that part is quite taken care of. So then they're like, hey. It's easier to just take a pretrained model,
85
00:06:29.245 --> 00:06:34.125
maybe augment it with our industry data, do rag, or maybe fine tune it in some cases,
86
00:06:34.525 --> 00:06:37.645
and leverage it for the use cases rather than start from scratch.
87
00:06:37.805 --> 00:06:39.965
So that has been a mindset shift
88
00:06:40.750 --> 00:06:42.349
that organizations
89
00:06:42.590 --> 00:06:46.030
that led to not many organizations completely embracing
90
00:06:46.189 --> 00:06:48.669
or having a data science muscle within them.
91
00:06:48.990 --> 00:06:55.885
Yeah. You you don't need to be an expert anymore. You're just you're prompt engineer. Exactly.
92
00:06:57.565 --> 00:07:02.685
Strange. Okay. We'll we'll circle back to this probably later on as well. I
93
00:07:03.085 --> 00:07:05.965
know you as one of the experts in a
94
00:07:06.285 --> 00:07:10.180
product suite or a set or solution, whatever you want to call it, called
95
00:07:10.500 --> 00:07:11.940
Microsoft Foundry,
96
00:07:11.940 --> 00:07:16.979
previously called AI Foundry. Perhaps it had a different name even before that. I don't know. Can
97
00:07:18.099 --> 00:07:21.300
you maybe let's dive into that. What is that even?
98
00:07:23.345 --> 00:07:35.905
Perfect. So so Microsoft Foundry. So I have been working with Foundry so that you said, indeed, it was Azure AI Foundry. It was Azure AI Studio even before that. Mhmm. And it was even before that cognitive services.
99
00:07:36.430 --> 00:07:37.470
Right
100
00:07:37.470 --> 00:07:42.670
now, the foundry that we see, it has gone through multiple iterations, also a maturity path
101
00:07:43.070 --> 00:07:47.870
within the product group or within Microsoft as well. Microsoft Foundry is a complete
102
00:07:48.405 --> 00:07:52.485
shop as it is today, is a complete shop for building
103
00:07:52.725 --> 00:07:56.565
AI apps and agents. It started out as being
104
00:07:56.725 --> 00:08:04.245
this provider of this entire model catalog, 11,000 plus models that you can leverage from multiple providers
105
00:08:04.460 --> 00:08:06.140
and you can build apps with.
106
00:08:06.460 --> 00:08:10.940
But now it is also evolving into having evaluation capabilities,
107
00:08:10.940 --> 00:08:12.140
tracing capabilities,
108
00:08:12.140 --> 00:08:24.005
building that agent harness around agentic AI systems that you can also not just build using the models within Foundry, but also leverage the hosting platform within Foundry with hosted agents and
109
00:08:24.405 --> 00:08:33.845
utilize the evaluations that come out of the box, tracing that comes out of the box, fine tuning, and all these capabilities, making it like a completely robust platform
110
00:08:34.020 --> 00:08:36.100
for building apps, agents,
111
00:08:36.580 --> 00:08:40.820
custom agents as well. So you don't just have to rely purely on
112
00:08:40.980 --> 00:08:41.940
Microsoft
113
00:08:42.100 --> 00:08:45.620
agent framework or or have any lock in to the platform.
114
00:08:45.940 --> 00:08:50.705
But just giving you that flexibility to build apps agents within Foundry
115
00:08:51.105 --> 00:08:54.385
or using just models from Foundry and any frameworks outside.
116
00:08:54.464 --> 00:08:55.425
So complete
117
00:08:55.665 --> 00:08:59.665
AI tool kit, I would say. That's what Foundry has evolved into.
118
00:09:00.065 --> 00:09:05.800
Yeah. Yeah. And it's huge now. So I remember the days when I used to use cognitive
119
00:09:05.800 --> 00:09:08.520
services for super specific things like
120
00:09:09.480 --> 00:09:14.280
speech to text or something. Or what else was there? There were several.
121
00:09:14.760 --> 00:09:19.775
I even made a course about them. And then they evolved into more generative
122
00:09:19.775 --> 00:09:26.575
AI, and you needed to fill out a form and wait, and then maybe you got access to them or something. It was kind quite gated.
123
00:09:26.895 --> 00:09:36.430
And now we have all this stuff with all these things you just mentioned, which is a lot of stuff. So I guess maybe the core capability is that you can get
124
00:09:36.830 --> 00:09:41.470
your models from a central place. Like so from your Azure subscription,
125
00:09:41.630 --> 00:09:42.670
can get GPT,
126
00:09:43.205 --> 00:09:45.845
Opus. I don't know what all sorts of models.
127
00:09:46.405 --> 00:09:51.925
Yeah. Indeed. Indeed. So that has that is the core indeed, the core capability
128
00:09:51.925 --> 00:10:06.250
from Foundry, the model catalog. The diverse model catalog where you find models from multiple providers. You have Anthropic models, which can be pro first party, which have been announced now recently. They're first party from Azure. So then you don't have to
129
00:10:06.570 --> 00:10:10.410
worry about the data processing. You have those guarantees about data processing,
130
00:10:11.855 --> 00:10:20.335
data retention from Microsoft. So you have those guarantees by using the models from Anthropic. Right? You also have models from MICE, so Microsoft AI.
131
00:10:20.495 --> 00:10:22.815
We have models from
132
00:10:24.710 --> 00:10:29.910
Azure OpenAI, of course. We have many other providers. I think there are 11,000 plus in the catalog.
133
00:10:30.149 --> 00:10:33.110
And also, they are categorized into the different
134
00:10:34.790 --> 00:10:39.270
use that you wanna use them for, usages, maybe for reasoning or you want to do
135
00:10:40.015 --> 00:10:42.895
document processing or you want to do voice agents.
136
00:10:43.135 --> 00:10:46.895
So they so it's and it also has leaderboard. It also has
137
00:10:47.935 --> 00:10:48.975
ways to benchmark
138
00:10:49.215 --> 00:10:55.560
public benchmark benchmarking has been done on public data for those models. So you have a possibility of going and checking,
139
00:10:55.800 --> 00:11:12.305
right, which model suits my use case the best and trying it out within your use case. I think that is the core capability within Foundry. And then once you identify the model, you can also then evolve into pending agents and having that entire agent development tool kit around it also embedded in Foundry
140
00:11:12.785 --> 00:11:14.865
to leverage. Yeah. Okay.
141
00:11:15.025 --> 00:11:22.160
Okay. So let's let's walk through like I'm gonna use this. Right? So I deploy Foundry in Azure.
142
00:11:22.320 --> 00:11:26.960
I pay for it or I don't don't yet. And then I pick my favorite model.
143
00:11:27.280 --> 00:11:38.185
Let's call it Fable for instance, and I agree to all the terms. It's fine. I am willing to pay for this Cause I want to use this in my application. I have a, let's call it a dot net application.
144
00:11:38.665 --> 00:11:40.905
Now what? What do I do?
145
00:11:42.025 --> 00:11:52.850
So, personally for Fable, that is restricted access. As as you also mentioned, we know why. But that is still possible. So if you get access to the model, you can build your application
146
00:11:53.010 --> 00:11:53.810
locally
147
00:11:54.130 --> 00:11:57.570
in framework of your choice. You chose the languages.net.
148
00:11:58.050 --> 00:11:59.490
For agent orchestration,
149
00:11:59.490 --> 00:12:03.925
you want to maybe use Lama Index or you want to use Microsoft Microsoft Agent Framework
150
00:12:04.005 --> 00:12:07.445
to build. So agent is nothing but model
151
00:12:07.605 --> 00:12:08.405
plus
152
00:12:08.885 --> 00:12:09.365
a personality.
153
00:12:09.925 --> 00:12:12.245
So that's your system prompt telling your
154
00:12:12.405 --> 00:12:32.295
AI system what it is supposed to do. Right? Because a large language model could be used for anything, but you say maybe be a health assistant or be a personal coach Mhmm. Be a content writer for me. Right? So you give a personality with the system front, and you give it arms and legs with the with the way of giving it tools and data.
155
00:12:32.535 --> 00:12:45.095
Right? So that's how your entire so your LNM evolves into being an agent in AI system. So within Foundry, what you can do is once you select your model, you can choose a framework of choice, start building locally,
156
00:12:45.510 --> 00:12:51.430
and then you can push publish the agent in Foundry. Either it could be a prompt agent, which is used for
157
00:12:51.670 --> 00:12:52.470
simple
158
00:12:53.750 --> 00:12:55.510
Q and A kind of use cases
159
00:12:55.670 --> 00:13:10.654
or simple chat chatbot agents that you wanna build. You give it tools via the Foundry Catalog, the tool catalog that is there. You can connect to xMCP servers, external tools. You can connect to data sources, AI search, MongoDB,
160
00:13:11.214 --> 00:13:12.014
Databricks even,
161
00:13:12.920 --> 00:13:17.720
and you can give it action. So that is tools for maybe go and update a
162
00:13:18.040 --> 00:13:22.360
record in SAP or create an incident in ServiceNow.
163
00:13:22.519 --> 00:13:40.455
So that is how you can attach those hands, legs, create a complete AI system, and have it hosted on Foundry. That will be a URL that you can then call, an API that you can then call, and your AI system based on the prompt that it receives will perform the actions that it you have defined for it to do.
164
00:13:41.894 --> 00:13:45.980
Okay. So I have this model, right, Fable or whatever.
165
00:13:46.380 --> 00:13:48.860
And then you said, then it turns into an agent
166
00:13:49.100 --> 00:13:58.705
because then it has a system prompt and
all these other things. What if I just wanna use it and prompt it? Like, I wanna send a prompt to Fable and just use what comes back.
167
00:13:59.105 --> 00:14:02.385
That is also possible. You can just play with it in the playground.
168
00:14:02.945 --> 00:14:22.075
There's a Foundry playground that is a UI based interface. Just select the model and start chatting with it. If you don't give it a system prompt, like, you don't give it a personality, it's a generic model. It was trained on public data, so it can do anything that's possible in that dataset. If you ask it, give me the weather weather right now.
169
00:14:22.875 --> 00:14:25.435
It won't be able to give you the weather today because
170
00:14:25.755 --> 00:14:28.715
it does not have that tool to find the
171
00:14:28.955 --> 00:14:30.155
weather that is,
172
00:14:30.635 --> 00:14:38.360
right now. It has data. It has knowledge only from the data that it was trained on. Right? So what you can do, you can just prompt with it. You can
173
00:14:38.680 --> 00:14:39.320
send
174
00:14:39.880 --> 00:15:04.690
throw prompts, say, hey. Can you suggest me how I can write this text better? How can you summarize this document for me? All of this you can do. But in order for it to be used for specific use case, it's better or or it is advised to give it a personality via system prompt and attach those actions and tools to make it very specific for the business case that you're trying to solve rather than have it generic. Ah,
175
00:15:05.810 --> 00:15:06.370
okay.
176
00:15:06.769 --> 00:15:13.730
And then, yeah, like you said, you can also just and that's kind of a click away, right? You can turn it into an agent within Foundry,
177
00:15:13.889 --> 00:15:19.665
give it tools and access to other systems and stuff, and then expose that through
178
00:15:19.825 --> 00:15:20.705
API.
179
00:15:20.785 --> 00:15:27.185
Are there other ways to expose it? Yes. You can publish it to Teams. There is a direct integration with M365.
180
00:15:27.265 --> 00:15:29.185
You can publish it on Slack.
181
00:15:29.905 --> 00:15:33.185
You can publish that agent in a website or embedded in a website.
182
00:15:34.360 --> 00:15:35.880
Yeah. You can host it.
183
00:15:36.519 --> 00:15:42.200
Even if it is hosted on Foundry, you can have an application that is built
184
00:15:43.000 --> 00:15:55.945
outside of Foundry, hosted on Container Apps, web application, you name it. Right? And it just calls the agent or AI system whenever it needs to within Foundry. So that is also possible. So there are multiple ways of exposing
185
00:15:56.505 --> 00:16:00.345
this agent AI system to the external VoIP.
186
00:16:00.585 --> 00:16:04.110
Right. Because an agent is kinda an app application
187
00:16:04.110 --> 00:16:06.510
that runs. Right? It needs a runtime,
188
00:16:06.750 --> 00:16:19.325
a compute somewhere, and we can do that in Foundry as well. Right? I don't have to do that myself. No. You don't have to. So then
about the possibilities. If you do it within Foundry, then natively, it uses Microsoft Agent Framework under the hood as orchestration.
189
00:16:19.485 --> 00:16:32.830
It has an agent runtime for, like, for hosted agents, you can actually host on it. It uses its agent runtime. And then you can just have an API that you can call, throw your prompts based on, the personality that you have given it, it will respond.
190
00:16:33.070 --> 00:16:38.590
Or you can build just use the models from Foundry and build your own app externally,
191
00:16:38.670 --> 00:16:43.790
hosting it in your platform of choice. It could also be Azure. It could be AWS. It could be on prem.
192
00:16:44.195 --> 00:16:52.995
Just leveraging the models from Foundry and then accessing the AI system or the agent outside Foundry via the APIs that or where it is hosted.
193
00:16:53.315 --> 00:17:05.900
Yeah. Yeah. So that is very cool because it has everything kinda built into it that you need, right? So the the compute that you need to to run it is called agent service, right? Foundry agent service.
194
00:17:06.060 --> 00:17:09.419
Yes. And as I understand, you can run these
195
00:17:09.900 --> 00:17:21.334
clicks together agents like low code agents that just runs there. Or, you know, like you just mentioned, you build it yourself with Microsoft Agent Framework or something else just to and just run it there.
196
00:17:21.735 --> 00:17:23.815
Now, this agent service,
197
00:17:23.975 --> 00:17:26.934
what type of compute does it use, and can I control that?
198
00:17:27.790 --> 00:17:31.070
So that's a very good question. So under the hood,
199
00:17:31.310 --> 00:17:32.269
it uses
200
00:17:32.510 --> 00:17:33.710
Azure Container
201
00:17:33.790 --> 00:17:34.669
Instances.
202
00:17:35.630 --> 00:17:42.110
They are single isolated ones that are from a from a tenant that is managed completely on Microsoft side.
203
00:17:42.350 --> 00:17:44.504
So I would say you don't have
204
00:17:45.304 --> 00:17:51.144
you cannot configure it completely custom yourself. That has been taken care of by
205
00:17:51.304 --> 00:17:52.744
Foundry itself.
206
00:17:52.985 --> 00:18:01.600
Under the hood, you have the possibility of deciding how many agents you can host there. The scaling and everything is taken care of by the platform itself.
207
00:18:02.160 --> 00:18:04.080
Right? If you really want
208
00:18:04.240 --> 00:18:05.919
complete customizations,
209
00:18:05.919 --> 00:18:15.895
hey, I wanna decide how many nodes my cluster has, my hosting infrastructure, then I would suggest you could also just deploy your own hosting platform like Azure Container
210
00:18:16.375 --> 00:18:18.934
Apps or Kubernetes cluster
211
00:18:18.935 --> 00:18:20.294
or web apps
212
00:18:20.775 --> 00:18:32.889
application in yeah. Web applications that you can host. Right? Or you can host it externally from Foundry as well. But within Foundry, it is managed. And the reason it is done that way is there are many customers who prefer
213
00:18:33.130 --> 00:18:35.929
not getting into the infrastructure management,
214
00:18:36.169 --> 00:18:40.570
the versioning, the version updates, the scaling, the HADR
215
00:18:42.075 --> 00:18:45.914
requirements from a hosting platform. So they prefer to host it on Foundry,
216
00:18:46.235 --> 00:18:52.554
and that's where the agent service comes into play, where you can host it on this managed infrastructure within Foundry.
217
00:18:53.195 --> 00:18:53.515
Right.
218
00:18:54.340 --> 00:18:57.860
But I can control some things, right, with capability
219
00:18:57.860 --> 00:19:00.179
hosts, for instance, on the Foundry
220
00:19:00.820 --> 00:19:07.779
instance or on a project where I can say, you know, for instance, my AI search, so one of the tools that you mentioned,
221
00:19:08.164 --> 00:19:14.404
that is this one that I already have, for instance, pointed to this thing. For sure. Yes. So that is the enterprise
222
00:19:14.885 --> 00:19:20.004
integration part of Foundry. So definitely, even if the infrastructure is managed by Microsoft,
223
00:19:20.245 --> 00:19:25.860
you have a possibility of injecting it into a network that is within your organization.
224
00:19:25.860 --> 00:19:34.899
So you can bring in your own virtual network and have the agents that you build within Foundry hosted within that virtual network. So they will also have a private IP,
225
00:19:35.059 --> 00:19:38.019
which is within the range that you have set out within your organization.
226
00:19:38.535 --> 00:19:41.415
And then you can have everything with private networking.
227
00:19:41.495 --> 00:19:46.695
All the integrations to AI search, for example, can be set up as ingress, egress on
228
00:19:47.095 --> 00:19:53.815
that subnet where the agents are hosted. So we're going a bit into networking here. But from enterprise integration perspective,
229
00:19:54.420 --> 00:19:59.220
even if the infrastructure is managed by Microsoft, it is injected into your own environment.
230
00:19:59.220 --> 00:20:02.420
So it's not like that the agents are running on
231
00:20:03.780 --> 00:20:06.260
Microsoft network. It is running within your enterprise.
232
00:20:07.255 --> 00:20:09.894
Yeah. So so for for details,
233
00:20:10.215 --> 00:20:15.494
the compute is still managed by Microsoft. So the the container instances.
234
00:20:15.815 --> 00:20:17.575
But AI search,
235
00:20:17.655 --> 00:20:18.455
storage,
236
00:20:18.534 --> 00:20:25.690
and also Cosmos DB, which it uses to track conversations and things like that. Right? You can bring your own of those threads. Indeed.
237
00:20:26.169 --> 00:20:34.514
So you can bring your own network as well as you can bring your own thread store. So with Cosmos DB, AI search for any
238
00:20:34.514 --> 00:20:37.954
data that you is sent into the agents or the agent needs
239
00:20:38.514 --> 00:20:41.714
as a knowledge source. That as well you can bring your own
240
00:20:42.434 --> 00:20:43.235
resources,
241
00:20:43.235 --> 00:20:51.629
which are which are within your Azure tenant. Yeah. Yeah. And can I mix and match those where I say, you know what? I just wanna bring my own storage and
242
00:20:51.630 --> 00:20:56.029
Cosmos and AI search. You guys manage that. Yes. That is also possible.
243
00:20:56.270 --> 00:20:59.390
That is also possible. If you don't provide your own resources,
244
00:20:59.794 --> 00:21:03.634
Microsoft will deploy or Foundry will deploy itself
245
00:21:04.115 --> 00:21:16.360
the resources that it needs. So these three are what it needs to function. The agent service needs it to function. It will deploy it on Microsoft side. So if you don't provide your own, it will anyways be deployed under the hood on Microsoft side.
246
00:21:16.600 --> 00:21:25.720
Yeah. So so also in a hybrid approach where I say just the just Cosmos DB is the thing I manage and the rest you guys take care of. Yes. That is also possible.
247
00:21:26.200 --> 00:21:26.520
Okay.
248
00:21:27.154 --> 00:21:34.034
Okay, so that's cool. That's a bit of a nitty gritty thing there with agent service, but at least you can then run your agent there.
249
00:21:34.514 --> 00:21:38.994
You also, in the beginning, we had lots of terms, evaluations,
250
00:21:38.995 --> 00:21:41.010
you mentioned. Yep. What is that?
251
00:21:41.490 --> 00:21:42.690
So evaluations
252
00:21:42.690 --> 00:21:43.410
is
253
00:21:43.490 --> 00:21:44.290
based
254
00:21:44.690 --> 00:21:49.330
to determine if your agentic AI system is performing
255
00:21:49.810 --> 00:21:51.010
the tasks
256
00:21:52.595 --> 00:22:05.554
the task that it set out for, so the the personality that you have given it. So you told it to be a health adviser, for example. But it is someone asked it, hey. Can you give me advice about the financial markets
257
00:22:05.635 --> 00:22:07.875
in The Netherlands right now? Mhmm. And it responded.
258
00:22:08.490 --> 00:22:13.690
Right? So then in this case, it is not adhering to the task you set it out for.
259
00:22:14.010 --> 00:22:17.850
So then the task adherence of your AI system, it's queued,
260
00:22:18.090 --> 00:22:30.465
and you want to monitor it. You want your AI systems to do exactly what you built them for. And that's where evaluations come in. So you decide certain criteria categories. So you decide task adherence being one.
261
00:22:30.785 --> 00:22:31.665
Groundedness,
262
00:22:31.665 --> 00:22:48.119
is it responding based on the data that it has, the knowledge that it has, or is it just responding based on public knowledge? Or is it hallucinating even? So all these criteria as you set out based on the use case which you have built the AI system for, and you can set up an evaluation pipeline
263
00:22:48.120 --> 00:22:49.399
where every time
264
00:22:49.640 --> 00:23:05.775
it can be offline evaluations. Every time a new version of the AI system goes in, the evaluate evaluation pipeline runs to
check if your system is still performing to the threshold that has been set, to the percentage that it has been set for the different categories of evaluators
265
00:23:05.775 --> 00:23:07.695
or it could be online evaluation.
266
00:23:08.095 --> 00:23:09.455
When your system is being used,
267
00:23:09.980 --> 00:23:15.340
every prompt that is sent and every response that is given out by your system is evaluated
268
00:23:15.340 --> 00:23:20.140
to see if it is adhering to the task, if it is grounded, if it is hallucinating,
269
00:23:20.140 --> 00:23:28.304
if it is it has any security issues, if it has any errors. Right? So all this can be stepped up within Foundry in an automated way.
270
00:23:29.505 --> 00:23:30.945
That's very cool. So
271
00:23:31.424 --> 00:23:35.504
offline evaluations are kinda like unit tests for a model.
272
00:23:35.904 --> 00:23:39.825
Yeah. You can definitely, definitely think about it that way. Yeah. Okay.
273
00:23:41.160 --> 00:23:42.120
So,
274
00:23:42.120 --> 00:23:43.720
you know, we want quality,
275
00:23:43.720 --> 00:23:45.240
so that's an evaluation.
276
00:23:45.240 --> 00:24:01.924
See if this thing because that's so difficult with a nondeterministic thing. Right? What happens when you update to a new version of the model? Is it still the same or who knows? Exactly. Okay. That's good. So there are many other things that people wanna measure and control.
277
00:24:02.245 --> 00:24:04.325
One of them is costs,
278
00:24:04.725 --> 00:24:06.565
and that's always a very difficult one
279
00:24:07.530 --> 00:24:08.170
Because
280
00:24:08.570 --> 00:24:11.530
what I see a lot as well is that people use
281
00:24:11.690 --> 00:24:13.929
foundries or models come out of foundry
282
00:24:14.250 --> 00:24:15.929
and then they put for instance
283
00:24:16.250 --> 00:24:21.370
API management as your API management in front of that like as a gateway.
284
00:24:21.664 --> 00:24:30.065
So applications connect to API management and an API management can expose models from, for instance, Foundry and or somewhere else.
285
00:24:30.625 --> 00:24:35.904
And then API management has all sorts of rules and things like this many tokens.
286
00:24:37.730 --> 00:24:39.970
But in this scenario,
287
00:24:39.970 --> 00:24:42.690
or if you just connect directly to Foundry,
288
00:24:43.010 --> 00:24:45.570
how do you measure the costs
289
00:24:45.649 --> 00:24:56.755
that, for instance, an agent makes or your application makes, right? If I write it myself. How do you control that? Because let's say I deploy this thing to the Internet and people just start using it. Yeah.
290
00:24:57.075 --> 00:25:01.715
Maybe it's popular and, you know, how do I not wake up to a huge bill?
291
00:25:02.035 --> 00:25:06.279
Yeah. So so the course are one of the main
292
00:25:07.800 --> 00:25:11.640
use cases for having an AI gateway in front of Foundry.
293
00:25:11.800 --> 00:25:16.600
So there is some capabilities within. So if you just
have Foundry and you deploy models,
294
00:25:16.975 --> 00:25:19.374
you also there's a concept called projects.
295
00:25:19.534 --> 00:25:27.374
So within one Foundry, you deploy project for each business or each use case, depending on how your organization is set in.
296
00:25:27.855 --> 00:25:37.740
At the project level, you have the possibility of setting rate limits. So, hey, you can use because all model usage within Foundry is based is charged on tokens,
297
00:25:37.980 --> 00:25:41.340
tokens per minute. So imagine a token to be four
298
00:25:41.900 --> 00:25:43.420
roughly four characters
299
00:25:43.795 --> 00:25:49.794
contributes to one token. Right? And characters include blank spaces, letters, numbers, everything.
300
00:25:50.035 --> 00:25:55.075
So the tokens per minute is the criteria in which the costs are calculated.
301
00:25:55.395 --> 00:26:02.600
So how Foundry does is at the project level, you can decide, hey, this project can give be given a code of 10,000
302
00:26:02.600 --> 00:26:04.280
per minute and,
303
00:26:04.920 --> 00:26:07.400
a 1,000,000 tokens for a month.
304
00:26:08.280 --> 00:26:10.040
That is how you set budgets.
305
00:26:10.120 --> 00:26:14.280
So then when you bid agents within your project, they consume
306
00:26:14.135 --> 00:26:17.735
the code the tokens from these budgets that have been set up.
307
00:26:18.055 --> 00:26:22.375
So that is for some token management is possible within Foundry buyer projects.
308
00:26:22.615 --> 00:26:25.335
But if you want more granular segregation,
309
00:26:25.335 --> 00:26:35.249
like at the agent level for a one use case, I've built an agent and I want the tokens to be measured exclusively for this use case, like you mentioned, a public facing agent.
310
00:26:35.330 --> 00:26:38.369
In that case, you have an AI gateway.
311
00:26:38.690 --> 00:26:42.530
AI gateway comes in where you can set up for API
312
00:26:42.530 --> 00:26:44.134
rate limits and
313
00:26:44.294 --> 00:26:48.215
token limits. So per minute as well as weekly,
314
00:26:48.375 --> 00:26:49.974
monthly, or quarterly
315
00:26:50.135 --> 00:26:55.255
rate limits that you can set up. So then as soon as it starts hitting those limits, you'll get notifications
316
00:26:55.789 --> 00:27:04.669
or the people start getting four two nines. They start getting blocked that, hey, you are crossing your rate limit, so please don't and retry after a certain
317
00:27:04.750 --> 00:27:18.225
amount of time. So that is where AI gateway comes in, to have that overall cost governance as one of the topics. There is a lot more to AI gateway with AI governance that you can do. But that gives you that feature of segregating
318
00:27:18.225 --> 00:27:19.024
costs
319
00:27:19.105 --> 00:27:21.584
and creating budgets and monitoring
320
00:27:21.904 --> 00:27:24.065
cost token utilization
321
00:27:24.065 --> 00:27:28.109
at API level as well as agent level or project level.
322
00:27:28.590 --> 00:27:29.710
And
323
00:27:29.710 --> 00:27:31.389
so you just mentioned
324
00:27:31.549 --> 00:27:34.350
Foundry projects, for instance, you can manage
325
00:27:35.789 --> 00:27:40.124
tokens. So how many tokens? API management also how many tokens?
326
00:27:40.285 --> 00:27:49.244
But how do I know what a token costs? Because an output token for Fable is a very different price from an output token from a different model. So how do you do that?
327
00:27:49.645 --> 00:27:51.565
Yeah, we do have a pricing calculator.
328
00:27:51.980 --> 00:27:54.780
So online, it is published, Microsoft
329
00:27:55.500 --> 00:28:00.380
pricing calculator, you where you can actually see what each model, how much it costs
330
00:28:00.460 --> 00:28:15.335
per token or or tokens. It is also in TPM. So how much is the cost for tokens per minute for certain model? And then you can make an estimate. There's a pricing calculator where you can go and make an estimate of how much roughly my use case would cost
331
00:28:15.735 --> 00:28:17.174
if these were the
332
00:28:17.760 --> 00:28:21.679
amount of calls it comes in, these were the amount of prompts,
333
00:28:21.760 --> 00:28:25.760
prompt tokens coming in, completion tokens coming in coming out.
334
00:28:26.080 --> 00:28:29.840
And, yeah, you can make a rough estimate, and that's how basically the costs are calculated.
335
00:28:30.445 --> 00:28:33.804
Ah, okay. So currently, I need to do this manually.
336
00:28:34.125 --> 00:28:47.570
So I need to calculate what it would cost and then set a budget based on then it is x amount of tokens because I'm gonna use Fable for this. Yes. Indeed. Uh-huh. Okay. I'm asking for a friend, so I'm wondering
337
00:28:48.050 --> 00:28:55.890
For sure. Is this gonna be because, obviously, you want this to be automatic. Right? I wanna say this project can cost me €300
338
00:28:55.890 --> 00:28:57.410
and this one 5,000,
339
00:28:57.575 --> 00:29:00.534
Not tokens because I don't know the tokens. Is
340
00:29:00.855 --> 00:29:01.654
that common?
341
00:29:02.455 --> 00:29:03.575
So
342
00:29:04.934 --> 00:29:12.455
I think that I don't know, Furt. So short answer, I've not seen that on the roadmap yet to have the cost base segregation.
343
00:29:12.780 --> 00:29:14.299
It still is TPM,
344
00:29:14.300 --> 00:29:19.179
but the calculation is also pretty straightforward. Right? When you set 10,000 TPM,
345
00:29:19.420 --> 00:29:20.460
you also have
346
00:29:20.700 --> 00:29:26.700
an idea of how much prompts and completions it can handle and then how many requests also it can handle.
347
00:29:27.315 --> 00:29:28.354
Six requests,
348
00:29:28.514 --> 00:29:45.489
10 requests, 20 requests. So then you also have an idea, okay, my app will be used within this time window, how much it can be used. And as soon as it starts hitting limit, you'll start getting four to nine. That's a very popular error Mhmm. And says, hey, rates limit reached. So there is to answer your question
directly, there is
349
00:29:45.809 --> 00:29:49.489
I I have not seen a cost based segregation coming yet,
350
00:29:49.889 --> 00:29:53.889
but that would be a very nice feature to probably discuss with the product group.
351
00:29:54.370 --> 00:29:56.529
Yeah. It would be. Yeah. Yeah. Feedback.
352
00:29:57.490 --> 00:29:57.970
Okay.
353
00:29:58.555 --> 00:30:03.275
And and so whilst talking about this, you already said, so you have Foundry,
354
00:30:03.275 --> 00:30:07.515
which is kind of an instance within Azure, right? And that thing can have projects.
355
00:30:07.755 --> 00:30:15.239
Yeah. Which you can then give to, here's your use case or here is a team working on something. So what is now a typical
356
00:30:16.919 --> 00:30:18.200
overview architecture
357
00:30:18.200 --> 00:30:20.519
within, let's say, you have a larger enterprise,
358
00:30:20.840 --> 00:30:27.320
like one central team that manages things, infrastructure team or something. And then we have all these other teams that build things.
359
00:30:28.144 --> 00:30:33.344
So would I, for instance, then deploy one Foundry instance centrally,
360
00:30:33.664 --> 00:30:37.984
maybe an API gateway connection there, and then projects.
361
00:30:37.985 --> 00:30:44.519
I give a project to team A, project to team B, project to team C. Is that kinda how you do it?
362
00:30:46.039 --> 00:30:49.479
So there is a Microsoft AI reference architecture
363
00:30:49.720 --> 00:30:54.679
that is that we have as a starting point that is proposed by Microsoft.
364
00:30:56.145 --> 00:31:01.265
I do and in there, the idea is you have a central AI hub
365
00:31:01.265 --> 00:31:04.065
with where you have the AI gateway,
366
00:31:04.545 --> 00:31:06.945
which manages access to the models,
367
00:31:07.025 --> 00:31:10.545
manages access to external systems, MCP servers,
368
00:31:10.870 --> 00:31:15.989
other agents, etcetera. You do rate limiting there, cost management,
369
00:31:16.870 --> 00:31:22.549
everything. All AI governance capabilities are set up at the gateway. So that is within an AI hub subscription.
370
00:31:22.789 --> 00:31:25.510
And then you have agent spoke subscriptions.
371
00:31:25.975 --> 00:31:28.454
And that's where you have Foundry deployed
372
00:31:28.615 --> 00:31:32.854
per team, per use case, per business unit within an organization
373
00:31:32.934 --> 00:31:35.574
where they start building their apps and agents.
374
00:31:35.894 --> 00:31:41.210
So this is how the reference architecture is proposed from Microsoft. But
375
00:31:41.210 --> 00:31:42.570
within customers,
376
00:31:42.810 --> 00:31:47.370
in reality on the on the ground, I see this being adopted
377
00:31:47.370 --> 00:32:05.414
in various ways. Mhmm. Depending on the scale of the organization and how they are set up. There are some customers who who have a central IT team who also want to govern everything centrally. So they not just have the AI gateway centrally, they also have a foundry resource centrally and provide projects to business units to build apps and agents.
378
00:32:05.655 --> 00:32:10.999
That works tools. But there are some organizations who want to completely federate it and adopt the
379
00:32:11.799 --> 00:32:19.399
reference architecture as it is prescribed. So then there is an AI governance team, which also owns the AI gateway,
380
00:32:19.480 --> 00:32:20.840
also is responsible
381
00:32:20.840 --> 00:32:26.644
for triaging the AI use cases to decide which use case would need what tools and capabilities.
382
00:32:26.725 --> 00:32:39.480
And then there is the business units, which will have their own subscription or own resource group with Foundry in there who build their apps and agents within it. So and then they have a top governance layer with Agent three sixty five
383
00:32:39.640 --> 00:32:40.520
integrated
384
00:32:40.520 --> 00:32:41.480
vendor,
385
00:32:41.560 --> 00:32:43.799
you know, complete enterprise setup
386
00:32:44.440 --> 00:32:50.440
that then adds to adds to also the security monitoring and governance requirements from enterprises.
387
00:32:50.600 --> 00:32:56.554
That is different flavors we see, all adopted from the reference architecture that we have.
388
00:32:57.115 --> 00:32:57.915
Okay.
389
00:32:57.915 --> 00:33:10.380
And and maybe like a final topic ish that you just mentioned agent three sixty five. Because I was gonna ask, so how you have all these agents, they are running everywhere. How do you control them and even know that they exist?
390
00:33:10.540 --> 00:33:14.539
It's kinda what it is, right? Yes. So an agent sprawl is
391
00:33:15.260 --> 00:33:19.420
is a topic and a a concern that many enterprises
392
00:33:19.420 --> 00:33:27.424
are who are adopting agents and building apps and agents with AI are are noticing it and are trying to find solutions for it.
393
00:33:27.985 --> 00:33:39.630
So you can build agents in Foundry. You can build agents in Cobalt Studio. You can build custom agents. You can build agents with the m three six extend the m three sixty five agents that are there or even outside.
394
00:33:39.710 --> 00:33:51.395
Right? But then how do you control with what is happening where? Where is my data going? Where is it being used? So with agent three sixty five, are we are at Microsoft, they are bringing a consolidated
395
00:33:51.395 --> 00:33:52.195
pane,
396
00:33:52.595 --> 00:33:57.715
a control pane, right, for agents built anywhere in their enterprise,
397
00:33:57.955 --> 00:33:59.235
anywhere within your tenant.
398
00:34:00.580 --> 00:34:02.100
You could monitor
399
00:34:02.260 --> 00:34:23.115
what is happening in the agents, where it is hosted, what tools it's accessing, what data it is accessing, also set up guardrails on top of it to to ensure that they access only the data that they should, the systems that they should, they perform only the set of actions that they should. If there are any threat actors acting on AI systems,
400
00:34:23.275 --> 00:34:28.315
then alerts are sent out to the SOC team via Defender for AI integration
401
00:34:29.220 --> 00:34:33.140
to also take action in real time. Right? So so that
402
00:34:33.380 --> 00:34:39.940
is Agent 365 will become the central control center or command center for all agents
403
00:34:40.020 --> 00:34:41.620
built in your organizations,
404
00:34:41.620 --> 00:34:46.235
be it pro code, custom agents with Foundry, outside Foundry,
405
00:34:46.395 --> 00:34:51.195
or more no code agents built via M365 or CoPilot Studio.
406
00:34:51.595 --> 00:34:52.155
Yeah.
407
00:34:52.475 --> 00:34:55.595
So that is where Agent three sixty five comes in. Okay.
408
00:34:56.315 --> 00:34:57.035
So much stuff.
409
00:34:58.650 --> 00:34:59.690
It's complicated.
410
00:34:59.690 --> 00:35:07.130
And it keeps keeps getting added. More and more things keep coming in every week, every two weeks. So, yeah, it is quite
411
00:35:07.130 --> 00:35:08.170
a changing
412
00:35:08.170 --> 00:35:08.650
space.
413
00:35:09.965 --> 00:35:11.725
Space. Absolutely.
414
00:35:12.205 --> 00:35:12.765
So
415
00:35:13.085 --> 00:35:15.005
what's your favorite model today?
416
00:35:16.605 --> 00:35:18.365
So plot four dot eight
417
00:35:19.485 --> 00:35:24.685
is is favorite. So I've that's the developer in me talking. Right? So it's I for me,
418
00:35:25.250 --> 00:35:34.369
if I have to build something, that's the one I rely on. But also the auto mode. So talking a bit also about GitHub Copilot Mhmm. Which is the environment
419
00:35:34.770 --> 00:35:36.690
which is the IDE I use, basically,
420
00:35:38.945 --> 00:35:41.985
which is the agent I use for my development within
421
00:35:41.985 --> 00:35:50.145
Versus Code. So for me, four dot eight is what gives me the best results. Five dot five also, GPT five .5 also at times,
422
00:35:50.385 --> 00:35:58.670
but I've also recently seen the auto more tends to perform really well if I just want to ask some questions, if I want to understand my code better,
423
00:35:59.070 --> 00:36:01.390
then even the auto mode performs
424
00:36:01.390 --> 00:36:06.430
really well. So, yeah, that space has evolved. The developer productivity space is evolving as well.
425
00:36:07.235 --> 00:36:14.595
Yeah. Absolutely. Yeah. And that's good. So you're not token maxing. You're just actually turning on auto mode. Indeed.
426
00:36:14.675 --> 00:36:17.715
Okay. Good. You should check out the latest
427
00:36:18.115 --> 00:36:19.475
GPT 5.6.
428
00:36:19.475 --> 00:36:31.599
There are three of them. Oh, I haven't I haven't tested those out yet. They're pretty good. So so for reference listener, we're recording this on the July 14. I'm sure by now you guys have GPT six or beyond.
429
00:36:31.760 --> 00:36:33.440
We'll see what happens in the future.
430
00:36:34.375 --> 00:36:34.935
So
431
00:36:35.735 --> 00:36:39.335
and what would you advise your your children to study?
432
00:36:39.895 --> 00:36:43.895
Oh, to my children, I would I would advise the people to and
433
00:36:44.615 --> 00:36:46.695
this is this is a bit
434
00:36:47.170 --> 00:36:48.530
a tricky answer,
435
00:36:48.530 --> 00:36:57.410
but I would advise them to definitely get their basics right and not rely completely on AI because your AI system is as good as the prompter.
436
00:36:57.490 --> 00:37:00.850
Right? So you the prompt the person who is sending the prompt
437
00:37:01.010 --> 00:37:04.225
needs to have the fundamentals
438
00:37:04.305 --> 00:37:05.345
said straight.
439
00:37:06.145 --> 00:37:10.065
And then you see the AI AI system as helping you
440
00:37:10.545 --> 00:37:23.579
or augmenting to your knowledge rather than, you know, relying fully on it. So to my daughter, I would say, you do need to study. You need do need to get your basics and fundamentals right, and then use AI as a as a assistant
441
00:37:23.579 --> 00:37:27.340
rather than, you know, completely relying on it. Yeah.
442
00:37:27.660 --> 00:37:30.460
Yeah. That's gonna be interesting times. We'll see. Yes.
443
00:37:31.194 --> 00:37:35.515
For sure. I already see that happening. Yeah. Yeah. Lands increasing.
444
00:37:36.635 --> 00:37:41.835
Alright. And where can people go and what's the best place to connect with you?
445
00:37:42.234 --> 00:37:43.435
Yeah. I have
446
00:37:43.835 --> 00:37:44.954
my LinkedIn profile.
447
00:37:46.190 --> 00:37:48.910
It's by the name of Hina Rifai. You can look me up.
448
00:37:49.710 --> 00:37:51.630
Yeah. And just drop me a DM.
449
00:37:51.869 --> 00:37:55.230
I usually respond within the day. I'm quite active there.
450
00:37:56.029 --> 00:37:58.510
So, yeah, that is the place to reach out.
451
00:37:58.829 --> 00:38:03.145
Okay. Well, thank you very much for for teaching me all this stuff today. I learned a lot.
452
00:38:03.545 --> 00:38:06.905
Oh, thank you. And thank you for having me here. I love the conversation.
453
00:38:07.865 --> 00:38:10.905
Yeah. And this was a super nice experience.
454
00:38:11.460 --> 00:38:12.260
Okay.
455
00:38:12.260 --> 00:38:14.980
Thank you very much, and thank you all for listening.
456
00:38:15.540 --> 00:38:16.900
Thank you. Bye bye.