TRANSCRIÇÃO 🔗
WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 06/29/2025 22:18:20
Duration: 3258.283
Channels: 1
1
00:00:11.280 --> 00:00:15.460
Hello, and welcome to the Data Engineering Podcast, the show about modern data management.
2
00:00:16.395 --> 00:00:22.895
Data migrations are brutal. They drag on for months, sometimes years, burning through resources and crushing team morale.
3
00:00:23.675 --> 00:00:35.550
DataFold's AI powered migration agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches.
4
00:00:36.170 --> 00:00:40.430
And they're so confident in their solution, they'll actually guarantee your timeline in writing.
5
00:00:40.970 --> 00:00:46.685
Ready to turn your year long migration into weeks? Visit dataengineeringpodcast.com/datafolds
6
00:00:46.685 --> 00:00:47.825
today for the details.
7
00:00:48.605 --> 00:00:52.225
Poor quality data keeps you from building best in class AI solutions.
8
00:00:52.525 --> 00:00:56.785
It costs you money and wastes precious engineering hours. There is a better way.
9
00:00:57.165 --> 00:01:07.480
Core signal's multi source enriched cleaned data will save you time and money. It covers millions of companies, employees, and job postings and can be accessed via API or as flat files.
10
00:01:08.100 --> 00:01:14.440
Over 700 companies work with Core Signal to develop AI solutions in investment, sales, recruitment, and other industries.
11
00:01:15.060 --> 00:01:17.965
Go to dataengineeringpodcast.com/coresignal
12
00:01:17.965 --> 00:01:21.185
and try Core Signal's self-service platform for free today.
13
00:01:22.685 --> 00:01:33.150
Your host is Tobias Macy, and today I'm interviewing Arun Joseph about building an agent platform to empower the business to adopt agentic capabilities. So, So, Arun, can you start by introducing yourself?
14
00:01:33.850 --> 00:01:34.910
Hey. Hello, Tobias.
15
00:01:35.290 --> 00:01:37.230
Thanks for inviting me to this podcast.
16
00:01:37.530 --> 00:01:39.390
So I'm Arun. Arun Joseph.
17
00:01:40.490 --> 00:01:42.750
I'm based out of Germany, Bonn.
18
00:01:43.370 --> 00:01:46.335
Well, I've always built distributed systems.
19
00:01:46.875 --> 00:01:49.135
I've been leading engineering organizations.
20
00:01:50.075 --> 00:01:59.890
In my most recent role, I was heading the AI engineering program for Deutsche Telekom Group, where we build something referred to as Elmos, which is the agent platform,
21
00:02:00.190 --> 00:02:04.930
which is open source. And now, right now, I'm on my entrepreneurial
22
00:02:05.230 --> 00:02:05.730
path,
23
00:02:06.190 --> 00:02:18.705
building something around multi agent systems because this has been something which is close to my heart quite a for quite a long time. And, well, I I'm originally from India. I used to work for large scale enterprises
24
00:02:19.085 --> 00:02:23.585
and leading engineering teams across the globe, also in San Francisco, Juniper,
25
00:02:24.200 --> 00:02:24.700
then,
26
00:02:25.080 --> 00:02:26.060
Merck Pharmaceuticals,
27
00:02:27.080 --> 00:02:28.380
for I built their
28
00:02:28.920 --> 00:02:34.540
industrial IoT platform for Merck. Always build large scale distributed systems. It was mostly entrepreneurship
29
00:02:35.160 --> 00:02:35.800
kind of,
30
00:02:36.520 --> 00:02:41.845
kind of, endeavors is the first time I'm jumping headfirst into, you know, entrepreneurial
31
00:02:42.465 --> 00:02:50.165
stuff outside an organization. So I'm passionate about distributed systems and large scale AI systems, which is gonna define the future.
32
00:02:51.120 --> 00:02:57.140
And do you remember how you first got started working in data and also in the AI and ML space?
33
00:02:58.239 --> 00:03:01.379
Yeah. I think the first introduction to large scale
34
00:03:01.680 --> 00:03:02.900
data was
35
00:03:03.360 --> 00:03:05.459
when I was used to work for Nielsen,
36
00:03:05.795 --> 00:03:08.455
which is into a market research company as
37
00:03:08.755 --> 00:03:12.295
and me Nielsen had this interesting subsidiary called Arbitron.
38
00:03:12.595 --> 00:03:13.334
And Arbitron
39
00:03:13.795 --> 00:03:21.270
had this device which used to collect listen to radio signals placed in these homes where people sign up and collect
40
00:03:21.570 --> 00:03:27.110
listening data, listenership data. And this used to go through these large data pipelines to build listenership
41
00:03:27.410 --> 00:03:28.470
market intelligence.
42
00:03:28.850 --> 00:03:37.045
So this was the first time, we started thinking about large scale data pipelines. This is I think this was right around the time Hadoop
43
00:03:37.345 --> 00:03:37.845
was,
44
00:03:38.625 --> 00:03:41.845
brought in. And, AIML pipelines, MLOps,
45
00:03:42.625 --> 00:03:48.165
which was also there in in several of their endeavors, but that's essentially the the the recent experience.
46
00:03:49.120 --> 00:03:50.099
And so
47
00:03:50.480 --> 00:03:57.780
in terms of the overall space of building agentic systems, particularly when you're looking at the organizational
48
00:03:58.080 --> 00:04:02.260
scale and not just a little toy implementation or a proof of concept,
49
00:04:02.625 --> 00:04:04.965
There is a lot of complexity involved.
50
00:04:05.504 --> 00:04:06.004
So
51
00:04:06.305 --> 00:04:13.045
the early stages was very oriented around just simple rag bots. I I say simple in air quotes because they could be very complex.
52
00:04:13.424 --> 00:04:19.419
But as we move to more agentic capabilities where we're adding to the sets of tools and we're giving more
53
00:04:19.800 --> 00:04:26.139
free rein to the language models to make decisions and determine the execution paths in nondeterministic
54
00:04:26.680 --> 00:04:32.060
fashion, there is a lot more that goes into it. And particularly, if you are trying to
55
00:04:32.444 --> 00:04:32.944
Load full transcript
empower
56
00:04:33.324 --> 00:04:33.824
nontechnical
57
00:04:34.125 --> 00:04:42.384
stakeholders who don't necessarily have the deep domain expertise in how these systems work, you wanna make sure that they're fairly foolproof.
58
00:04:42.844 --> 00:04:46.705
And I'm wondering if you can talk to some of the ways that you thought about
59
00:04:47.190 --> 00:04:48.330
the different
60
00:04:48.870 --> 00:05:01.210
components and domain segments that go into building something like an agentic system and then being able to scale that to the overall organization, just how you started to approach that overall problem definition.
61
00:05:02.205 --> 00:05:14.224
Yeah. Absolutely. So this is a fascinating topic because there are several constructs to unpack in here of which one of the most important construct is English is the new programming language, or your thought is the new design.
62
00:05:15.240 --> 00:05:15.740
And,
63
00:05:16.200 --> 00:05:21.820
essentially, what happens what is happening right now is this this thing called as agentic orchestration,
64
00:05:22.680 --> 00:05:32.634
which is merely a feedback loop if you look at it. Right? You provide a goal. And to an intelligent system, it could also this is how even people work. You give a goal,
65
00:05:33.014 --> 00:05:34.074
and the entity
66
00:05:34.535 --> 00:05:37.194
attempts to do it given enough tools.
67
00:05:37.495 --> 00:05:38.634
If it doesn't work,
68
00:05:38.935 --> 00:05:43.115
loop through it, get new insights, and then iterate. So this orchestration
69
00:05:43.574 --> 00:05:44.074
loop
70
00:05:44.375 --> 00:05:45.435
makes it possible
71
00:05:46.010 --> 00:05:47.470
that if you're defining
72
00:05:47.850 --> 00:05:49.710
your instructions very precisely,
73
00:05:50.090 --> 00:06:20.069
like as a program, add two numbers, it doesn't have to go through that loop multiple times. But if you're specifying something in an abstract manner, I have two chocolates and then somebody's giving me something and I need to figure out what's the best way to do it without knowing the construct of addition, then this is the loop first figures out what should be the mathematical construct to apply. Okay. This is good. Now let let's apply this. Great. And then let's verify it. So in this example, what just happened? So you move from
74
00:06:20.449 --> 00:06:28.285
programming, which was very deterministic, used to be written by SMEs or specialized expertise, which was required. It shifts
75
00:06:29.225 --> 00:06:32.205
to this capability allows people to describe
76
00:06:32.745 --> 00:06:40.365
in a more broader manner what they want, and the loop takes care of it to some degree. The implications are profound,
77
00:06:40.824 --> 00:06:41.965
especially in enterprises.
78
00:06:42.740 --> 00:06:52.840
What is most enterprise information? What does most enterprise information systems do? They merely move data from one place to the another and then do some kind of transformation.
79
00:06:53.220 --> 00:07:03.664
And, of course, there is resilience, reliability, and distributed systems. But most of the requirements are around moving data from one place to another, and and some business stakeholder would say,
80
00:07:03.965 --> 00:07:15.860
I want it this way. I want it that way. And and now tying it back to the analogy that I said, now the engineers of an organization need not be building systems
81
00:07:16.639 --> 00:07:18.020
exactly as how
82
00:07:18.560 --> 00:07:21.860
the business people would say, but rather build systems
83
00:07:22.240 --> 00:07:23.780
which allows them to
84
00:07:24.240 --> 00:07:34.955
the business to tell what is required, and the system makes it come true. So this is this is the fundamental mental model. But in order to make that possible, you need great models. You need,
85
00:07:35.675 --> 00:07:51.770
a robust way to connect data and tools, a robust way to do orchestration loops, a robust way to manage the growing complexity as you build more such programs, and the complexity on how these programs can interact with each other. That, in essence,
86
00:07:52.085 --> 00:07:55.225
is the anatomy of a multi agent system.
87
00:07:56.245 --> 00:07:57.705
And in terms of
88
00:07:58.085 --> 00:08:04.425
the primitives that you have available to build a system like that, that's obviously a very rapidly evolving
89
00:08:04.805 --> 00:08:05.305
ecosystem
90
00:08:05.900 --> 00:08:13.040
to your point of multi agent systems that, of course, brings to mind the a to a protocol that Google just donated to the Linux Foundation.
91
00:08:13.580 --> 00:08:19.920
In terms of tool call definitions, it brings up the hype that's been growing around the model context protocol.
92
00:08:20.665 --> 00:08:21.485
There are obviously
93
00:08:22.185 --> 00:08:24.365
custom tool definitions that you can create.
94
00:08:24.665 --> 00:08:33.085
The idea of handing tools to these language models has been around for at least a year or two. And I'm wondering how you think about what are the
95
00:08:33.450 --> 00:08:51.615
stable primitives that you have available to be able to build an enterprise scale system on top of without risking that one of these protocol authors is going to introduce a new revision that's going to massively change the capabilities or require a lot of rework in terms of how you actually think about your implementation?
96
00:08:53.035 --> 00:08:59.935
Yeah. This is a very, very good question in terms of why went through the other side of it. Because we built a product all first
97
00:09:00.360 --> 00:09:12.460
before MCP, before a two way, into Elmos, the platform that we built. But then these MCPs came up, and our stakeholders and as well as our team started to ask, what are you gonna do now? Yeah.
98
00:09:13.320 --> 00:09:19.205
So this is a valid question, but there there there's, let's unpack a little a a few things around here.
99
00:09:19.825 --> 00:09:21.045
We went into production
100
00:09:21.425 --> 00:09:21.925
back
101
00:09:22.545 --> 00:09:38.779
in the first quarter of twenty twenty four as as an agentic platform, not not only, the first two agents which went to production was in the first quarter of, of twenty twenty four. It used to connect to telco APIs, which are really
102
00:09:39.240 --> 00:09:39.740
complex.
103
00:09:40.205 --> 00:09:41.345
So there was no MCP.
104
00:09:41.725 --> 00:09:47.105
There was no eight way, but we could do it. The first principles don't change. That's the whole point, essentially.
105
00:09:47.405 --> 00:09:52.705
And then we started to when we started to build more such agents and more such APIs,
106
00:09:53.085 --> 00:09:54.065
the need for
107
00:09:54.530 --> 00:09:55.910
some sort of harmonization
108
00:09:56.370 --> 00:10:00.390
came in. What is the pattern that you tell an engineer without reinventing
109
00:10:01.090 --> 00:10:03.590
the wheel or the pattern which we follow?
110
00:10:03.970 --> 00:10:14.445
This was the beginning of what we refer to as our Elmos protocol, which was based on web of things, because Elmos was built on the fundamental construct that everything should be open,
111
00:10:14.985 --> 00:10:24.525
and not reinvent the wheel. So let's pick up the best protocols out there to without reinventing a new protocol, how do you allow agents to collaborate? But at the same time,
112
00:10:25.130 --> 00:10:36.830
organizations, especially when models needed to connect to data, MCB was the first one which came out with that topic. And it immediately gained a lot of adoption because of the size and scale of the cloud and desktop,
113
00:10:37.255 --> 00:10:45.595
as well because it could connect to the models quickly the tools quickly. So but the first principle of how
114
00:10:45.975 --> 00:10:58.490
you would need to connect your data is not gonna change. How you need to the need for bringing your data, because most of these enterprise APIs are have been there for years. When GraphQL came in, a lot of people tried to build
115
00:10:59.030 --> 00:10:59.530
GraphQL,
116
00:11:00.550 --> 00:11:05.370
wrappers on top. And now MCP comes in, and then people are trying to retrofit
117
00:11:05.715 --> 00:11:20.855
MCP into the traditional APIs. And then on top of it, a two way. My suggestion is before betting into any big protocol, it's pretty simple to connect your existing API to any of these LLM prompts.
118
00:11:21.160 --> 00:11:21.980
Start there,
119
00:11:22.440 --> 00:11:25.900
experiment it, and then if it scales, then bring in new additional
120
00:11:26.280 --> 00:11:29.100
layers. And MCP looks like
121
00:11:29.960 --> 00:11:31.260
is adopted by OpenAI
122
00:11:31.800 --> 00:11:33.180
and also Entropic
123
00:11:33.795 --> 00:11:36.455
and Microsoft. And a lot of MCP servers
124
00:11:36.835 --> 00:11:39.815
is good enough to bet, but asynchronous
125
00:11:40.595 --> 00:11:44.214
large scale asynchronous communication is not solved yet in MCP.
126
00:11:44.595 --> 00:11:44.835
And,
127
00:11:45.555 --> 00:11:46.774
a two way, similarly,
128
00:11:47.475 --> 00:11:48.615
whether the need
129
00:11:48.959 --> 00:11:58.100
for a large scale agent collaboration is required or not is is still a question. Because if you need a large scale agent collaboration, you need large scale state management,
130
00:11:58.560 --> 00:12:02.420
which should be true. Otherwise, two agents cannot collaborate.
131
00:12:02.754 --> 00:12:06.194
So a two a is still shady area in my view, but,
132
00:12:06.834 --> 00:12:09.735
and most of the enterprise system might not require
133
00:12:10.435 --> 00:12:14.535
every problem to be seen as agents. That's that's another thing. In microservices,
134
00:12:14.915 --> 00:12:18.709
well, we say don't start with microservices. Right? Start with the monolith.
135
00:12:19.250 --> 00:12:24.870
So it's easier to build your particular department's use case as a single system.
136
00:12:25.410 --> 00:12:33.985
Connect your data with MCP. Or if it's legacy API, don't try to MCPify it. Keep it simple, the case principle. Keep it simple. Stupid.
137
00:12:34.845 --> 00:12:37.824
And then think about scaling it to other protocols.
138
00:12:39.324 --> 00:12:40.785
The other interesting challenge
139
00:12:41.165 --> 00:12:44.480
when you're dealing with an organizational scale
140
00:12:44.860 --> 00:12:45.360
enablement
141
00:12:45.740 --> 00:12:46.240
system,
142
00:12:46.620 --> 00:12:57.600
particularly for something like agentic use cases, is to your point, you need to be able to connect it to the underlying data that the agent needs to be able to use either to operate against or to use as context.
143
00:12:58.105 --> 00:13:14.329
And that's a substantial challenge in and of itself, again, because of the fact that data in and of itself is challenging, but, also, you're working with people who don't have that domain knowledge to understand all of the complexity that goes into all of the data prep, all of the ETL pipelining,
144
00:13:14.790 --> 00:13:18.009
how to manage the different chunking strategies, etcetera.
145
00:13:18.470 --> 00:13:21.850
And so I'm wondering how you think about that element
146
00:13:22.709 --> 00:13:36.135
of exposing the existing organizational data and then streamlining the work of actually doing the additional preparatory work for being able to load that context into something like a vector database and standardizing
147
00:13:36.515 --> 00:13:37.655
on that core technology.
148
00:13:38.770 --> 00:13:43.270
In fact, I mean, this clearly has been a missing piece in most enterprises.
149
00:13:43.650 --> 00:13:55.505
Enterprises used to have data pipelines, but it was mostly for transactional data. Transactional systems would emit the data, which is going into huge data pipelines, NiFi's, etcetera, landing in warehouses.
150
00:13:56.365 --> 00:13:56.865
What
151
00:13:57.485 --> 00:14:00.145
right now emerged is a need for,
152
00:14:00.925 --> 00:14:01.425
unstructured
153
00:14:01.805 --> 00:14:03.265
natural language data
154
00:14:03.645 --> 00:14:04.865
to be parsed
155
00:14:05.339 --> 00:14:05.839
into
156
00:14:06.220 --> 00:14:13.680
semantic vectors, which could be queried by and and and finally collapsed for an answer by the LLMs.
157
00:14:14.300 --> 00:14:14.800
So
158
00:14:15.420 --> 00:14:20.014
this is a new skill and technical skill sets that most organizations
159
00:14:20.634 --> 00:14:22.894
need to learn, but the first principles
160
00:14:23.274 --> 00:14:23.774
of
161
00:14:24.315 --> 00:14:28.654
how large scale data ingestions work still applies, which means
162
00:14:29.035 --> 00:14:36.940
the data still would need to go through phases and pipelines. For example, in, when when in in in Deutsche Telekom, when,
163
00:14:37.320 --> 00:14:38.700
we wanted to ingest,
164
00:14:39.640 --> 00:14:40.040
this,
165
00:14:40.680 --> 00:14:48.300
corporate knowledge base for customer support people and FAQs, if you just ingest it as it is, it would not
166
00:14:48.725 --> 00:14:51.865
the the the the search endpoints were not
167
00:14:52.565 --> 00:14:53.065
performing
168
00:14:53.605 --> 00:14:57.385
as we would have wanted it to be. The accuracy rate was subpar.
169
00:14:57.685 --> 00:15:21.825
So we had to create the ontology and cleaning up of the domain objects and the domain topics to create the ontology of the knowledge for the customer support groups and build that ontology into the pipeline. So it it requires two skill sets. One is a technical skill set on what is a chunking strategy, what is an embedding model, what kind of vectorization approaches can you use. The other is the domain knowledge
170
00:15:22.365 --> 00:15:34.490
on, hey. My customer support is broken into domain billing domain and customer domain and contract. I have Magenta TV and and all that. So both the knowledge are essential,
171
00:15:34.870 --> 00:15:41.610
and this is this in essence need to come hand in hand to create those pipelines. But but at the same time,
172
00:15:41.990 --> 00:15:44.330
there are only a few things that you can tweak
173
00:15:45.555 --> 00:15:54.454
to bring the best answers, especially in RAG. One is definitely the vector database. The second is the pipeline itself on how do you create the ontology and the cleanup.
174
00:15:54.755 --> 00:15:56.695
And the third is the actual
175
00:15:57.074 --> 00:15:59.255
search endpoint that you built, which
176
00:15:59.600 --> 00:16:14.500
should be able to rely not just maybe on vectorization, but also on additional dimensions to create a hybrid search approach. These are the levers which you have. And and I think the vector side plays a huge role in terms of
177
00:16:15.605 --> 00:16:29.720
not many developers are going to create their own vector databases for sure. Rest of the pieces, they can tweak. What kind of search algorithm that they need to put in? What kind of pipelines that they need to build in? So
the choice of, the vectorization
178
00:16:30.340 --> 00:16:40.145
approaches and and and how that system would scale with large ingestion pipelines is something which need to be really thought through. I think in in DT,
179
00:16:40.605 --> 00:16:42.385
we brought in Quadrant,
180
00:16:42.685 --> 00:16:49.425
which served really well the purposes of operational simplicity that Quadrant baked in. And in terms of the vectorizations,
181
00:16:49.805 --> 00:16:54.385
it it Quadrant has this brilliant user interface, which shows the entire embeddings
182
00:16:55.070 --> 00:16:58.290
space, and the operational simplicity was unparalleled.
183
00:16:58.750 --> 00:17:05.970
And this helped in streamlining, at least, for the developers to focus on the ingestion pipelines and not on operational characteristics
184
00:17:06.270 --> 00:17:07.810
of maintaining the topology
185
00:17:08.155 --> 00:17:16.735
of the of the vector databases because you would need multiple multi tenancy and all that, which resulted in that pipeline that we referred to as.
186
00:17:17.515 --> 00:17:22.095
So this was developed by a couple of engineers in my team, Thomas Weigel.
187
00:17:22.410 --> 00:17:23.710
So in German,
188
00:17:24.410 --> 00:17:24.910
means
189
00:17:25.290 --> 00:17:37.150
root. So this was, like, built as pipelines where it it's it's like roots that can go into the soil. So it's like the enterprise world is like soil, and it need to go in and collect the nutrients,
190
00:17:37.565 --> 00:18:11.485
which is the data, which is in many different formats. So, essentially, it's it did not try to replace anything, but rather what's the best embedding strategy or what is the best frame the a lot of frameworks were coming up, and these frame there will be one or two capabilities within the framework, which would be really good. How do you club together these capabilities and not bet on only one Lama index or, something like that? So this was WordCell, which is a pipeline, and it ingest the data, and it can land in the quadrant cluster that we had. And then the search endpoints were built.
191
00:18:12.105 --> 00:18:13.405
In terms of the
192
00:18:13.865 --> 00:18:15.005
embeddings and
193
00:18:15.305 --> 00:18:18.605
the additional context that you need to bring in,
194
00:18:18.910 --> 00:18:23.010
one of the common pieces of wisdom when you're building RAG systems
195
00:18:23.310 --> 00:18:33.090
is that you don't know what chunk chunking strategy you want to use until after you've tried it and experimented a little bit. And so I'm wondering how you thought about
196
00:18:33.405 --> 00:18:34.065
either determining
197
00:18:34.765 --> 00:18:41.905
this is the standard strategy we're going to use because it's good enough on most use cases or how much control you wanted to give to those
198
00:18:42.205 --> 00:18:55.490
end builders of these agents to be able to say, you know that content that you're working on, so you're going to set some parameters to
determine which chunking approach to use and just how to think about building that into a
199
00:18:55.790 --> 00:18:59.650
flexible system without making it flexible to the point of being useless.
200
00:19:00.430 --> 00:19:07.915
Yeah. Absolutely. I think too much obstruction. It's it's a right balance that we need to think about which went into the system as well. So, essentially,
201
00:19:08.375 --> 00:19:11.035
since because we were not building a platform
202
00:19:11.655 --> 00:19:18.040
right from the start as in we we were supposed to solve these use cases. So we were only measured by one one,
203
00:19:18.360 --> 00:19:19.100
one metric.
204
00:19:19.480 --> 00:19:21.100
How accurate are the answers
205
00:19:22.360 --> 00:19:23.500
for the German
206
00:19:23.800 --> 00:19:34.235
customer service bot for the customer? And and in order to achieve that, we did not start with by thinking what flexibility do we need to have in the framework.
207
00:19:34.535 --> 00:19:35.755
We did the
208
00:19:36.135 --> 00:19:44.100
heavy lifting of figuring out what is the right chunking strategy. And even embedding models, we ended up having our own fine tuned embedded models,
209
00:19:44.560 --> 00:19:47.440
because it was German. Yeah. So we started building our own,
210
00:19:48.320 --> 00:19:54.675
embedding models back then, which which proved with better accuracy than the off the shelf models back then,
211
00:19:55.315 --> 00:20:03.415
from OpenAI. So but once we started figuring this out, then the next country came in, which which was, I I believe rightly, it was Croatia.
212
00:20:03.715 --> 00:20:08.775
And then you needed to abstract away what's the best way for the Croatian
213
00:20:09.660 --> 00:20:16.700
knowledge ingestion pipeline to be created such that the data scientists and the people who know Croatian, which is a Croatian telecom,
214
00:20:17.660 --> 00:20:32.235
group, can manage that pipeline. And then we built World Soul into in in such a manner that they can play around with multiple chunking strategies, not on a UI level because these are small atomic units, which was built on so
215
00:20:32.855 --> 00:20:33.595
there is DVC,
216
00:20:33.895 --> 00:20:39.909
the DVC pipelines. Right? So this Woodrow was built on top of DVC, and you have something called as
217
00:20:40.210 --> 00:20:43.030
the the Woodrow steps or the Woodrow routes here.
218
00:20:43.409 --> 00:20:53.445
One of the step is there will be a default chunking strategy step, which is, again, not exposed as a UI. It's it's a simple module, which is a boot sales step, which is a Python program. And
219
00:20:53.745 --> 00:20:57.205
you can bring in your favorite library, which has
220
00:20:57.585 --> 00:21:08.679
better utilities for the chunking strategies that you might want, import it into this unit, and try it out, and then plug it into rest of the pipeline, and it will work. So it's like the UNIX pipes approach,
221
00:21:09.460 --> 00:21:13.320
that we
use to build the Woods Hole pipeline. So this flexibility
222
00:21:13.700 --> 00:21:25.295
was crucial if you want to expose it to other people who know the language because, otherwise, you are limited by the engineering group who might know only one language. So that's the way we started to,
223
00:21:25.755 --> 00:21:41.560
expand it to then Hungarian and Croatian so that the data scientists in those group are not working as a black box with some UI drop down chunking strategy, but they can bring in their favorite library into the step and only change that. Rest of the pipeline remains the same.
224
00:21:42.180 --> 00:21:46.205
So based on the work that you did at Deutsche Telecom, you ultimately
225
00:21:46.905 --> 00:21:48.365
created the LMOS
226
00:21:49.065 --> 00:21:50.205
open source project.
227
00:21:50.585 --> 00:21:52.684
And one of the perennial challenges
228
00:21:53.145 --> 00:22:01.970
of releasing something as open source that you first built for a particular organizational use case is that the design of the system as it's implemented
229
00:22:02.270 --> 00:22:03.090
in the business
230
00:22:03.549 --> 00:22:10.289
implicitly encodes a lot of the organizational patterns of that organization, which don't necessarily translate externally.
231
00:22:10.635 --> 00:22:19.855
And so I'm wondering how you approached that translation process of figuring out what are the pieces that are generally applicable, but not so general that they're useless,
232
00:22:20.235 --> 00:22:33.360
and then design it in a way that it can be used as a foundational substrate that other people can build on top of without making it so complicated or confusing that they're never going to start down that path?
233
00:22:33.900 --> 00:22:37.360
That's a very sharp question. It's like the Conway's law design
234
00:22:38.220 --> 00:22:41.085
Conway's law design of a framework. Right?
235
00:22:41.545 --> 00:22:54.510
Right? So two things that really, really worked out well in terms of reaching a just enough point in abstraction is in order to mention this, there's a few things. Yeah. For example, when we started in, 2023,
236
00:22:54.750 --> 00:22:59.150
there were these frameworks. Slack chain was a major one in 2023.
237
00:22:59.150 --> 00:23:03.970
We started picking Lang chain, but almost all our transactional systems and APIs,
238
00:23:04.270 --> 00:23:07.890
the the profile API, the the the billing API,
239
00:23:08.385 --> 00:23:12.005
All of that were built in JVM stack. The second point
240
00:23:12.385 --> 00:23:14.325
is almost the entire operational systems
241
00:23:15.425 --> 00:23:18.085
with distributed systems were built on the Kubernetes
242
00:23:18.545 --> 00:24:05.315
CNCF stack. So you have the Kubernetes, the observability, the Grafana, the Prometheus, and everything. Now comes a new framework which was invented somewhere else based on some use cases. Now how do you fit this in into the people who actually know the data, Do know the domain, and what happens to all the client SDKs that has been built? What's the reason we went back to the drawing board in starting creating Elmos? And and we did not stand out creating Elmos by wanting to create a framework. The problem statement was quite simply this. We we implemented something in Langchain. It took a couple of months, and after that, no one knew how to build it into a platform because it was so chaotic. And how does it actually fit in into the rest of the power stack? So we went back to the drawing board and came down to the realization. Only if we provide
243
00:24:05.855 --> 00:24:25.090
the right amount of tooling to the people who know these APIs and domains and let them build it. That's the only way this will scale. Otherwise, you'll have a new team building something else, and then you need to ask data from this team and Jira tickets and this and that, Conway's form. Now second point is the only thing that you can do is
244
00:24:25.465 --> 00:24:40.685
you cannot state that this is how the framework should be. What you can only build is how to shorten that loop of doing an experiment. Let's say changing something in an agent and how to take it to staging, for example. For that, you need to have a robust pipeline from the time a developer
245
00:24:41.250 --> 00:24:42.710
changes some behavior
246
00:24:43.010 --> 00:24:47.830
in an agent. An ephemeral environment is immediately spun up to test this in isolation.
247
00:24:48.130 --> 00:25:01.045
And this is all coming in from the distributed systems back up, right, from the Kubernetes world. So this is how we approached it and not as a framework, but how do we shorten the feedback loop for testing because no one knows how to build these systems reliably.
248
00:25:01.505 --> 00:25:13.200
So which resulted in the stack which goes not as a framework. It's not Elmos is not a framework. Elmos has something called as ARC, which is the agent reactor as we call it. Right? Like the
249
00:25:13.820 --> 00:25:16.000
the Jarvis ARC. It was built on Kotlin
250
00:25:16.540 --> 00:25:20.559
so that we could build a DSL, which is just enough information
251
00:25:21.034 --> 00:25:31.215
for the engineers who know their APIs who are on Java to build agents, not having to figure out hundreds of new APIs and Spring AI and this and that. Then these agents
252
00:25:31.595 --> 00:25:36.174
need to live somewhere, which is a life cycle management. So we build the Elmos platform
253
00:25:36.620 --> 00:25:52.595
to deploy these agents with one Git push, for example. Right? It spins up an ephemeral environment. And in that Elmos platform, it's entirely Kubernetes based, which means you're not reinventing the wheel. Agents were created as first class citizens there. So you could do kubectl
254
00:25:52.975 --> 00:25:54.434
get agents. And then
255
00:25:54.735 --> 00:25:59.155
life cycle management of an agent is taken care. When you push an agent, you could say,
256
00:25:59.695 -->
00:26:02.595
I'm a billing agent. I can handle billing queries,
257
00:26:03.270 --> 00:26:05.210
and I can handle billing disputes.
258
00:26:05.990 --> 00:26:11.690
I advertise this as an agent. I advertise it to the network when I am deployed into the Kubernetes,
259
00:26:12.310 --> 00:26:16.410
platform. So you use the discovery mechanisms, which is already proven in Kubernetes,
260
00:26:16.790 --> 00:26:22.305
right, through STOs of the world, and then bring that into the Kubernetes registry,
261
00:26:22.765 --> 00:26:25.265
the STO registry. So you're not reinventing
262
00:26:25.805 --> 00:26:28.865
an a two a registry or something like that. It's all
263
00:26:29.405 --> 00:26:33.919
stack which enterprises and operational teams are familiar with. And this,
264
00:26:34.700 --> 00:26:36.639
the Elmo stack is
265
00:26:37.019 --> 00:26:47.919
should be universally applicable for distributed systems teams without trying to do too much into ARC. And Woodzel, for example, is also similar. So we knew exactly where we wanna stop.
266
00:26:48.424 --> 00:26:54.044
If you try to build a UI and drop down with, chunking strategy and this and that,
267
00:26:54.424 --> 00:26:59.325
it will immediately lose the flexibility of bringing in the best, you know, Python
268
00:26:59.705 --> 00:27:04.909
frameworks or stacks, which is on the right pipeline side. So we just stopped at that, which allows
269
00:27:05.289 --> 00:27:13.070
and picked DVC for the point in time recovery and all that. So it's also bet on large scale ingestion pipelines based on Kubernetes,
270
00:27:13.370 --> 00:27:17.149
etcetera. So it's universally applicable in enterprises would be my answer.
271
00:27:17.825 --> 00:27:18.325
Interestingly,
272
00:27:18.945 --> 00:27:19.765
the LMOS
273
00:27:20.145 --> 00:27:20.645
acronym
274
00:27:21.105 --> 00:27:23.605
is language model operating system.
275
00:27:23.985 --> 00:27:24.965
Operating systems
276
00:27:25.665 --> 00:27:38.669
are a ubiquitous concept in computing for decades now. I'm curious what you were thinking in terms of the naming about what you're trying to convey with that concept of operating systems for this language model context
277
00:27:38.970 --> 00:27:40.190
and some of the core
278
00:27:40.570 --> 00:27:42.990
theories around operating system design
279
00:27:43.450 --> 00:27:46.505
that you brought into this framework to enable
280
00:27:46.885 --> 00:27:48.025
such a generalized
281
00:27:48.645 --> 00:27:51.145
substrate for agentic use cases?
282
00:27:52.245 --> 00:27:52.965
Yeah. This is,
283
00:27:53.445 --> 00:27:57.125
this is clearly one of the reasons I'm starting the start up itself. Yeah. Because,
284
00:27:57.529 --> 00:28:05.149
I was so fascinated by the idea when AI took off. Different people saw it as, oh, it's magic. Right? But there's
285
00:28:05.450 --> 00:28:11.950
I I love thinking in terms of fundamental abstractions, either in physics or biology or computational systems.
286
00:28:12.315 --> 00:28:12.815
And,
287
00:28:13.115 --> 00:28:18.255
yeah, I I used to wonder, oh, I was I wasn't there when Linux was born or something.
288
00:28:18.875 --> 00:28:25.695
As in, so it was like a new way when computing was emerging on how do you build programs. For example,
289
00:28:26.010 --> 00:28:30.990
when I first interacted with the language models, it felt like now you have a new microprocessor,
290
00:28:31.770 --> 00:28:35.550
and this is made up of some magical silicon. So instead of
291
00:28:36.090 --> 00:28:36.590
precise,
292
00:28:37.210 --> 00:28:39.815
you know, x 86 instructions, you
293
00:28:40.195 --> 00:28:40.695
can
294
00:28:41.075 --> 00:28:42.375
give natural language instructions,
295
00:28:42.835 --> 00:28:47.014
and it's gonna chunk out some response into your registry. So now,
296
00:28:47.315 --> 00:28:49.095
suddenly, the it it flips.
297
00:28:49.475 --> 00:28:51.815
Oh, this would mean you would need to build
298
00:28:52.115 --> 00:28:52.615
new
299
00:28:53.130 --> 00:28:53.630
programming
300
00:28:53.930 --> 00:28:55.390
and operating constructs
301
00:28:55.770 --> 00:29:03.950
from the scratch to build a new computational unit, which could be agents is was the thought process. So so that shift from language models
302
00:29:04.490 --> 00:29:05.230
as microprocessors
303
00:29:05.770 --> 00:29:07.070
resulted into thinking,
304
00:29:07.485 --> 00:29:18.225
let's build all the layers above starting with how do you interact with these models. And models are going to emit strings as tokens. And these strings and tokens need to
305
00:29:18.765 --> 00:29:24.510
control the program flow, which is a totally different programming paradigm. Right? And how do you build
306
00:29:24.890 --> 00:29:34.670
scheduling on top of it? For example, let's say there is a request coming in, and it need to do some planning while at the same time, the same program need to
307
00:29:35.075 --> 00:29:50.640
give response to a task. So you need a scheduler which optimizes the resources, which in this case is the language models, which could be either cost or unit of time. And and all those constructs needed to be revisited in an and how do you handle nondeterminism
308
00:29:51.180 --> 00:29:55.200
fundamentally in a program? So that resulted in this thought process of
309
00:29:55.580 --> 00:29:56.080
Elmos,
310
00:29:56.460 --> 00:29:57.280
and we thought
311
00:29:57.660 --> 00:29:58.960
Linux had the Tux,
312
00:29:59.340 --> 00:30:00.480
mascot. So
313
00:30:01.755 --> 00:30:04.895
let's, let's bring in Sesame Street Elmo
314
00:30:05.195 --> 00:30:10.495
into this being the next stack for agentic computing. This was a vision in 2023
315
00:30:11.115 --> 00:30:25.630
as in, like, like the Xerox part group. Yeah. This was how most of the engineers joined the team as well when, you know, personal computing taken off and and, Xerox Spark came up with object oriented programming and all that. So we were a couple of
engineers
316
00:30:25.930 --> 00:30:34.375
passionate about building something great. And then, oh, let's let's do something. Let's build the foundations for agent computing, and it's called Elmos. Let's build
317
00:30:34.755 --> 00:30:46.900
agent communication protocols, agent computational units. Let's build the scheduler for interacting with language models, while at the same time solve the customer use cases for Deutsche Telekom. So this was the background,
318
00:30:47.440 --> 00:30:49.620
and the storyline behind, Elmos.
319
00:30:50.880 --> 00:30:55.140
One of the other interesting challenges that has become exacerbated
320
00:30:55.600 --> 00:30:58.155
by the scale of capabilities
321
00:30:58.535 --> 00:31:05.995
of these language models and the high degree of variability in terms of their pricing structures and the complete unpredictability
322
00:31:06.375 --> 00:31:11.515
in terms of the number of input and output tokens that are used for the context.
323
00:31:12.059 --> 00:31:15.519
How do you think about reporting on and managing
324
00:31:15.820 --> 00:31:16.320
the
325
00:31:16.620 --> 00:31:24.159
costs and budgeting of these different agents to make sure that you're not going to bankrupt the company in the process of solving their problems?
326
00:31:25.184 --> 00:31:45.360
Absolutely. This is so, essentially, this is also this need to be fundamentally looked at from a computational point of view. Most of the agent building process these days falls into the category of let's use LLM invocations for every computation. So let's assume if you're if if there is a flow when a customer
327
00:31:45.980 --> 00:31:47.440
requests a refund,
328
00:31:47.899 --> 00:31:52.935
then call the refund API, then get the response, then then do the,
329
00:31:53.495 --> 00:31:55.515
account update API, etcetera, etcetera.
330
00:31:55.895 --> 00:31:59.195
What is being observed today is most people are using LLMs
331
00:31:59.735 --> 00:32:03.195
all the time for this invocation. When you take a step back
332
00:32:03.735 --> 00:32:12.649
and and and and and and think about how computation works in nature for energy conservation, we don't try to do we don't try to, you know, reinvent
333
00:32:13.110 --> 00:32:18.250
the same process using our brains all the time once it becomes deterministic
334
00:32:18.710 --> 00:32:22.855
or like the habit formation loop, you you put it into the low energy
335
00:32:23.315 --> 00:32:25.175
execution phase. So
336
00:32:25.715 --> 00:32:28.855
for agents too, this sort of a paradigm need to
337
00:32:29.235 --> 00:32:32.215
emerge is is the way I'm thinking as well. So
338
00:32:32.520 --> 00:32:46.940
instead of invoking LLMs all the time, if the LLM has figured out or the agent has figured out 80% of the use cases can be solved by this deterministic flow, the agent itself constructs that deterministic flow. And for all the invocations
339
00:32:47.320 --> 00:32:56.615
that comes in, uses this deterministic flow and then falls back into this rest 20% flow. But all of this requires rigorous instrumentation
340
00:32:57.315 --> 00:33:04.049
into looking into costs. So, essentially, just like in the in the operating system. Right? For each process, you assign
341
00:33:04.350 --> 00:33:09.570
CPU and memory, and then you do the stamping and observation, the process IDs.
342
00:33:10.110 --> 00:33:14.529
You would need to think that unit, in this case, is agent,
343
00:33:15.015 --> 00:33:18.395
such resource allocation, and then monitoring it, and
344
00:33:18.775 --> 00:33:30.530
then massive observability platforms, intelligent decision making platforms keeping in check on this. So two things. It's like the OODA loop. Right? Observe you need to keep observing what is going on, and then
345
00:33:30.830 --> 00:33:42.625
you need to make the decisions on what need to be optimized. In this case, computation need to be optimized. It doesn't make any sense to do LLM calls all the time. It is soon going to come
346
00:33:43.085 --> 00:33:45.345
crashing is what I what I bet.
347
00:33:46.285 --> 00:33:54.785
And then in terms of the focus on openness and local control for businesses that want to be able to
348
00:33:55.085 --> 00:33:57.300
capitalize on these agent capabilities,
349
00:33:57.680 --> 00:34:15.705
the ability to use local models for more of that cost control or more predictability in terms of being able to freeze or pin to a specific model version so that you're not at the whim of whatever the model provider is going to do under the covers of their API. I'm curious how you see the work that you're doing
350
00:34:16.085 --> 00:34:16.585
complementing
351
00:34:17.205 --> 00:34:20.185
or overlapping with other projects in the ecosystem
352
00:34:20.485 --> 00:34:22.425
such as Oomi or OUMI,
353
00:34:22.980 --> 00:34:25.880
which is focused more on enabling organizations
354
00:34:26.420 --> 00:34:32.280
to build their own foundation models and control more of that element of the life cycle as well.
355
00:34:33.140 --> 00:34:34.680
Yeah. So this is also
356
00:34:35.214 --> 00:34:42.595
one of the reasons. So I'm going into this entrepreneurial journey on on betting on also one of these key constructs.
357
00:34:42.895 --> 00:34:45.395
What we have observed is in enterprises,
358
00:34:46.575 --> 00:34:53.329
once a pattern has been set in let's assume if somebody has built in with the best model available for some use cases.
359
00:34:53.630 --> 00:35:02.609
No. There is no incentive for it to be shifted later. No. In enterprises, typically, once once something has been set, it's very difficult to shift. But the worrisome
360
00:35:02.990 --> 00:35:05.174
point might be all this information
361
00:35:05.635 --> 00:35:06.375
in terms
362
00:35:06.755 --> 00:35:10.535
of the feedback loop. For example, if you had control on
363
00:35:10.914 --> 00:35:28.180
the fine tune models or or getting information on the model outputs that
you need to rigorously track, This is a wealth of information that you could use as we move forward to do fine tuning and maybe bringing down the computational cost and also building better intelligence
364
00:35:28.480 --> 00:35:36.275
for your organization. And that opportunity is missed if you keep betting on these large models. So for that to happen,
365
00:35:36.575 --> 00:35:51.779
you need to have a platform or a layer which allows this mitigation strategy to be baked in. So when you call the completions API, instead of directly going into, let's say, OpenAI or Gemini or whatever it is, there should be a way in which
366
00:35:52.079 --> 00:36:12.155
this is mitigated by that semi proxy layer, which allows quickly shifting different models to be plugged in. And, essentially, as OpenAI recently has shown with the responses API, which is a higher order API, there is a beautiful example where OpenAI shows with three lines of code, you are writing a call command, call, and say, add
367
00:36:12.455 --> 00:36:12.955
this
368
00:36:13.580 --> 00:36:16.800
toner pad to my shopping cart, to a Shopify
369
00:36:17.260 --> 00:36:27.920
MCP server tools which are plugged into the responses API. This is the only only, input that was given, and you configure the MCP server. There is no additional code. It does the agent decorecastration,
370
00:36:28.460 --> 00:36:31.235
calls the search a product, add a product, and,
371
00:36:31.795 --> 00:36:32.535
check out
372
00:36:32.835 --> 00:36:34.615
a shopping cart. All of that,
373
00:36:35.075 --> 00:36:37.175
by that agent deco. This happens
374
00:36:37.555 --> 00:36:40.215
underneath the platform. So if organizations
375
00:36:41.250 --> 00:36:57.395
start to use these APIs, it is super simple. The the the simplicity of such an API and the value it brings in is tremendous. You don't you could write any any number of use cases in a in any given day. But the problem is if you don't have the mitigation strategy
376
00:36:57.775 --> 00:37:00.435
of how your organization is orchestrated
377
00:37:00.975 --> 00:37:03.395
to to to some some black box API,
378
00:37:03.775 --> 00:37:12.420
you will miss out on a lot of information which you could have preserved for the time when model tuning is gonna become much cheaper,
379
00:37:12.720 --> 00:37:17.539
and you would soon be tied to these large model companies and just consuming
380
00:37:17.920 --> 00:37:40.600
some black box higher order API. So the layer that I'm also building is something like the responses API. One of the components that we are building is this one, which allows the same OpenAI like structure, but it allows different models to be plugged in while you can be deployed on your premises. And this data is immediately accessible for you to fine tune your models. It's immediately traceable
381
00:37:41.060 --> 00:37:45.800
with the same simplicity as OpenAI that in four lines of code, you can build a number of orchestrations.
382
00:37:46.985 --> 00:37:51.565
One of the other interesting elements of this new world of agent and capabilities
383
00:37:52.025 --> 00:38:02.390
is the broad applicability of the problem spaces that they can be applied to. And I'm wondering how you see the potential for something like Elmos to
384
00:38:02.770 --> 00:38:12.550
be employed in the context of a data team to be able to build their own agents to help to manage some of the pipeline design implementation,
385
00:38:13.795 --> 00:38:15.175
do some of the, maybe,
386
00:38:15.555 --> 00:38:21.255
analytical queries that you want to expose to the business or some of the other end user
387
00:38:21.875 --> 00:38:22.375
capabilities
388
00:38:22.675 --> 00:38:28.455
that are particularly interesting or innovative that you have either conceived of or seen in action?
389
00:38:29.520 --> 00:38:31.440
Yeah. So, essentially, there was,
390
00:38:31.920 --> 00:38:39.300
recently a meetup for the Eclipse software defined vehicle group in, I think it was in Copenhagen. So where we demonstrated
391
00:38:39.920 --> 00:38:41.860
with Elmos, so the vehicles
392
00:38:42.320 --> 00:38:56.305
are emitting telemetry data. So the software defined vehicle group is a consortium of some of the major companies in Europe. They're focusing on telemetry data emitted from vehicles. Yeah. Software defined vehicles. So we demonstrated
393
00:38:56.925 --> 00:39:02.470
a simple agent, which was built with Elmos Arc, the agent developer framework, the agent,
394
00:39:02.930 --> 00:39:11.110
reactor framework, which could connect to this telemetry data. It was a time series database, and the query building was being done
395
00:39:11.410 --> 00:39:18.615
by the agent using OpenAI APIs. But the fun part is so you could ask questions like, how many of my
396
00:39:18.915 --> 00:39:32.990
vehicles are running low on fuel, for example, amongst maybe thousand vehicles, and all of that being returned in a few lines of code. So, essentially, what this actually means is for data analytics teams, it is all about querying.
397
00:39:33.369 --> 00:39:36.430
Right? Because the query language, it could be a SQL or
398
00:39:36.730 --> 00:39:45.915
a cipher or, a number of approaches are there. But this acts as a layer on top for them to quickly build this dynamic
399
00:39:46.295 --> 00:39:47.755
query creation agents
400
00:39:48.055 --> 00:39:57.829
even though the underlying systems won't change. You don't have to change your Athena. You don't have to change your Redshift. You don't have to change your in any of yours, your your snowflake.
401
00:39:58.210 --> 00:39:58.950
You could
402
00:39:59.250 --> 00:40:07.510
build an agent that constructs these queries because language models are also good in understanding the existing querying languages.
403
00:40:08.025 --> 00:40:10.765
And, it could be done pretty easily,
404
00:40:11.464 --> 00:40:12.444
as we have
405
00:40:12.825 --> 00:40:13.885
seen so far.
406
00:40:14.744 -->
00:40:19.405
And in your experience of building these agentic frameworks,
407
00:40:20.250 --> 00:40:21.950
figuring out how to
408
00:40:22.330 --> 00:40:29.950
manage the creation and maintenance of the context corpus to enable these agents to work effectively?
409
00:40:30.490 --> 00:40:34.830
What are some of the most interesting or unexpected or challenging lessons that you've learned in the process?
410
00:40:35.785 --> 00:40:37.885
Yes. Andrew Karpathy, you recently
411
00:40:38.345 --> 00:40:39.245
pointed out,
412
00:40:40.425 --> 00:40:45.085
yeah, every engineering information engineering is gonna converge into context engineering.
413
00:40:45.785 --> 00:40:47.160
So it's a man.
414
00:40:48.039 --> 00:41:07.025
Simple questions are easy to handle, but as the system grows in complexity, how do you pass the right context to the same agent or different agents? Yeah. There are many approaches which are emerging. Nobody has fully figured out this pattern yet because as it grows and scales, this this is gonna become challenging. But one of the things is if you have a platform
415
00:41:07.645 --> 00:41:10.625
or a layer let's take an ecommerce organization,
416
00:41:11.165 --> 00:41:18.119
a company with champions commerce. If that company either if different departments starts to build different agents, then already
417
00:41:18.420 --> 00:41:20.680
what you would end up is today's
418
00:41:20.980 --> 00:41:22.040
spaghetti systems.
419
00:41:22.340 --> 00:41:33.275
You would lose out on on deriving the true value of AI because the context is now segregated. So the true value of AI in enterprises is going to emerge only from unifying
420
00:41:33.655 --> 00:41:41.194
at least the foundation for the context so that other agents can be built. What that would mean is a big shift in enterprise architectures.
421
00:41:41.575 --> 00:41:42.234
The microservices
422
00:41:42.615 --> 00:41:43.115
world
423
00:41:43.494 --> 00:41:51.950
taught that small is good, small, nimble teams. They can go and do whatever that they want. But with AI, there is a problem because if you don't
424
00:41:52.250 --> 00:41:55.630
have a unified view of the truth of your enterprise
425
00:41:56.250 --> 00:41:56.990
stored somewhere,
426
00:41:57.290 --> 00:41:57.790
then
427
00:41:58.090 --> 00:42:02.605
the agents that are being built will not be as effective as
428
00:42:02.985 --> 00:42:07.325
if it had not been, which brings in the need for a core
429
00:42:07.705 --> 00:42:08.205
context
430
00:42:08.505 --> 00:42:09.725
and model management
431
00:42:10.105 --> 00:42:10.605
platform
432
00:42:10.905 --> 00:42:17.799
where you connect your enterprise tools so that your different departments and engineers can build easily without asking for permission.
433
00:42:18.260 --> 00:42:24.599
Hey. Can I use your department's data? Because what will matters is the business outcome. Suddenly, your orchestrator,
434
00:42:25.059 --> 00:42:27.640
if you say, I want to optimize my profits,
435
00:42:28.484 --> 00:42:29.545
for the teenagers,
436
00:42:30.085 --> 00:42:36.265
what suggestions exist. Now if the platform has the tools on, you know, you know, sales projections
437
00:42:36.964 --> 00:42:38.424
analysis tool, then
438
00:42:38.885 --> 00:42:40.984
buyer patterns tool, then
439
00:42:41.285 --> 00:42:50.590
inventory tool, and market analysis tool. Then suddenly with this question, immediately, the agent orchestration picks in. It's able to come up with immediate results.
440
00:42:50.970 --> 00:42:55.230
So this layer is the essence. If you're not building your organization
441
00:42:55.530 --> 00:42:56.910
for agent dec orchestration,
442
00:42:57.444 --> 00:42:58.424
that is the definition
443
00:42:58.724 --> 00:43:01.224
of AI native as I would call it, the architectural
444
00:43:01.845 --> 00:43:02.345
equivalent
445
00:43:02.645 --> 00:43:11.145
of an AI native. What is the definition of an AI native? How to prepare your organization to be a native? Models are gonna get better, cheaper,
446
00:43:11.550 --> 00:43:28.785
New paradigms will emerge in programming. But one thing certainly is not gonna change, which is you need to have this context building platform layer for the world which is going to come, And and that is going to be the core of your AI native transformation journey.
447
00:43:29.805 --> 00:43:32.385
And I think to extending on your
448
00:43:32.685 --> 00:43:34.385
reference to the microservices
449
00:43:34.765 --> 00:43:35.265
architecture
450
00:43:35.645 --> 00:43:40.880
is that we went through a similar problem where as you go from the monolith to the microservices,
451
00:43:41.820 --> 00:43:43.440
you still have to figure out
452
00:43:43.740 --> 00:43:47.840
where that state gets created and maintained and how it gets distributed.
453
00:43:48.300 --> 00:43:51.915
And then that also leads to all of the complexity of
454
00:43:52.315 --> 00:43:52.815
reconstituting
455
00:43:53.515 --> 00:43:54.975
all of the relevant context
456
00:43:55.275 --> 00:43:56.575
in the warehouse
457
00:43:57.035 --> 00:44:00.735
context where you pull the data out of all of these different microservices
458
00:44:01.115 --> 00:44:03.935
systems, and then you have to figure out what are the linkages,
459
00:44:04.670 --> 00:44:07.250
recreate some of the API based connections
460
00:44:07.630 --> 00:44:23.645
at the data layer to figure out how to bring it all back together to make it semantically meaningful. And we're in a similar stage with these AI systems where if you have that warehouse catalog, you can use that to a certain extent, but we need to figure out how do we feed a lot of these
461
00:44:23.945 --> 00:44:29.005
LLM interactions back into that to help maintain and grow and evolve the corpus
462
00:44:29.305 --> 00:44:34.730
without having it shard into these different domains that then have to be reconstituted
463
00:44:35.190 --> 00:44:37.450
if you even know that they exist
in the first place.
464
00:44:38.150 --> 00:44:38.650
Absolutely.
465
00:44:38.950 --> 00:44:51.915
Absolutely spot on. And that's what most people are I think it's it's from what I've seen and and some of the stories from other places, the spaghetti that we have seen in the previous world, it's only going to get multiplied.
466
00:44:53.175 --> 00:44:56.955
Everyone is building three agents on top of the existing microservice.
467
00:44:57.335 --> 00:44:59.435
Now the microservice is now four
468
00:44:59.895 --> 00:45:01.355
four units of computation,
469
00:45:01.809 --> 00:45:04.069
and no one knows the context anymore.
470
00:45:04.529 --> 00:45:05.029
My
471
00:45:05.329 --> 00:45:08.549
agent can only respond address customer address.
472
00:45:09.250 --> 00:45:19.734
Then then now you think about what protocol should I use, a two a, to connect to the order order thing? And and now you have scrum meetings, and it's insanity.
473
00:45:21.875 --> 00:45:25.575
Alright. For people who are figuring out how to
474
00:45:26.035 --> 00:45:30.615
build their own enablement layer for these agentic use cases in their organization,
475
00:45:31.610 --> 00:45:35.390
what are the cases where you would say that Elmos is the wrong choice?
476
00:45:36.890 --> 00:45:48.055
So Elmos has multiple components. So let's assume if you're so Elmos itself has the agent framework per se, which is around Kotlin. So if you're a non Kotlin JVM organization,
477
00:45:48.915 --> 00:45:52.055
it doesn't make sense to use that agent framework. But,
478
00:45:52.755 --> 00:45:53.415
but then
479
00:45:53.715 --> 00:45:58.300
the platform layer, it's still an incubation phase. It's standard Kubernetes.
480
00:45:58.760 --> 00:46:04.620
It at least will provide a lot of inspiration for you to pick up irrespective of whether you run Python agents
481
00:46:04.920 --> 00:46:09.420
or so. So organizations who are not into JVM and Kubernetes
482
00:46:10.645 --> 00:46:11.145
or,
483
00:46:11.605 --> 00:46:17.465
if it's entirely based on Python or c sharp, then it might not make sense. But there is still a lot to learn,
484
00:46:17.845 --> 00:46:23.225
on how to manage the life cycle of agents. But that's also the reason why the learnings
485
00:46:23.750 --> 00:46:34.730
and also where it goes is it's going beyond agent frameworks and which is one of the components which is built under the new startup is is, is, is the Mesioc open responses component.
486
00:46:35.030 --> 00:46:35.770
So, essentially,
487
00:46:36.309 --> 00:46:38.490
it does not restrict you
488
00:46:38.855 --> 00:46:45.035
from using any framework. As I described earlier, it's the complete the responses API just like OpenAI.
489
00:46:45.575 --> 00:46:51.595
It provides once it's deployed this component, every agent framework that you can think of, OpenAI agent SDK
490
00:46:52.080 --> 00:46:52.580
and,
491
00:46:52.960 --> 00:46:56.100
AutoGen or Kotlin or whatever should work
492
00:46:56.400 --> 00:46:56.900
because
493
00:46:57.200 --> 00:46:57.940
this addresses
494
00:46:58.320 --> 00:47:06.675
this problem of context building for large scale enterprises and model switching. So it goes just if if if the OpenAI developer platform,
495
00:47:07.055 --> 00:47:16.275
if it were to be deployed on your premise, what would it be like? That's that's the way we are thinking. That's the essence of what we need to build in your organization,
496
00:47:16.655 --> 00:47:19.110
irrespective of whether you use this component or not.
497
00:47:20.310 --> 00:47:21.930
And as you continue
498
00:47:22.950 --> 00:47:32.330
to contribute to Elmos and continue along your own venture that you're building, what are some of the new capabilities that you have planned? What are some of the
499
00:47:32.925 --> 00:47:33.425
ecosystem
500
00:47:33.965 --> 00:47:36.705
evolutions that you're paying particular attention to
501
00:47:37.005 --> 00:47:39.505
or any of the capabilities that you're particularly
502
00:47:39.805 --> 00:47:43.025
excited to dig into building and growing?
503
00:47:44.445 --> 00:47:48.260
Yeah. I think the on the Elmos side, there are now two initiatives.
504
00:47:48.560 --> 00:47:50.340
One is Elmos, which is on under,
505
00:47:50.640 --> 00:47:51.780
the Eclipse Foundation,
506
00:47:52.080 --> 00:47:54.500
and we are listening to the industry
507
00:47:54.880 --> 00:48:00.260
as well on what is actually required and not try to make it bloated to become,
508
00:48:00.655 --> 00:48:17.800
you know, let's add this feature or that feature. So it's kind of at least the agent framework is very stable. Kotlin is used in production. The Elmos platform, we originally built it as you should be able to deploy any agent to be Python or this into and and convert them into
509
00:48:18.180 --> 00:48:20.200
agents as first class citizens in Kubernetes.
510
00:48:20.660 --> 00:48:21.560
This, we are
511
00:48:21.940 --> 00:48:23.880
rethinking the approach because
512
00:48:24.260 --> 00:48:25.880
if you try to fit for everybody,
513
00:48:26.340 --> 00:48:33.744
it will never work. So we just wanna convert that into a minimalistic registry based on Kubernetes, which does ex external orchestration.
514
00:48:34.204 --> 00:48:34.704
So
515
00:48:35.085 --> 00:48:35.585
agents
516
00:48:35.964 --> 00:48:45.184
for JVM stack, Elmos should hold true for today. I would say it's it's it's one of the best frameworks out there, which allows business
517
00:48:45.860 --> 00:48:46.520
and engineers
518
00:48:47.380 --> 00:48:53.160
to allow the same thing, which is very rare because a business is able to use if you're using Eleazar,
519
00:48:53.700 --> 00:49:00.040
the business can define the use case. I want to define the billing dispute use case. The business would write it in English language,
520
00:49:00.375 --> 00:49:06.315
and we are able to do it because if you just randomly write prompts, it will never work. So you need a runtime
521
00:49:06.615
--> 00:49:07.755
for the English language,
522
00:49:08.215 --> 00:49:16.660
meta language, which we created, which is referred to as ADL in ARC, so it works. The part which I'm most excited about is is this
523
00:49:17.040 --> 00:49:17.859
open responses
524
00:49:18.480 --> 00:49:18.980
component,
525
00:49:19.359 --> 00:49:21.619
which is being built by my cofounders,
526
00:49:22.560 --> 00:49:23.700
Jasbir and Amanth.
527
00:49:24.000 --> 00:49:25.460
They are building this component
528
00:49:26.160 --> 00:49:32.994
to become, like I mentioned, that that orchestration layer for models and context building layer for the whole enterprise
529
00:49:33.615 --> 00:49:36.115
so that your developers, your departments,
530
00:49:36.575 --> 00:49:43.740
no one is gonna change the department's structure. They should still build agents on their favorite stack, but the minimum guidance
531
00:49:44.120 --> 00:49:44.860
is use
532
00:49:45.400 --> 00:49:46.940
to to to prepare yourself
533
00:49:47.400 --> 00:49:49.020
to change models quickly
534
00:49:49.480 --> 00:49:58.775
and don't lose context. You should have the ability to connect to any number of tools, and don't you bring your siloed tool registry and all that. That layer is called the Mosaic,
535
00:49:59.315 --> 00:50:03.095
open responses layer. And by the way, Mosaic is the name of
536
00:50:03.555 --> 00:50:05.075
the company I'm building, and,
537
00:50:05.555 --> 00:50:07.895
Mosaic stands for multi agent systems
538
00:50:08.355 --> 00:50:15.539
MAS. So this is the thing which I'm most looking for. It's also open core, open core model as open source, then building on top. Yeah.
539
00:50:16.799 --> 00:50:25.845
Are there any other aspects of the work that you're doing on Elmos or Mosaic or the overall space of enabling agentic use cases,
540
00:50:26.145 --> 00:50:31.445
all of the data requirements that go into it that we didn't discuss yet that you'd like to cover before we close out the show?
541
00:50:32.225 --> 00:50:44.299
Yeah. I think we went into the idea of orchestration. Yeah. So I would say everything revolves around the idea of orchestration from here on. And the idea of orchestration is simply feedback loops and feedback loops of experimentation,
542
00:50:44.920 --> 00:50:50.619
like in real world, yeah, as in how the the Apollo model versus the SpaceX model.
543
00:50:51.055 --> 00:50:53.775
In Apollo model, failure was not an option, and,
544
00:50:54.415 --> 00:50:55.395
SpaceX model
545
00:50:55.695 --> 00:50:58.835
failed fast and learned. So orchestration is
546
00:50:59.135 --> 00:51:03.475
a programming engineering, systems engineering view, which is gonna transform organizations.
547
00:51:04.000 --> 00:51:08.260
And it's more than a pro programming construct. It's a mindset
548
00:51:08.640 --> 00:51:11.780
construct and behavioral change inducing thing
549
00:51:12.160 --> 00:51:13.540
is the thing which enterprises
550
00:51:13.840 --> 00:51:16.420
need to think through as well if you try to retrofit
551
00:51:16.795 --> 00:51:20.015
your existing processes and and your ways of working
552
00:51:20.395 --> 00:51:36.589
into using any framework and AI, it will not work. It should be used as a lever to become space excess of the wall so that you focus only on one thing, which is feedback loop shortening and learning and reapplying. That's, AI nativity part, I would rather say.
553
00:51:37.609 --> 00:51:53.265
Well, for anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gap in the tooling or technology that's available for data management today.
554
00:51:54.365 --> 00:51:56.305
I think I have my personal
555
00:51:56.760 --> 00:52:01.420
opinion on see, first of all, I have seen large corporations try to see essentially,
556
00:52:01.720 --> 00:52:05.660
it's time to take out things rather than adding more things. This is
557
00:52:06.280 --> 00:52:22.285
this is the way I would rather see it. Yeah. Every tooling that has come in has resulted in there is transactional system, then there is data pipeline, then there is some other team sitting somewhere building some new other tools. And when a business stakeholder asks, hey. What is going wrong?
558
00:52:22.700 --> 00:52:43.255
Now you're deracing Jira tickets between five teams. And every team would say I have the greatest framework and the tool. So I think it is time for a cleansing, and it's more like adding more but taking things out. For example, can there be new kind of systems that can be built, which can be queried both transactionally and operationally because agents might be able to reconstruct the truth,
559
00:52:43.575 --> 00:52:44.395
using computation
560
00:52:44.775 --> 00:52:49.035
as well, for example. So I would if I were to frame it as one statement,
561
00:52:49.575 --> 00:52:50.875
it's over engineering
562
00:52:51.175 --> 00:52:58.610
has crept up the data space as well. The it's time to clean it up, and AI provides a good lever to do it.
563
00:52:58.990 --> 00:53:06.450
Alright. Well, thank you very much for taking the time today to join me and share the work that you've been doing on building the Elmos and
564
00:53:07.095 --> 00:53:14.315
the thought that you've put into how to enable the creation and maintenance of these agentic use cases at organizational
565
00:53:14.615 --> 00:53:30.570
scales. Definitely a very interesting and complex and timely problem, so I appreciate the time and energy that you're putting into making that more tractable for everyone else. Thank you, Topias. I really enjoyed the conversation as well. So thanks for the invite, and thanks for the thoughtful questions.
566
00:53:38.295 --> 00:53:42.615
Thank you for listening, and don't forget to check out our other shows. Podcast.net
567
00:53:42.615 --> 00:53:51.515
covers the Python language, its community, and the innovative ways it is being used. And the AI Engineering Podcast is your guide to the fast moving world of building AI systems.
568
00:53:52.350 --> 00:54:02.430
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email hosts@dataengineeringpodcast.com
569
00:54:02.430 --> 00:54:08.645
with your story. Just to help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.