Governing AI That Keeps Evolving With Maryam Ashoori (VP of Product and Engineering at IBM watsonx.governance)

1
00:00:06,557 --> 00:00:08,427
Welcome to our AI Explained.
2
00:00:08,541 --> 00:00:10,831
today we have a very special guest, uh,
3
00:00:10,901 --> 00:00:15,261
coming on our webinar, uh, Maryam Ashoori.
4
00:00:15,261 --> 00:00:16,661
Uh, she's the VP of Product and
5
00:00:16,661 --> 00:00:19,511
Engineering at IBM Watsonx Governance,
6
00:00:20,361 --> 00:00:22,321
where, uh, she's spearheads
7
00:00:22,341 --> 00:00:24,081
product delivery and strategy.
8
00:00:24,791 --> 00:00:26,461
And prior to that, she led engineering
9
00:00:26,461 --> 00:00:29,681
teams at Lyft and, and, uh, spent, you
10
00:00:29,681 --> 00:00:32,081
know, many years at IBM Research and,
11
00:00:32,881 --> 00:00:34,491
and worked a lot in AI and quantum
12
00:00:34,491 --> 00:00:36,901
computing, holds a PhD in systems
13
00:00:36,901 --> 00:00:38,791
design and engineering at Waterloo.
14
00:00:39,501 --> 00:00:41,311
So we're very excited to bring,
15
00:00:41,421 --> 00:00:44,211
uh, a veteran in AI and an expert.
16
00:00:44,271 --> 00:00:46,241
Uh, Maryam, welcome to the podcast.
17
00:00:46,941 --> 00:00:48,207
Krishna, thanks for having me
18
00:00:48,757 --> 00:00:49,277
Absolutely.
19
00:00:49,727 --> 00:00:52,267
So  Maryam, um, you've, you've
20
00:00:52,407 --> 00:00:55,247
probably seen AI from enterprise AI,
21
00:00:55,247 --> 00:00:56,767
from traditional machine learning to
22
00:00:56,767 --> 00:00:58,937
generative AI, and now agentic systems.
23
00:00:59,857 --> 00:01:01,647
have you seen the, like, the
24
00:01:01,647 --> 00:01:03,257
fundamental changes each time, you
25
00:01:03,257 --> 00:01:04,727
know, in these transformations?
26
00:01:05,253 --> 00:01:08,483
Even prior to that, Krishna, funny
27
00:01:08,493 --> 00:01:10,693
things, 20 years ago I did multi-agent
28
00:01:10,693 --> 00:01:12,873
systems, like two master's degree,
29
00:01:12,903 --> 00:01:15,463
two dissertation working on MAS well
30
00:01:15,473 --> 00:01:17,473
before they were under spotlight by
31
00:01:21,303 --> 00:01:21,883
LLM.
32
00:01:23,103 --> 00:01:23,218
So we've seen it all from traditional,
33
00:01:23,218 --> 00:01:24,043
like with agents being on the papers
34
00:01:24,063 --> 00:01:26,703
to generative AI to agentic AI.
35
00:01:27,593 --> 00:01:28,833
But you know what's interesting?
36
00:01:28,843 --> 00:01:30,683
The lesson that I've learned over the
37
00:01:30,683 --> 00:01:33,663
years is not to follow technology,
38
00:01:33,673 --> 00:01:36,393
follow the trends that solve problems.
39
00:01:36,833 --> 00:01:38,583
And when you look into enterprise
40
00:01:38,583 --> 00:01:42,533
challenges, the core to those
41
00:01:42,533 --> 00:01:44,663
challenges haven't changed much, like
42
00:01:44,693 --> 00:01:46,613
understanding the risk, governance.
43
00:01:46,953 --> 00:01:48,723
Like GRC, governance, risk,
44
00:01:48,723 --> 00:01:50,253
and compliance, for 20 years
45
00:01:50,253 --> 00:01:51,653
companies have been doing it.
46
00:01:51,953 --> 00:01:53,783
The nature of risks changes, but
47
00:01:53,783 --> 00:01:55,493
the foundation doesn't change.
48
00:01:56,263 --> 00:01:58,853
And I think, um, there are a lot of
49
00:01:58,853 --> 00:02:00,523
excitement on where it's going, but
50
00:02:00,523 --> 00:02:04,833
also lots of uncertainty along with
51
00:02:06,623 --> 00:02:09,163
the foundations that are not uncertain
52
00:02:09,173 --> 00:02:12,323
that much that we can potentially go
53
00:02:12,323 --> 00:02:14,783
and look into and focus and overcome
54
00:02:14,783 --> 00:02:17,923
a lot of, um, challenges that I keep
55
00:02:17,983 --> 00:02:20,493
observing in the market for the enterprise
56
00:02:21,467 --> 00:02:22,157
Yeah, absolutely.
57
00:02:22,337 --> 00:02:24,777
So, uh, you know, the fundamental
58
00:02:24,817 --> 00:02:26,497
difference that we have observed is,
59
00:02:26,497 --> 00:02:28,367
of course, models make predictions.
60
00:02:28,547 --> 00:02:30,157
You know, large language models
61
00:02:30,157 --> 00:02:32,367
came about and sort of saw
62
00:02:32,367 --> 00:02:33,857
all that content generation.
63
00:02:34,647 --> 00:02:36,077
agents are taking actions, right?
64
00:02:36,467 --> 00:02:38,517
So how does, how did, like, trust
65
00:02:38,517 --> 00:02:40,667
and governance change across
66
00:02:40,697 --> 00:02:42,057
all of these different phases?
67
00:02:42,057 --> 00:02:43,157
You know, like what, what have
68
00:02:43,157 --> 00:02:44,277
you seen in terms of that?
69
00:02:45,689 --> 00:02:47,589
Um, back to foundations.
70
00:02:47,699 --> 00:02:49,439
The foundations doesn't change much.
71
00:02:49,469 --> 00:02:52,659
Like, when I think about what makes trust,
72
00:02:53,119 --> 00:02:55,479
um, it's really the three layers that I've
73
00:02:55,479 --> 00:02:57,219
been thinking about and talking about.
74
00:02:57,829 --> 00:02:59,489
The first layer is visibility.
75
00:02:59,519 --> 00:03:00,589
Like, you need to know
76
00:03:00,609 --> 00:03:01,669
what you're doing, right?
77
00:03:02,179 --> 00:03:04,979
Um, in terms of AI assets, I
78
00:03:04,979 --> 00:03:06,169
keep hearing from some of the
79
00:03:06,169 --> 00:03:08,729
customers, like, the term shadow AI.
80
00:03:10,719 --> 00:03:12,849
The developers are building at a
81
00:03:12,849 --> 00:03:15,239
faster rate that the organizations
82
00:03:15,239 --> 00:03:19,129
can govern them or even understand
83
00:03:19,129 --> 00:03:20,579
the risk associated to them.
84
00:03:20,729 --> 00:03:22,659
That's why the question of
85
00:03:22,689 --> 00:03:24,509
the can I trust that workload?
86
00:03:25,249 --> 00:03:26,599
Maybe not, because I don't know
87
00:03:26,599 --> 00:03:28,089
what that workload is, right?
88
00:03:28,339 --> 00:03:30,249
So the first layer is visibility, which
89
00:03:30,249 --> 00:03:32,959
is the foundation to assess your exposure.
90
00:03:33,299 --> 00:03:35,099
The second one is control.
91
00:03:35,459 --> 00:03:37,819
Trust is all about control, understanding
92
00:03:37,819 --> 00:03:40,129
what are the risks associated, what are
93
00:03:40,129 --> 00:03:42,089
the constraints that I have, and do I have
94
00:03:42,109 --> 00:03:45,169
proper guardrails to control that, right?
95
00:03:45,419 --> 00:03:47,919
Any, any, uh, risk is
96
00:03:47,919 --> 00:03:49,299
mitigated against a control.
97
00:03:49,299 --> 00:03:51,209
A control is measured by a metric.
98
00:03:51,239 --> 00:03:53,709
Now you have a way to go and actually
99
00:03:53,729 --> 00:03:56,069
see if you have proper tooling in
100
00:03:56,069 --> 00:03:58,359
place to monitor those metrics,
101
00:03:58,379 --> 00:04:00,099
then you have the control, right?
102
00:04:00,519 --> 00:04:04,639
And the last piece is, um, accountability.
103
00:04:05,739 --> 00:04:07,679
Control without accountability
104
00:04:07,689 --> 00:04:09,769
is just good intention, right?
105
00:04:10,159 --> 00:04:11,689
We need to make sure that
106
00:04:11,689 --> 00:04:13,339
there is enforcement.
107
00:04:13,809 --> 00:04:15,479
You have the control to enforce
108
00:04:15,479 --> 00:04:17,259
them, and you have an option to
109
00:04:17,259 --> 00:04:18,709
pull back information to make
110
00:04:18,709 --> 00:04:20,579
sure those controls are working.
111
00:04:21,139 --> 00:04:22,579
Once you have these three elements
112
00:04:22,579 --> 00:04:25,289
together, visibility, control, and
113
00:04:25,289 --> 00:04:27,629
accountability, that's really the
114
00:04:27,629 --> 00:04:29,779
foundations to what makes people
115
00:04:29,859 --> 00:04:31,549
to trust that the system is working
116
00:04:33,265 --> 00:04:35,155
So why do you think the enterprise
117
00:04:35,165 --> 00:04:37,315
adoption… I know it has picked
118
00:04:37,315 --> 00:04:39,655
up in, in the agentic era compared
119
00:04:39,655 --> 00:04:41,265
to the model era quite a bit,
120
00:04:41,265 --> 00:04:43,695
but it's still as much, right?
121
00:04:43,695 --> 00:04:44,845
Like, you know, why do you think like,
122
00:04:44,845 --> 00:04:46,385
you know, companies are struggling,
123
00:04:46,425 --> 00:04:48,265
you know, to kind of put a lot of
124
00:04:48,265 --> 00:04:50,185
this stuff into production quickly
125
00:04:50,185 --> 00:04:51,975
and, know, where are there some of the
126
00:04:51,975 --> 00:04:54,335
challenges in your sort of, uh, you
127
00:04:54,335 --> 00:04:55,595
know, experience working with them?
128
00:04:56,443 --> 00:04:56,803
Yeah.
129
00:04:57,083 --> 00:04:58,663
I'd like to take you on a very quick
130
00:04:58,663 --> 00:05:00,563
journey of the past three years.
131
00:05:01,183 --> 00:05:03,513
At the beginning and after ChatGPT.
132
00:05:03,713 --> 00:05:06,053
When ChatGPT happened, we saw a
133
00:05:06,053 --> 00:05:09,203
rush on doing something with genAI.
134
00:05:09,723 --> 00:05:13,033
Like, I, I got many CIOs, CEOs coming
135
00:05:13,033 --> 00:05:15,053
to us, and they got a mandate from
136
00:05:15,053 --> 00:05:17,143
board to just do something with genAI.
137
00:05:18,013 --> 00:05:20,843
Very soon we saw that the applications
138
00:05:20,843 --> 00:05:23,433
and the use cases were focused on
139
00:05:23,443 --> 00:05:26,263
a very niche area around a series
140
00:05:26,273 --> 00:05:28,613
of well-defined use cases like
141
00:05:28,623 --> 00:05:30,943
information extraction, summarization,
142
00:05:30,953 --> 00:05:33,293
content-grounded question and answering.
143
00:05:33,313 --> 00:05:34,443
Later in the year, code
144
00:05:34,443 --> 00:05:36,083
generation was added to that.
145
00:05:36,633 --> 00:05:40,103
Um, so if you had a scenario
146
00:05:40,243 --> 00:05:42,323
that fit that, you're golden.
147
00:05:42,483 --> 00:05:44,453
If not, okay, what problem
148
00:05:44,453 --> 00:05:45,803
does it solve for me?
149
00:05:46,153 --> 00:05:49,843
So at the second year after, um, ChatGPT,
150
00:05:49,843 --> 00:05:51,773
we started seeing companies are struggling
151
00:05:51,793 --> 00:05:54,293
to see the value of ROI because they
152
00:05:54,293 --> 00:05:56,343
thought they could just apply that
153
00:05:56,343 --> 00:05:58,963
to every single scenarios, and this
154
00:05:59,053 --> 00:06:01,273
gives them efficiencies, which wasn't.
155
00:06:01,273 --> 00:06:02,523
So they were struggling.
156
00:06:02,553 --> 00:06:05,573
And then mid-year 2024, we started
157
00:06:05,573 --> 00:06:09,413
seeing LLMs taking action, as
158
00:06:09,413 --> 00:06:11,313
known as agents at the time.
159
00:06:12,103 --> 00:06:13,873
The market got super excited about
160
00:06:13,873 --> 00:06:15,403
this because they perceived this
161
00:06:15,443 --> 00:06:18,833
as an opportunity to go and apply
162
00:06:18,863 --> 00:06:23,273
that niche set of use cases, the
163
00:06:23,273 --> 00:06:24,943
productivity and acceleration that
164
00:06:24,943 --> 00:06:27,383
you get here to every single part
165
00:06:27,383 --> 00:06:29,313
of your organization, including
166
00:06:29,323 --> 00:06:31,253
legacy systems through API calling.
167
00:06:31,893 --> 00:06:33,913
And that's why, like, suddenly we
168
00:06:33,913 --> 00:06:35,623
saw a lot of excitement around,
169
00:06:35,663 --> 00:06:39,133
oh, agentic AI, automation, it
170
00:06:39,133 --> 00:06:40,713
can come and solve my problem.
171
00:06:40,753 --> 00:06:42,993
And again, boards now are asking
172
00:06:43,043 --> 00:06:44,713
about having a couple agents in
173
00:06:44,713 --> 00:06:46,153
production by the end of the year.
174
00:06:47,333 --> 00:06:47,943
This is 2024.
175
00:06:48,023 --> 00:06:50,223
Which my question was for those
176
00:06:50,223 --> 00:06:52,243
execs was, to do what exactly?
177
00:06:52,473 --> 00:06:54,593
Like, when you go to the core of
178
00:06:55,313 --> 00:06:58,413
not chasing technology, but looking
179
00:06:58,633 --> 00:07:01,013
at what problem does it solve for
180
00:07:01,013 --> 00:07:02,763
me, and if this is the right sol-
181
00:07:06,235 --> 00:07:08,345
But back to your question, why the market
182
00:07:08,375 --> 00:07:11,405
is excited about, because it's, it's
183
00:07:11,415 --> 00:07:14,505
very clear the opportunities that this
184
00:07:14,515 --> 00:07:16,465
set of technologies represent in terms
185
00:07:16,465 --> 00:07:18,495
of automation, in terms of generation,
186
00:07:18,505 --> 00:07:20,465
in terms of productivity, in terms of,
187
00:07:21,045 --> 00:07:23,295
uh, accelerating enterprise workloads.
188
00:07:23,565 --> 00:07:24,795
The promise is there.
189
00:07:25,403 --> 00:07:25,413
Hmm.
190
00:07:25,605 --> 00:07:27,245
The experimentations are done.
191
00:07:27,285 --> 00:07:28,895
They have seen the values.
192
00:07:28,965 --> 00:07:31,665
Now the blocker is the reality
193
00:07:31,675 --> 00:07:34,005
of, okay, things can go wrong.
194
00:07:34,835 --> 00:07:35,935
There is a certain level of
195
00:07:35,965 --> 00:07:37,965
autonomy associated with agents.
196
00:07:38,665 --> 00:07:40,155
Do we have the proper instrumentations
197
00:07:41,585 --> 00:07:43,705
and guardrails and guard lines or
198
00:07:43,715 --> 00:07:46,255
processes in place to protect ourselves?
199
00:07:46,875 --> 00:07:47,225
Yeah
200
00:07:47,265 --> 00:07:49,355
They can't solve that problem, they
201
00:07:49,355 --> 00:07:51,215
can't really utilize any of the
202
00:07:51,215 --> 00:07:55,065
accelerations that comes with the new tech
203
00:07:55,787 --> 00:07:57,447
So I think, you know, when like as
204
00:07:57,447 --> 00:07:58,567
you mentioned, when you kind of went,
205
00:07:58,637 --> 00:07:59,987
when we went back like maybe three
206
00:07:59,987 --> 00:08:01,787
years ago when, you know, ChatGPT
207
00:08:01,787 --> 00:08:03,497
came about, you know, models, models
208
00:08:03,497 --> 00:08:06,137
were actually the thing, right?
209
00:08:06,157 --> 00:08:07,327
Like, you know, you know, uh, you know,
210
00:08:07,327 --> 00:08:10,507
model, model layer had the most mode.
211
00:08:10,567 --> 00:08:13,277
Um, you know, and, and of course now three
212
00:08:13,277 --> 00:08:15,137
years later, uh, you know, models are
213
00:08:15,137 --> 00:08:16,657
becoming increasingly interchangeable.
214
00:08:16,707 --> 00:08:17,837
You know, I mean, yes, there are
215
00:08:17,837 --> 00:08:19,537
differences, you know, between, you know,
216
00:08:19,547 --> 00:08:20,907
one model and the other, but, you know,
217
00:08:21,277 --> 00:08:22,657
those are becoming more and more subtle.
218
00:08:23,087 --> 00:08:25,597
So where will the durable enterprise
219
00:08:25,627 --> 00:08:27,207
value accumulate, in your opinion?
220
00:08:27,487 --> 00:08:29,727
Uh, there's model, there's
221
00:08:29,857 --> 00:08:30,027
proprietary data, there's
222
00:08:30,027 --> 00:08:33,017
workflows, and there's of course
223
00:08:33,017 --> 00:08:35,147
the control guardrails policy layer.
224
00:08:35,487 --> 00:08:37,007
Where do you see like enterprises--
225
00:08:37,037 --> 00:08:38,287
Where should you enterprises
226
00:08:38,377 --> 00:08:39,877
focus on, you know, as they kind
227
00:08:39,877 --> 00:08:41,717
of go about this enter- AI stack?
228
00:08:42,095 --> 00:08:44,535
Well, there are multiple places
229
00:08:44,565 --> 00:08:46,215
when you look into the entire AI
230
00:08:46,225 --> 00:08:48,195
stack that depending on what kind
231
00:08:48,195 --> 00:08:49,875
of enterprise you have and what kind
232
00:08:49,875 --> 00:08:51,525
of assets that you have, you can
233
00:08:51,525 --> 00:08:54,275
shine and expand your market share.
234
00:08:54,875 --> 00:08:57,475
Let's say if you are sitting on
235
00:08:57,645 --> 00:09:00,645
enterprise data, most of these LLMs
236
00:09:00,715 --> 00:09:03,625
are trained on public data, which is
237
00:09:03,625 --> 00:09:06,245
a fraction of the entire data that
238
00:09:06,245 --> 00:09:09,925
is available as of like 2026, right?
239
00:09:10,265 --> 00:09:13,145
So if you have access to that data,
240
00:09:13,195 --> 00:09:15,885
specialized data, that's your moat, right?
241
00:09:16,075 --> 00:09:16,605
So that's one.
242
00:09:17,385 --> 00:09:20,595
If you go on the application layer,
243
00:09:20,685 --> 00:09:23,625
monitoring, governance, like if
244
00:09:23,625 --> 00:09:25,695
we truly believe that the cost of
245
00:09:25,695 --> 00:09:27,335
software development keeps going
246
00:09:27,335 --> 00:09:30,195
down, and that's something that
247
00:09:30,195 --> 00:09:32,245
we saw in the past with X-ray.
248
00:09:32,285 --> 00:09:34,695
You remember like when at, at, at first
249
00:09:34,695 --> 00:09:37,745
when not LLMs, uh, the generative…
250
00:09:37,755 --> 00:09:39,315
What did they, we call them?
251
00:09:39,325 --> 00:09:41,025
Like, um, the,
252
00:09:41,221 --> 00:09:41,861
LSTMs or,
253
00:09:42,115 --> 00:09:42,285
the
254
00:09:42,331 --> 00:09:42,481
are they?
255
00:09:42,615 --> 00:09:43,985
Early generations of that.
256
00:09:43,995 --> 00:09:45,475
They came in, we are like, "Oh, okay,
257
00:09:45,475 --> 00:09:48,985
so we don't need the, um, uh, we don't
258
00:09:48,985 --> 00:09:50,845
need the person to actually look at
259
00:09:50,845 --> 00:09:52,935
the X-ray to figure out what's going
260
00:09:52,935 --> 00:09:56,115
wrong." So they started, radiologists,
261
00:09:56,115 --> 00:09:59,395
they started losing their jobs because
262
00:09:59,425 --> 00:10:00,935
we didn't need them at the time.
263
00:10:00,945 --> 00:10:03,435
But the cost of X-ray went down to
264
00:10:03,435 --> 00:10:05,495
the point that we can now, we go to
265
00:10:05,495 --> 00:10:07,345
the doctor, they send us to X-ray.
266
00:10:07,515 --> 00:10:08,065
Guess what?
267
00:10:08,345 --> 00:10:10,545
The number of requests for X-ray
268
00:10:10,955 --> 00:10:13,255
keeps going up, and suddenly we are
269
00:10:13,255 --> 00:10:15,895
like, "Oh, so much complications,
270
00:10:15,925 --> 00:10:18,015
we need radiologists." So the need
271
00:10:18,015 --> 00:10:20,385
for radiologists kept going back up.
272
00:10:21,165 --> 00:10:23,665
I think we see the same for software.
273
00:10:23,855 --> 00:10:25,435
The cost of software development
274
00:10:25,435 --> 00:10:28,055
keeps going up, but the generation
275
00:10:28,575 --> 00:10:30,395
keeps going down, but the generation
276
00:10:30,415 --> 00:10:31,795
of software keeps going up.
277
00:10:31,805 --> 00:10:34,005
The demand for that is gonna go up, to
278
00:10:34,005 --> 00:10:36,325
the point that we automatically create
279
00:10:36,325 --> 00:10:38,225
software at the point of interaction.
280
00:10:38,905 --> 00:10:41,655
With all that automation, the
281
00:10:41,655 --> 00:10:44,705
key point here, and back to
282
00:10:44,705 --> 00:10:46,465
opportunities, is governance.
283
00:10:46,945 --> 00:10:49,505
You need to have and make sure that
284
00:10:49,655 --> 00:10:51,615
the, the guidelines, the controls,
285
00:10:51,615 --> 00:10:54,285
the risk, the tracing, the security,
286
00:10:54,315 --> 00:10:56,815
everything is there to accommodate
287
00:10:56,815 --> 00:10:59,685
that new world that collectively in the
288
00:10:59,685 --> 00:11:01,925
market we are not equipped to today.
289
00:11:02,175 --> 00:11:04,015
So depending on where you are operating,
290
00:11:04,085 --> 00:11:07,465
what part of the stack there is
291
00:11:07,465 --> 00:11:10,595
opportunity to figure out where the
292
00:11:10,595 --> 00:11:13,875
market is going and how can I utilize
293
00:11:13,875 --> 00:11:15,745
the advantage that I have to be
294
00:11:15,745 --> 00:11:18,255
differentiated versus just take the
295
00:11:18,325 --> 00:11:20,845
technology and go chase adoption of it.
296
00:11:21,691 --> 00:11:22,651
Yeah, makes sense.
297
00:11:22,981 --> 00:11:24,611
And so I think that's a good segue, right?
298
00:11:24,611 --> 00:11:26,131
So essentially, you know, that
299
00:11:26,141 --> 00:11:26,961
you mentioned like, you know,
300
00:11:26,971 --> 00:11:28,971
models and proprietary data and
301
00:11:28,971 --> 00:11:30,421
then governance and controls.
302
00:11:30,431 --> 00:11:31,131
So let's talk about the
303
00:11:31,131 --> 00:11:32,171
governance and controls.
304
00:11:32,641 --> 00:11:34,441
I would say maybe in the last six
305
00:11:34,441 --> 00:11:37,061
months, this this phrase called the AI
306
00:11:37,271 --> 00:11:40,701
control plane has just become uh, a,
307
00:11:40,771 --> 00:11:42,911
a popular, a popular category, right?
308
00:11:43,221 --> 00:11:44,371
What do you, what's, what
309
00:11:44,371 --> 00:11:45,501
you, what is your opinion?
310
00:11:45,501 --> 00:11:47,101
What are, what are the capabilities
311
00:11:47,131 --> 00:11:48,741
a control plane should, should,
312
00:11:48,791 --> 00:11:50,601
should have, you know, so that it
313
00:11:50,601 --> 00:11:51,971
sort of like, you know, basically
314
00:11:51,971 --> 00:11:53,551
can help these enterprise customers?
315
00:11:53,915 --> 00:11:54,335
Yeah.
316
00:11:54,535 --> 00:11:56,985
So as AI becomes the backbone of how
317
00:11:57,085 --> 00:12:00,045
enterprises operate, and AI is changing
318
00:12:00,045 --> 00:12:03,655
rapidly, when you're going at that speed,
319
00:12:03,655 --> 00:12:06,145
control is not the enemy of the progress.
320
00:12:06,195 --> 00:12:07,485
It's like a race car.
321
00:12:08,575 --> 00:12:09,985
The, the brake is not
322
00:12:09,995 --> 00:12:11,425
designed to slow you down.
323
00:12:11,695 --> 00:12:14,035
It's designed to make you go faster,
324
00:12:14,035 --> 00:12:16,205
corner harder, and stay on the track.
325
00:12:16,425 --> 00:12:17,315
That's your governance.
326
00:12:17,705 --> 00:12:19,865
That's why now we are having this
327
00:12:19,865 --> 00:12:21,685
conversation about controls and
328
00:12:21,685 --> 00:12:23,525
control planes, because we need
329
00:12:23,525 --> 00:12:25,015
to be able to accommodate that
330
00:12:25,035 --> 00:12:27,215
speed of moving really fast.
331
00:12:27,625 --> 00:12:30,525
And the reality is when you look into
332
00:12:30,525 --> 00:12:33,325
the market, the founders that are putting
333
00:12:33,325 --> 00:12:35,225
products out there, the startups, like
334
00:12:35,225 --> 00:12:38,655
75% of the business executives, they
335
00:12:38,695 --> 00:12:41,825
are not confident they would pass an
336
00:12:42,195 --> 00:12:44,795
independent AI audit within 90 days,
337
00:12:44,985 --> 00:12:47,575
and yet they have their agents and
338
00:12:47,575 --> 00:12:49,685
their products in, like, production.
339
00:12:50,299 --> 00:12:50,669
Yeah.
340
00:12:50,709 --> 00:12:51,079
Yeah
341
00:12:51,205 --> 00:12:52,475
And the cost is real, too.
342
00:12:52,555 --> 00:12:53,825
Like, I'll give you an example.
343
00:12:53,845 --> 00:12:56,425
A 20-billion enterprise with weak
344
00:12:56,465 --> 00:12:58,475
governance in average is losing
345
00:12:58,505 --> 00:13:00,635
about 70 million a year, not from
346
00:13:00,635 --> 00:13:03,495
random errors, not from unavoidable
347
00:13:03,525 --> 00:13:06,855
glitches, from preventable oversight.
348
00:13:07,265 --> 00:13:09,825
So how do they get that control plane?
349
00:13:10,415 --> 00:13:12,975
It, it's the promise of giving
350
00:13:12,975 --> 00:13:15,185
them a chance to monitor all of
351
00:13:15,185 --> 00:13:16,965
those, to understand what they
352
00:13:16,965 --> 00:13:19,595
need to control, meaning that…
353
00:13:19,745 --> 00:13:21,125
And controls, you know where these
354
00:13:21,125 --> 00:13:22,685
controls are coming really from.
355
00:13:22,825 --> 00:13:24,405
Corporate policies that you have,
356
00:13:24,605 --> 00:13:26,625
enterprise AI risk that you're exposed
357
00:13:26,625 --> 00:13:29,485
to, regulatory compliance that, um,
358
00:13:29,495 --> 00:13:31,825
that the market is defining for you, and
359
00:13:31,845 --> 00:13:34,585
operational efficiencies like the payload
360
00:13:35,055 --> 00:13:37,945
uh, cost and everything that you care
361
00:13:37,945 --> 00:13:41,065
about, really understanding them, mapping
362
00:13:41,075 --> 00:13:44,175
them to metrics that you can go enforce.
363
00:13:44,905 --> 00:13:46,445
And that's the challenge that
364
00:13:46,445 --> 00:13:47,745
the market is facing now.
365
00:13:48,155 --> 00:13:50,195
So now control plane, the promise of
366
00:13:50,195 --> 00:13:52,445
control plane is really three things.
367
00:13:52,475 --> 00:13:55,335
One, to help you define those controls.
368
00:13:56,525 --> 00:13:57,755
The second one is to
369
00:13:57,835 --> 00:13:59,455
implement those controls.
370
00:13:59,475 --> 00:14:01,225
Let's say if we are building
371
00:14:01,235 --> 00:14:05,215
an HR system, we are, uh, using
372
00:14:05,215 --> 00:14:07,295
sensitive employee information.
373
00:14:07,675 --> 00:14:09,885
When sensitive information are
374
00:14:09,885 --> 00:14:11,795
involved, you need to put together
375
00:14:11,825 --> 00:14:13,605
guardrails to mask the information
376
00:14:13,615 --> 00:14:15,325
or make sure that none, none of them
377
00:14:15,355 --> 00:14:18,575
are, um, shown up in the output.
378
00:14:18,575 --> 00:14:19,245
This is a guardrail.
379
00:14:19,445 --> 00:14:21,015
This is a definition of control.
380
00:14:21,315 --> 00:14:22,875
So control plane should help you
381
00:14:22,875 --> 00:14:25,095
to implement this either out of the
382
00:14:25,095 --> 00:14:28,565
box or with AI-assisted coding, um,
383
00:14:28,635 --> 00:14:30,055
just to implement what you need.
384
00:14:30,435 --> 00:14:31,525
And then last piece of
385
00:14:31,565 --> 00:14:33,125
this is enforcement.
386
00:14:33,345 --> 00:14:34,925
Actually, four, four, four parts.
387
00:14:35,125 --> 00:14:36,655
Enforcement is the third one.
388
00:14:36,735 --> 00:14:38,485
You need, now that you have implemented
389
00:14:38,525 --> 00:14:40,595
that, you have to go and enforce no matter
390
00:14:40,625 --> 00:14:42,445
where the workloads are being deployed.
391
00:14:42,745 --> 00:14:45,415
And the last one, which is extremely
392
00:14:45,415 --> 00:14:47,185
important, is to close the loop, is
393
00:14:47,185 --> 00:14:50,037
enforce- of controls happens, what
394
00:14:50,037 --> 00:14:51,587
was the impact to my business if
395
00:14:51,587 --> 00:14:53,437
the control is working, do I need
396
00:14:53,477 --> 00:14:55,197
to adjust the control and come back?
397
00:14:55,487 --> 00:14:57,817
Really defining, implementing,
398
00:14:57,837 --> 00:15:00,367
enforcing and enforcement tracking is
399
00:15:00,367 --> 00:15:01,847
what you need to implement this cycle.
400
00:15:03,107 --> 00:15:03,527
Super.
401
00:15:03,577 --> 00:15:04,317
That's, that's awesome.
402
00:15:04,637 --> 00:15:07,377
And so in your opinion, uh, should like
403
00:15:07,407 --> 00:15:09,287
enterprises like say banks, insurance,
404
00:15:09,287 --> 00:15:11,107
or like any large enterprise company,
405
00:15:11,447 --> 00:15:13,467
should they look at a control plane about
406
00:15:13,507 --> 00:15:16,167
every model application tool, whatever
407
00:15:16,167 --> 00:15:17,817
their, you know, data source, or they
408
00:15:17,847 --> 00:15:20,147
should look at from a… Do you think
409
00:15:20,147 --> 00:15:21,887
this will be a fragmented set of control
410
00:15:21,907 --> 00:15:23,037
planes, or do you think there should
411
00:15:23,047 --> 00:15:25,577
be an, kind of a unified control plane?
412
00:15:25,577 --> 00:15:26,497
You know, where do you see, where
413
00:15:26,497 --> 00:15:27,857
do you see like how do, how do
414
00:15:27,857 --> 00:15:29,187
you think the deployment and
415
00:15:29,657 --> 00:15:30,827
sort of the adoption will be?
416
00:15:31,149 --> 00:15:33,569
Well, here, um, uh, the answer depends
417
00:15:33,569 --> 00:15:35,109
on who you are asking this question.
418
00:15:35,579 --> 00:15:38,059
If you are asking the risk and compliance
419
00:15:38,099 --> 00:15:40,259
officer in a company or you are asking
420
00:15:40,749 --> 00:15:44,399
CISO, they care about the entire stack.
421
00:15:44,719 --> 00:15:46,749
It's like at the application layer, if I'm
422
00:15:46,749 --> 00:15:48,999
exposed to risk, you need to observe that.
423
00:15:49,269 --> 00:15:52,179
At the AI layer, if I'm at, um, exposed
424
00:15:52,189 --> 00:15:53,469
to that, you need to observe that.
425
00:15:53,479 --> 00:15:54,819
At the infrastructure
426
00:15:54,819 --> 00:15:56,149
layer, the same, right?
427
00:15:56,569 --> 00:15:59,869
But then when you go to the business, um,
428
00:15:59,949 --> 00:16:02,539
pillar, depending on what, what you are
429
00:16:02,539 --> 00:16:04,469
controlling, do you have an application?
430
00:16:04,529 --> 00:16:06,049
Do you have an AI asset?
431
00:16:06,519 --> 00:16:08,509
is it a GPU, um,
432
00:16:08,539 --> 00:16:10,559
infrastructure-based layer?
433
00:16:10,599 --> 00:16:12,349
You are subject to different things.
434
00:16:12,639 --> 00:16:13,709
I'll give you an example.
435
00:16:14,169 --> 00:16:15,389
Security risks.
436
00:16:16,229 --> 00:16:17,799
A risk and compliance officer or the
437
00:16:17,799 --> 00:16:20,299
CISO, as part of their work, they need to
438
00:16:20,299 --> 00:16:22,099
understand the risk they are exposed to.
439
00:16:23,399 --> 00:16:24,909
security risk is a category,
440
00:16:24,919 --> 00:16:27,359
like OWASP top 10 agentic,
441
00:16:27,369 --> 00:16:28,959
uh, application risk, right?
442
00:16:29,369 --> 00:16:31,749
But then now that they are defined, you
443
00:16:31,749 --> 00:16:33,629
remember the four pillars of control.
444
00:16:33,679 --> 00:16:35,109
Now they are defined, now
445
00:16:35,109 --> 00:16:36,319
we need to figure out how to
446
00:16:36,319 --> 00:16:38,529
implement them and enforce them.
447
00:16:38,559 --> 00:16:40,679
Usually, the implementation of security
448
00:16:40,679 --> 00:16:43,029
risk are within a security tool.
449
00:16:43,429 --> 00:16:45,229
It's not an observability dashboard.
450
00:16:45,529 --> 00:16:48,249
But if it… If the risk is monitoring
451
00:16:48,249 --> 00:16:50,309
to make sure the sensitive information
452
00:16:50,309 --> 00:16:53,099
is not surfaced at output, the HR case
453
00:16:53,099 --> 00:16:54,989
that we mentioned, this is at AI layer.
454
00:16:55,059 --> 00:16:56,699
This is not a security enforcement.
455
00:16:56,959 --> 00:16:58,329
You're gonna need to go to
456
00:16:58,469 --> 00:17:00,199
observability dashboards to make sure
457
00:17:00,199 --> 00:17:02,489
that the tracing is capturing it.
458
00:17:02,499 --> 00:17:05,289
So really depending on what the use
459
00:17:05,309 --> 00:17:08,559
case is and what layer of AI assets are
460
00:17:08,599 --> 00:17:10,389
covering, you wanna make sure that you
461
00:17:10,389 --> 00:17:13,029
have a full coverage across your stack
462
00:17:13,209 --> 00:17:15,489
on a series of different observability
463
00:17:15,489 --> 00:17:19,519
platform, but a unified single view
464
00:17:19,589 --> 00:17:22,229
of risk across the organizations and
465
00:17:22,239 --> 00:17:25,479
all the controls pulling back to, um,
466
00:17:25,489 --> 00:17:27,299
the we call it governance console.
467
00:17:27,309 --> 00:17:28,639
The governance console that you
468
00:17:28,649 --> 00:17:30,229
have to track if breach happened
469
00:17:30,229 --> 00:17:31,539
or if the controls are working.
470
00:17:32,395 --> 00:17:33,225
No, it's amazing, yeah.
471
00:17:33,625 --> 00:17:34,225
That's really cool.
472
00:17:34,425 --> 00:17:37,035
So, uh, you know, how sort of, uh,
473
00:17:37,155 --> 00:17:39,035
important it is for the control
474
00:17:39,035 --> 00:17:40,885
planes to remain independent of
475
00:17:40,885 --> 00:17:42,565
the platforms that are building AI?
476
00:17:42,945 --> 00:17:44,505
So there's this whole sort of AI
477
00:17:44,515 --> 00:17:45,905
factories that have emerged, right?
478
00:17:45,905 --> 00:17:48,545
So this foundation model companies, cloud
479
00:17:48,545 --> 00:17:50,085
companies, they all have AI factories.
480
00:17:50,295 --> 00:17:52,745
What-- So is, is there, like, a role for
481
00:17:52,745 --> 00:17:54,825
an, an independent control plane or what,
482
00:17:54,825 --> 00:17:56,075
what do you think about that sort of?
483
00:17:56,589 --> 00:17:57,829
Well, I wouldn't, I wouldn't call
484
00:17:57,829 --> 00:17:59,529
it an independent control plane
485
00:17:59,529 --> 00:18:01,459
because control plane is just a tool.
486
00:18:02,199 --> 00:18:05,589
I would come one step back and look
487
00:18:05,589 --> 00:18:09,819
into the needs and the foundations.
488
00:18:09,859 --> 00:18:11,659
Like control plane is to implement
489
00:18:11,669 --> 00:18:13,429
some sort of governance, right?
490
00:18:14,099 --> 00:18:16,099
If you like to control and govern
491
00:18:16,109 --> 00:18:18,189
something, as a best practice,
492
00:18:18,219 --> 00:18:19,999
it's better to have a third party
493
00:18:20,229 --> 00:18:22,619
than the party that is building it.
494
00:18:22,649 --> 00:18:25,619
Like the, um, compliance certificates,
495
00:18:25,619 --> 00:18:28,859
like SOC 2, like there's the- company
496
00:18:28,859 --> 00:18:30,689
that is defining what the standard
497
00:18:30,689 --> 00:18:32,239
is is not necessarily the one that
498
00:18:32,239 --> 00:18:33,829
is building the applications, right?
499
00:18:34,119 --> 00:18:35,249
The same concepts here.
500
00:18:35,529 --> 00:18:38,009
But then for the implementation, it
501
00:18:38,009 --> 00:18:40,759
can be implemented in different ways.
502
00:18:41,039 --> 00:18:43,189
If there is a technology company that
503
00:18:43,199 --> 00:18:45,229
has the expertise to accelerate the
504
00:18:45,229 --> 00:18:47,659
implementation of it and you are an
505
00:18:47,669 --> 00:18:49,859
enterprise, you need a generic dashboard,
506
00:18:49,959 --> 00:18:51,619
that's probably your option to go.
507
00:18:51,859 --> 00:18:54,499
If you need an specialized thing
508
00:18:54,519 --> 00:18:57,069
on a cert- a certain part of the
509
00:18:57,079 --> 00:18:59,529
stack that is not covered by the
510
00:18:59,529 --> 00:19:01,549
market, you probably wanna build
511
00:19:01,549 --> 00:19:03,519
that specialized control plane.
512
00:19:04,009 --> 00:19:05,909
So there is no standard definition
513
00:19:05,909 --> 00:19:07,549
of what this control plane is.
514
00:19:07,579 --> 00:19:09,269
It's really back to those four
515
00:19:09,359 --> 00:19:11,349
categories of what you I- call
516
00:19:11,349 --> 00:19:12,899
it the flywheel of control
517
00:19:13,439 --> 00:19:15,389
And what is the best way for you to
518
00:19:15,389 --> 00:19:17,259
implement those control all the way from
519
00:19:17,259 --> 00:19:18,979
definitions to enforcement tracking?
520
00:19:19,853 --> 00:19:20,093
Hmm.
521
00:19:20,303 --> 00:19:20,653
Awesome.
522
00:19:21,073 --> 00:19:22,703
So, so the, the other interesting thing
523
00:19:22,703 --> 00:19:24,123
that's happening now is actually on a
524
00:19:24,133 --> 00:19:27,283
customer call yesterday, and the the,
525
00:19:27,293 --> 00:19:28,983
the customer was from a bank, and he
526
00:19:28,983 --> 00:19:31,873
was asking that, "Look, you know, like,
527
00:19:31,873 --> 00:19:33,453
agents are becoming like these digital
528
00:19:33,453 --> 00:19:35,933
employees. You know, are, are you sort
529
00:19:35,933 --> 00:19:37,833
of, like, managing them like people do?"
530
00:19:37,843 --> 00:19:38,813
Like, you know, and, and that was
531
00:19:38,813 --> 00:19:39,913
a very interesting thing, right?
532
00:19:39,923 --> 00:19:41,673
You know, he was asking, "Do you sell
533
00:19:41,673 --> 00:19:43,143
to the chief people officers and head
534
00:19:43,143 --> 00:19:45,243
of HRs out there?" And I was like, "Not
535
00:19:45,243 --> 00:19:46,723
yet." But, like, but that's actually
536
00:19:46,723 --> 00:19:47,873
a very interesting thing, right?
537
00:19:47,883 --> 00:19:49,523
Like, will… And do you think enterprises
538
00:19:49,523 --> 00:19:51,613
will manage agents the way they manage
539
00:19:51,613 --> 00:19:53,323
employees, like with identities,
540
00:19:53,363 --> 00:19:56,003
owners, roles, permissions, you know,
541
00:19:56,013 --> 00:19:57,813
performance expectations, right?
542
00:19:57,833 --> 00:19:59,773
Kind of like how you manage people.
543
00:19:59,853 --> 00:20:01,443
Or where do you see this going?
544
00:20:01,741 --> 00:20:03,861
Well, I will bring up two stories for you.
545
00:20:04,121 --> 00:20:05,861
The first one is the role
546
00:20:05,861 --> 00:20:07,791
of human is evolving.
547
00:20:08,731 --> 00:20:12,021
Historically, we were like,
548
00:20:12,121 --> 00:20:14,191
uh, the role of AI was to
549
00:20:14,211 --> 00:20:16,401
accelerate our human processes.
550
00:20:17,091 --> 00:20:18,301
And by historically, I mean
551
00:20:18,311 --> 00:20:19,711
like the last three years.
552
00:20:19,841 --> 00:20:22,501
Like I write an email, an agent can
553
00:20:22,501 --> 00:20:24,291
help me… Not agent AI, a generative
554
00:20:24,291 --> 00:20:27,021
AI can help me with a better written
555
00:20:27,031 --> 00:20:28,751
email or the content that I generate.
556
00:20:29,051 --> 00:20:31,131
So it's a need that I had, and now
557
00:20:31,161 --> 00:20:34,381
AI is coming to help me deliver that.
558
00:20:34,931 --> 00:20:37,241
Moving forward, the role of the human
559
00:20:37,311 --> 00:20:39,431
is gonna change to an orchestrator.
560
00:20:39,741 --> 00:20:40,861
So instead of… Like
561
00:20:41,011 --> 00:20:42,241
just, just picture this.
562
00:20:42,501 --> 00:20:44,101
I use AI to write an
563
00:20:44,101 --> 00:20:46,061
email, send it to someone.
564
00:20:46,171 --> 00:20:47,201
Someone else is using
565
00:20:47,211 --> 00:20:48,831
AI to read the email.
566
00:20:49,291 --> 00:20:50,651
So at some point we don't need
567
00:20:50,651 --> 00:20:52,391
to exchange these emails, right?
568
00:20:53,151 --> 00:20:53,891
It goes away.
569
00:20:54,241 --> 00:20:56,001
It's a temporary thing that
570
00:20:56,011 --> 00:20:57,961
like… And it's just an example.
571
00:20:58,467 --> 00:20:58,597
Yeah.
572
00:20:58,901 --> 00:21:01,021
But it's a, it's a temporary thing
573
00:21:01,031 --> 00:21:03,091
that for a period of time we are gonna
574
00:21:03,091 --> 00:21:05,711
handle, but eventually everything
575
00:21:05,781 --> 00:21:07,441
is gonna be automated, and the role
576
00:21:07,441 --> 00:21:09,671
of us is really an orchestrator.
577
00:21:10,721 --> 00:21:13,831
So now in enterprises, in the current
578
00:21:13,841 --> 00:21:16,381
format of the processes and hierarchy, we
579
00:21:16,381 --> 00:21:19,781
do not have mechanism to accompany this.
580
00:21:19,831 --> 00:21:21,591
And, and even, even I don't think
581
00:21:21,591 --> 00:21:23,931
market collectively understand what
582
00:21:23,931 --> 00:21:26,851
it means in terms of the identity
583
00:21:26,851 --> 00:21:28,541
of the agent, what it should be,
584
00:21:28,551 --> 00:21:30,731
who is the accountable party.
585
00:21:30,741 --> 00:21:31,891
So for example, agents.
586
00:21:31,901 --> 00:21:33,321
Let's go a little bit deeper.
587
00:21:33,991 --> 00:21:35,791
If there is an HR agent
588
00:21:37,035 --> 00:21:38,365
Let's even simplify that.
589
00:21:38,685 --> 00:21:40,705
A customer care agent and a
590
00:21:40,735 --> 00:21:43,935
bank is using to interact with
591
00:21:43,935 --> 00:21:45,405
their customers, end users.
592
00:21:46,305 --> 00:21:49,595
A bad actor comes in, does the
593
00:21:49,595 --> 00:21:53,345
jailbreak, and things happen, right?
594
00:21:54,705 --> 00:21:56,475
You don't know what exactly happened.
595
00:21:56,925 --> 00:21:58,335
This can be potentially
596
00:21:58,335 --> 00:21:59,225
who is accountable here.
597
00:21:59,235 --> 00:22:00,165
Something went wrong.
598
00:22:00,525 --> 00:22:02,285
We need to figure out what happened.
599
00:22:02,705 --> 00:22:05,555
It can be potentially the bad actor, or
600
00:22:05,555 --> 00:22:08,405
it can be the business unit that made
601
00:22:08,405 --> 00:22:11,995
the call to expose that to the user, or
602
00:22:11,995 --> 00:22:15,795
it can be the provider of agent itself.
603
00:22:15,825 --> 00:22:17,445
Let's say it was a third-party agent
604
00:22:17,445 --> 00:22:20,015
that you brought into your custom
605
00:22:20,015 --> 00:22:22,735
workflows, or it can be the model
606
00:22:22,735 --> 00:22:25,505
provider that trained the model that
607
00:22:25,505 --> 00:22:27,405
powered up this agent, or it could
608
00:22:27,435 --> 00:22:29,385
be the tools that the agent called.
609
00:22:29,885 --> 00:22:32,095
So I don't think this is collectively
610
00:22:32,095 --> 00:22:34,885
clear, uh, in the market when you
611
00:22:34,895 --> 00:22:37,395
talk to lawyers, when when you talk to
612
00:22:37,395 --> 00:22:39,225
regulations, when you talk to the business
613
00:22:39,235 --> 00:22:42,465
owners of who is accountable here and
614
00:22:42,475 --> 00:22:45,195
what is really the identity of the agent.
615
00:22:45,265 --> 00:22:46,975
At least in the near future, the
616
00:22:46,975 --> 00:22:49,075
person that is using it or impl-
617
00:22:49,125 --> 00:22:51,345
uh, applying it is responsible.
618
00:22:51,355 --> 00:22:54,035
So, like, if you send a agent to do
619
00:22:54,035 --> 00:22:56,635
something, it inherits the access
620
00:22:56,635 --> 00:22:58,575
control that you have, and you are
621
00:22:58,665 --> 00:23:00,335
accountable if something goes wrong.
622
00:23:01,015 --> 00:23:03,615
But there are lots of fascinating unknowns
623
00:23:04,095 --> 00:23:06,625
that collectively with the market, uh,
624
00:23:06,635 --> 00:23:08,575
we are discovering as we move forward.
625
00:23:09,571 --> 00:23:11,071
And so it's very interesting
626
00:23:11,141 --> 00:23:12,301
question, accountability, right?
627
00:23:12,331 --> 00:23:14,141
Like in the past, you know, you know,
628
00:23:15,051 --> 00:23:16,371
where I used to work in social media
629
00:23:16,371 --> 00:23:18,881
companies, you know, was like one incident
630
00:23:18,891 --> 00:23:20,461
where the site is down, you know, for
631
00:23:20,491 --> 00:23:22,961
many hours and, you know, who did it?
632
00:23:22,961 --> 00:23:24,191
Who made, who made the change?
633
00:23:24,191 --> 00:23:25,301
You know, the, the intern made the
634
00:23:25,301 --> 00:23:26,521
change, and essentially that's the
635
00:23:26,521 --> 00:23:28,681
accountability would rest in a person.
636
00:23:28,681 --> 00:23:29,691
But as you said, you know,
637
00:23:30,161 --> 00:23:31,561
where does the accountability
638
00:23:31,561 --> 00:23:33,321
fall when an agent goes wrong?
639
00:23:33,351 --> 00:23:34,801
Is it like the creator of the agent?
640
00:23:34,801 --> 00:23:36,181
Is it the model that powering the
641
00:23:36,181 --> 00:23:38,221
agent and, or the one that's the
642
00:23:38,221 --> 00:23:39,751
identity that the agent is mimicking?
643
00:23:40,331 --> 00:23:41,761
But it's also it can be possible
644
00:23:41,761 --> 00:23:42,831
that I can create an agent, I
645
00:23:42,831 --> 00:23:44,631
can give it to you, then what…
646
00:23:44,821 --> 00:23:45,971
That you ran the agent
647
00:23:45,971 --> 00:23:47,521
and that agent went wrong,
648
00:23:47,523 --> 00:23:48,163
Exactly.
649
00:23:48,363 --> 00:23:48,943
Exactly
650
00:23:48,961 --> 00:23:50,171
who sort of is accountable?
651
00:23:50,171 --> 00:23:51,661
That's actually another problem, right?
652
00:23:52,259 --> 00:23:54,249
But you brought up a very good point
653
00:23:54,259 --> 00:23:56,709
that reminds me of a different aspects
654
00:23:57,119 --> 00:23:58,489
of this that we didn't talk about.
655
00:23:58,499 --> 00:24:00,369
Like in this climate, if I'm an
656
00:24:00,379 --> 00:24:01,939
enterprise, what should I do?
657
00:24:02,349 --> 00:24:03,819
Because I have to adopt these
658
00:24:03,819 --> 00:24:05,839
agents and the technology.
659
00:24:05,839 --> 00:24:08,409
I can't, I can't just say, "Hey, I
660
00:24:08,429 --> 00:24:11,229
wait in five years to see where the
661
00:24:11,229 --> 00:24:12,889
market is going, and then I come in."
662
00:24:13,109 --> 00:24:14,059
That's not an option.
663
00:24:14,499 --> 00:24:18,389
Um, the example that you, um, used.
664
00:24:19,269 --> 00:24:23,559
Historically, let's say I'm a, um, major
665
00:24:23,589 --> 00:24:27,489
provider of networks across a country.
666
00:24:27,849 --> 00:24:30,119
Historically, I was looking
667
00:24:30,119 --> 00:24:31,459
into it's a risk category that
668
00:24:31,459 --> 00:24:34,069
is called business continuity.
669
00:24:34,379 --> 00:24:35,779
They were looking at earthquake
670
00:24:35,779 --> 00:24:39,349
happens, fire happens, how can I have
671
00:24:39,359 --> 00:24:42,679
backups, backup MZRs, data centers,
672
00:24:42,689 --> 00:24:44,469
backup things to minimize that?
673
00:24:45,009 --> 00:24:47,619
Now moving towards AI, AI
674
00:24:47,619 --> 00:24:49,379
everywhere, literally everywhere.
675
00:24:50,119 --> 00:24:52,019
Now we need to ask the question, let's
676
00:24:52,019 --> 00:24:55,759
say for AI, if the API for the model
677
00:24:55,769 --> 00:24:58,399
provider is down for two minutes,
678
00:24:58,669 --> 00:25:00,599
what is the impact to my business?
679
00:25:00,729 --> 00:25:02,279
How can I future-proof
680
00:25:02,469 --> 00:25:03,859
and minimize that impact?
681
00:25:04,329 --> 00:25:06,379
So meaning that, like if things
682
00:25:06,399 --> 00:25:12,089
happen, do I have a, a, a mechanism
683
00:25:12,529 --> 00:25:14,219
to come in and protect myself?
684
00:25:14,219 --> 00:25:15,939
Because for agents and everything,
685
00:25:15,939 --> 00:25:17,299
we know what can go wrong.
686
00:25:18,609 --> 00:25:20,369
It's not a matter of, "Hey, don't adopt
687
00:25:20,369 --> 00:25:22,209
it." No, it's like before you adopt
688
00:25:22,209 --> 00:25:25,649
it, put together a mechanism to protect
689
00:25:25,649 --> 00:25:27,309
yourself for whatever you care about.
690
00:25:27,599 --> 00:25:29,349
Like business continuity, for example,
691
00:25:29,359 --> 00:25:31,519
for years in GRC world, for 20 years
692
00:25:31,729 --> 00:25:33,699
peop- enterprises have been using this.
693
00:25:34,429 --> 00:25:36,239
Now it's applicable to the AI world.
694
00:25:36,589 --> 00:25:38,339
Third-party risk is another one.
695
00:25:38,389 --> 00:25:39,769
Historically, they were using it
696
00:25:39,769 --> 00:25:42,569
for defining like what, what, what
697
00:25:42,569 --> 00:25:44,299
company to partner for procurement.
698
00:25:44,329 --> 00:25:46,849
Now agents sprawl, AI assets
699
00:25:46,849 --> 00:25:48,069
are coming from everywhere.
700
00:25:48,489 --> 00:25:50,769
How would you pick which third
701
00:25:50,769 --> 00:25:54,589
party to pick, um, um, to trust
702
00:25:54,669 --> 00:25:56,529
and bring their workloads to us?
703
00:25:56,589 --> 00:25:57,629
And if that happens, how
704
00:25:57,629 --> 00:25:58,929
are you protecting yourself?
705
00:25:59,689 --> 00:26:01,779
Um, so instead of like, uh, we, we
706
00:26:01,799 --> 00:26:04,989
obviously have to, we as enterprise look
707
00:26:04,989 --> 00:26:07,999
into what are the technology available
708
00:26:08,049 --> 00:26:10,009
to me, the best, the state of the art to
709
00:26:10,009 --> 00:26:12,359
adopt, but at the most important thing
710
00:26:12,369 --> 00:26:15,299
is what are the non-negotiables for me?
711
00:26:15,639 --> 00:26:19,359
Do I have the right system, processes,
712
00:26:19,409 --> 00:26:21,539
and the tools in place independent
713
00:26:21,569 --> 00:26:23,589
of what technology I'm consuming
714
00:26:23,589 --> 00:26:25,389
to protect myself and control that?
715
00:26:26,719 --> 00:26:27,149
That's awesome.
716
00:26:28,359 --> 00:26:29,589
So let's go a little deeper
717
00:26:29,589 --> 00:26:31,199
into your sort of subject matter
718
00:26:31,199 --> 00:26:32,459
expertise, which is governance
719
00:26:32,459 --> 00:26:33,739
and continuous assurance, right?
720
00:26:33,739 --> 00:26:36,229
So, so traditional governance
721
00:26:36,229 --> 00:26:37,849
is kind of seen as this, okay,
722
00:26:37,849 --> 00:26:39,329
it's a compliance checkbox.
723
00:26:39,329 --> 00:26:41,039
It's often based on documentation
724
00:26:41,039 --> 00:26:42,739
approvals, you know, some
725
00:26:42,759 --> 00:26:44,319
six-month review or whatever.
726
00:26:44,679 --> 00:26:47,219
Uh, is that sufficient for systems where,
727
00:26:47,219 --> 00:26:49,289
you know, this is like now all sort of
728
00:26:49,309 --> 00:26:52,359
agentic and, you know, sort of any one
729
00:26:52,359 --> 00:26:53,959
interaction can mess things up, right?
730
00:26:54,787 --> 00:26:56,617
You know, like you and I working in
731
00:26:56,617 --> 00:26:58,767
software development, historically
732
00:26:58,767 --> 00:27:02,747
we used to have a year-long plan with
733
00:27:02,777 --> 00:27:04,067
quarter deliv- plan delivery, like a
734
00:27:06,577 --> 00:27:06,697
roadmap.
735
00:27:06,697 --> 00:27:07,837
It's like my roadmap for next year and
736
00:27:07,857 --> 00:27:09,887
every quarter I like to deliver this.
737
00:27:10,317 --> 00:27:11,717
And then you take it to risk and
738
00:27:11,717 --> 00:27:13,297
compliance, they evaluate, they say,
739
00:27:13,297 --> 00:27:14,857
"Okay, we are gonna have a quarterly
740
00:27:15,117 --> 00:27:17,017
review for you," and all of this.
741
00:27:17,217 --> 00:27:18,797
That's the system that was working.
742
00:27:19,467 --> 00:27:21,427
Now, roadmaps are so
743
00:27:21,467 --> 00:27:23,857
agile, continuous delivery.
744
00:27:24,457 --> 00:27:26,397
We are changing things fast and
745
00:27:26,397 --> 00:27:30,877
deliver fast, and assets are being
746
00:27:30,887 --> 00:27:33,497
developed at a faster, much faster
747
00:27:34,497 --> 00:27:36,247
speed that it can be governed.
748
00:27:36,697 --> 00:27:38,987
So in this world, the question
749
00:27:38,987 --> 00:27:42,227
becomes, how can we protect ourself?
750
00:27:42,227 --> 00:27:43,527
How can we accommodate this
751
00:27:43,587 --> 00:27:45,317
so we are not slowing down the
752
00:27:45,317 --> 00:27:47,387
systems but we do the right thing?
753
00:27:47,657 --> 00:27:48,317
And I think that's
754
00:27:48,317 --> 00:27:49,467
really the core to focus.
755
00:27:50,367 --> 00:27:50,897
Interesting.
756
00:27:51,577 --> 00:27:53,959
So, what is the difference between a
757
00:27:53,959 --> 00:27:55,949
governing and AI system and continuously
758
00:27:55,949 --> 00:27:58,689
assuring that it is behaving as intended?
759
00:27:58,689 --> 00:28:00,819
You know, like how has… know,
760
00:28:00,819 --> 00:28:01,949
is it like governance has become
761
00:28:01,949 --> 00:28:03,779
more runtime now because of agents?
762
00:28:03,799 --> 00:28:04,519
You know, have you seen
763
00:28:04,519 --> 00:28:05,389
that sort of shift?
764
00:28:05,469 --> 00:28:05,729
Yeah.
765
00:28:06,051 --> 00:28:07,221
Yeah, it's the, it's the,
766
00:28:07,241 --> 00:28:08,201
a very good question.
767
00:28:08,221 --> 00:28:10,131
It's the entire life cycle.
768
00:28:10,241 --> 00:28:12,771
Historically, if someone wanted
769
00:28:12,771 --> 00:28:16,167
to build an application, they
770
00:28:16,167 --> 00:28:18,557
would come in, submit a use case.
771
00:28:18,907 --> 00:28:20,377
They would say how they
772
00:28:20,377 --> 00:28:22,847
are planning to use AI for.
773
00:28:23,307 --> 00:28:25,977
Someone would review and approve and say,
774
00:28:25,977 --> 00:28:28,277
"Hey, okay, you can start building it."
775
00:28:28,693 --> 00:28:29,003
Mm-hmm.
776
00:28:29,017 --> 00:28:31,737
The reality of today is our developers…
777
00:28:31,787 --> 00:28:32,927
or not just developers,
778
00:28:32,937 --> 00:28:34,217
everyone is a builder.
779
00:28:34,267 --> 00:28:35,757
Literally everyone is a builder.
780
00:28:35,777 --> 00:28:37,217
They go to an AI-assisted
781
00:28:37,267 --> 00:28:39,067
development building environment,
782
00:28:39,127 --> 00:28:40,247
and they start building.
783
00:28:41,127 --> 00:28:43,057
Let's say I'm a developer.
784
00:28:43,097 --> 00:28:44,887
I go in, and I start.
785
00:28:44,887 --> 00:28:47,597
The first thing that I say is, "I
786
00:28:47,597 --> 00:28:49,857
would like to make a customer care
787
00:28:49,857 --> 00:28:52,587
application that does this and this."
788
00:28:53,537 --> 00:28:54,897
The first moment that they think
789
00:28:54,897 --> 00:28:56,537
of an idea, so nothing has yet
790
00:28:56,567 --> 00:28:58,477
built, we need to be there.
791
00:28:59,097 --> 00:29:00,627
Governance needs to be there.
792
00:29:00,647 --> 00:29:03,497
We do a quick use case analysis,
793
00:29:03,507 --> 00:29:05,367
similarity analysis to figure out
794
00:29:05,367 --> 00:29:07,517
behind the scene, figure out if there
795
00:29:07,517 --> 00:29:10,297
are similar applications and use
796
00:29:10,297 --> 00:29:12,227
cases that are already in the system.
797
00:29:12,647 --> 00:29:14,957
And if they are, and they are
798
00:29:14,977 --> 00:29:16,967
approved with certain constraints
799
00:29:17,177 --> 00:29:19,097
and guardrails, we come back and
800
00:29:19,097 --> 00:29:20,947
say, "Oh, good thing you asked.
801
00:29:21,047 --> 00:29:22,467
Implement these three guardrails,
802
00:29:22,477 --> 00:29:24,657
and you're good to go," right? Or if
803
00:29:24,657 --> 00:29:26,147
it's a high-risk situation, we are
804
00:29:26,147 --> 00:29:27,767
gonna notify risk and compliance that,
805
00:29:27,767 --> 00:29:29,597
"Hey, someone, um, is thinking of it.
806
00:29:30,317 --> 00:29:31,027
Flag it.
807
00:29:31,057 --> 00:29:32,397
Go, go look into that."
808
00:29:32,487 --> 00:29:32,717
Someone's
809
00:29:32,777 --> 00:29:33,047
So
810
00:29:33,157 --> 00:29:34,097
a nuclear bomb in our
811
00:29:37,877 --> 00:29:37,947
blanket
812
00:29:37,947 --> 00:29:39,607
So this, not even… nothing is
813
00:29:39,627 --> 00:29:41,377
built yet, and you are there.
814
00:29:41,867 --> 00:29:45,357
Um, and obviously then, um, preparation
815
00:29:45,357 --> 00:29:47,957
to go to the runtime, once you have
816
00:29:47,957 --> 00:29:49,897
the controls well-defined, they need to
817
00:29:49,897 --> 00:29:51,797
provide evidence that they are actually
818
00:29:51,807 --> 00:29:53,967
implementing it as part of your approval.
819
00:29:54,247 --> 00:29:55,527
And once you get that, then
820
00:29:55,527 --> 00:29:56,867
you go to runtime and the
821
00:29:57,237 --> 00:29:58,737
traditional things that we had.
822
00:29:59,607 --> 00:30:01,347
It's even more important to
823
00:30:01,347 --> 00:30:04,717
continuously monitor this in runtime
824
00:30:04,727 --> 00:30:06,167
because of the certain level of
825
00:30:06,207 --> 00:30:08,457
autonomy that the agents have.
826
00:30:09,377 --> 00:30:10,927
Um, so just looking at
827
00:30:10,927 --> 00:30:12,317
the tracing is not enough.
828
00:30:12,747 --> 00:30:14,067
You'll like to make sure… Because,
829
00:30:14,067 --> 00:30:16,377
because looking at tracing is showing
830
00:30:16,397 --> 00:30:17,697
you what happened in the past.
831
00:30:17,697 --> 00:30:18,507
Something went wrong.
832
00:30:18,527 --> 00:30:19,357
The breach happened.
833
00:30:19,367 --> 00:30:20,957
The control cut that,
834
00:30:21,407 --> 00:30:22,717
but maybe it's too late.
835
00:30:22,817 --> 00:30:24,207
If it's, for example, leaking
836
00:30:24,207 --> 00:30:26,057
information, it's already leaked, right?
837
00:30:26,307 --> 00:30:28,927
So you wanna be ahead of it, proactively
838
00:30:28,927 --> 00:30:31,867
look for patterns to figure out what's,
839
00:30:31,897 --> 00:30:34,157
what's the area that is missing or, like,
840
00:30:34,157 --> 00:30:36,407
how can I go and do policy enforcement
841
00:30:36,417 --> 00:30:38,147
before this action has happened.
842
00:30:38,907 --> 00:30:43,227
So, um, obviously the traditional, um,
843
00:30:43,387 --> 00:30:45,547
governance was not looking into this
844
00:30:45,547 --> 00:30:48,307
continuous compliance and regulations.
845
00:30:48,537 --> 00:30:50,427
The traditional one quarterly
846
00:30:50,437 --> 00:30:51,787
was not designed for this.
847
00:30:51,837 --> 00:30:55,367
Everything is so, uh, continuous, and in
848
00:30:55,367 --> 00:30:58,067
order to deliver that continuous delivery
849
00:30:58,067 --> 00:31:00,097
of automated reports, applicability
850
00:31:00,997 --> 00:31:04,007
analysis, and, um, monitoring and
851
00:31:04,007 --> 00:31:06,077
enforcement, you really need to have
852
00:31:06,077 --> 00:31:09,315
a good grasp of the context Um, and,
853
00:31:09,315 --> 00:31:11,035
and, and this is, this is, this is
854
00:31:11,115 --> 00:31:13,075
really the part that fascinates me about
855
00:31:13,075 --> 00:31:15,045
governance versus just understanding
856
00:31:15,055 --> 00:31:18,145
AI inventories or other assets, because
857
00:31:18,165 --> 00:31:20,375
governance is about relationships.
858
00:31:20,725 --> 00:31:24,235
Relationships from AI assets to use cases,
859
00:31:24,245 --> 00:31:27,145
from use cases exposing risk, from risk
860
00:31:27,185 --> 00:31:29,955
mitigated by controls, controls measured
861
00:31:29,955 --> 00:31:32,185
against metrics, metrics showing that
862
00:31:32,195 --> 00:31:34,215
the business objective is achieved.
863
00:31:35,145 --> 00:31:36,985
If we look into end-to-end, and like
864
00:31:36,985 --> 00:31:38,795
in, in our product we call it governance
865
00:31:38,795 --> 00:31:41,475
graph, really trying to extract all
866
00:31:41,485 --> 00:31:44,005
the relationships all the way from AI
867
00:31:44,055 --> 00:31:47,235
assets back to your business initiatives,
868
00:31:47,523 --> 00:31:47,933
Yeah
869
00:31:47,955 --> 00:31:50,425
you have a complete 360 view
870
00:31:50,425 --> 00:31:52,405
of what exactly is happening.
871
00:31:52,945 --> 00:31:54,595
And if control is breached, you know
872
00:31:55,035 --> 00:31:57,055
exactly what you are exposed to.
873
00:31:57,115 --> 00:31:59,405
That historically there was
874
00:31:59,435 --> 00:32:01,175
not much need on a continuous
875
00:32:01,175 --> 00:32:03,005
basis for this, but now there is
876
00:32:03,693 --> 00:32:04,243
Makes sense.
877
00:32:04,573 --> 00:32:06,583
So now, uh, many, uh, these large
878
00:32:06,583 --> 00:32:10,203
companies have vendor solutions, right?
879
00:32:10,203 --> 00:32:12,503
So you may have like a model from someone,
880
00:32:12,923 --> 00:32:15,483
data from another agentic framework, some
881
00:32:15,533 --> 00:32:17,553
some other one, maybe some other tools.
882
00:32:18,033 --> 00:32:19,893
So how do they think about AI
883
00:32:19,913 --> 00:32:21,093
governance, you know, across
884
00:32:21,093 --> 00:32:22,183
all of these things, you know?
885
00:32:22,433 --> 00:32:23,383
What, what's the, what's a
886
00:32:23,383 --> 00:32:24,543
good way to think about it?
887
00:32:25,025 --> 00:32:26,605
No, that's, that's a very good question,
888
00:32:26,635 --> 00:32:28,585
and this is where most governance
889
00:32:28,635 --> 00:32:31,415
approaches fall short because they
890
00:32:31,415 --> 00:32:34,145
treat enterprise AI asset and AI risk
891
00:32:34,155 --> 00:32:36,795
in isolation from enterprise operations.
892
00:32:37,055 --> 00:32:38,315
Take third-party risk as an
893
00:32:38,315 --> 00:32:39,755
example, the one that you mentioned.
894
00:32:40,205 --> 00:32:41,675
AI assets are coming from different
895
00:32:41,675 --> 00:32:44,335
places when different vendors, and you
896
00:32:44,335 --> 00:32:47,605
need to know who you are trusting, right?
897
00:32:48,175 --> 00:32:50,395
Um, there is a third-party risk
898
00:32:50,395 --> 00:32:51,885
management in the GRC world
899
00:32:51,935 --> 00:32:53,745
that we brought to our world.
900
00:32:53,825 --> 00:32:55,085
So for example, in my product,
901
00:32:55,105 --> 00:32:56,155
we have partnerships with
902
00:32:56,175 --> 00:32:57,535
companies like D&B, RiskRecon,
903
00:32:58,615 --> 00:33:00,995
Security Scorecard, Rapid Rating.
904
00:33:01,005 --> 00:33:03,415
These all give you deep vendor
905
00:33:03,415 --> 00:33:05,745
assessment coverage on a daily basis,
906
00:33:05,845 --> 00:33:07,575
including… And some of them even
907
00:33:07,575 --> 00:33:10,055
including real-time incident monitoring.
908
00:33:10,395 --> 00:33:12,495
So if you are using one of these
909
00:33:12,535 --> 00:33:14,795
third parties, you need to have the
910
00:33:14,795 --> 00:33:17,515
information about the supplier of these
911
00:33:17,545 --> 00:33:19,365
assets, not just the assets itself.
912
00:33:19,385 --> 00:33:20,855
It's like, what other risk I'm
913
00:33:20,855 --> 00:33:24,165
exposed to, and monitoring them and
914
00:33:24,185 --> 00:33:25,975
putting together a control around it.
915
00:33:26,295 --> 00:33:28,195
And unfortunately, most of the AI
916
00:33:28,245 --> 00:33:31,455
governance, um, solutions out there that
917
00:33:31,455 --> 00:33:34,645
I see, they focus on just the AI assets
918
00:33:35,043 --> 00:33:35,053
Mm-hmm.
919
00:33:35,695 --> 00:33:37,175
in isolations from all the
920
00:33:37,175 --> 00:33:38,715
operations, and this operational
921
00:33:38,715 --> 00:33:40,545
risk is not just third party.
922
00:33:40,545 --> 00:33:42,345
We talked about business continuity risk.
923
00:33:42,375 --> 00:33:44,025
We talked… Uh, other examples
924
00:33:44,025 --> 00:33:45,505
are IT risk, for example,
925
00:33:45,525 --> 00:33:47,295
or the environmental risk.
926
00:33:47,295 --> 00:33:48,795
So there is a series of them
927
00:33:48,815 --> 00:33:51,325
that come into the picture really
928
00:33:51,325 --> 00:33:53,065
connecting the AI assets to the
929
00:33:53,065 --> 00:33:54,465
operations of the enterprise.
930
00:33:55,805 --> 00:33:56,175
Very cool.
931
00:33:57,025 --> 00:33:58,285
One of the things that you mentioned
932
00:33:58,285 --> 00:34:00,395
as part of this is that you need
933
00:34:00,395 --> 00:34:02,185
to monitor the agentic behavior,
934
00:34:02,635 --> 00:34:04,315
and tracing is not enough, right?
935
00:34:04,505 --> 00:34:06,325
So there's this concept of evals
936
00:34:06,345 --> 00:34:08,035
and evaluating the agentic behavior
937
00:34:08,065 --> 00:34:09,295
that is a must because, you know,
938
00:34:10,065 --> 00:34:11,795
can collect the agentic trajectory,
939
00:34:11,795 --> 00:34:13,535
but if you cannot evaluate it, then
940
00:34:13,625 --> 00:34:16,035
it's not, it's not much meaningful.
941
00:34:16,475 --> 00:34:18,955
So, know, traditional, you know, model
942
00:34:18,965 --> 00:34:21,085
evaluation score, just the final output.
943
00:34:21,085 --> 00:34:22,675
Right now with agents like this, you
944
00:34:22,675 --> 00:34:23,625
know, we've seen the whole OpenAI
945
00:34:24,705 --> 00:34:26,675
Hugging Face incident where something
946
00:34:26,675 --> 00:34:28,295
happened and, you know, agent hacked
947
00:34:28,295 --> 00:34:30,055
into production servers of Hugging Face.
948
00:34:30,845 --> 00:34:32,795
do you think… How do, how do, how,
949
00:34:32,895 --> 00:34:34,575
how does teams need to think about the
950
00:34:34,605 --> 00:34:36,715
whole agentic trajectory evaluation,
951
00:34:36,865 --> 00:34:38,885
planning, reasoning, tool selection?
952
00:34:39,315 --> 00:34:40,285
Any thoughts on that?
953
00:34:42,157 --> 00:34:43,697
Well, back to, like, I'm coming from
954
00:34:43,757 --> 00:34:45,607
the governance world, back to control.
955
00:34:46,477 --> 00:34:48,827
So, um, and, and controls
956
00:34:48,867 --> 00:34:50,767
are not just risk-driven.
957
00:34:50,777 --> 00:34:52,807
Sometimes it's, for example, cost-driven.
958
00:34:52,827 --> 00:34:55,677
If, uh, a agent is using more than
959
00:34:55,687 --> 00:34:57,307
certain tokens, you want to stop
960
00:34:57,307 --> 00:34:59,537
it before you get the bill, Right?
961
00:34:59,887 --> 00:35:00,317
right
962
00:35:00,317 --> 00:35:02,827
So… or sometimes maybe it's
963
00:35:02,827 --> 00:35:04,507
the payload or access things
964
00:35:04,507 --> 00:35:06,947
that you like to, um, stop them.
965
00:35:07,297 --> 00:35:10,177
Um, so back to this, I would,
966
00:35:10,187 --> 00:35:13,987
I would start with, one, what
967
00:35:14,007 --> 00:35:15,817
this agent is supposed to do.
968
00:35:16,227 --> 00:35:16,627
Yeah.
969
00:35:16,997 --> 00:35:19,747
What are the risk or dimensions
970
00:35:20,027 --> 00:35:22,107
that I care to control around
971
00:35:22,107 --> 00:35:23,577
the behavior of the agent?
972
00:35:23,939 --> 00:35:24,239
Yeah
973
00:35:24,477 --> 00:35:27,507
Can I map them to some metrics that I can
974
00:35:27,507 --> 00:35:30,257
go back to tracing and start measuring
975
00:35:30,897 --> 00:35:33,157
as, like, the definition of signals?
976
00:35:33,187 --> 00:35:34,767
Like, what is the threshold that I need
977
00:35:34,767 --> 00:35:37,337
to be notified that it's working or not?
978
00:35:37,677 --> 00:35:39,097
And then throughout the life
979
00:35:39,097 --> 00:35:41,727
cycle, when it's the build time, I
980
00:35:41,727 --> 00:35:44,037
need to be able to evaluate them.
981
00:35:44,447 --> 00:35:46,617
Like, sometimes benchmark comes in,
982
00:35:46,647 --> 00:35:48,647
like if you want to pick a third-party
983
00:35:48,647 --> 00:35:50,977
agent, for example, you look into their
984
00:35:51,057 --> 00:35:53,607
offline evaluations that the company
985
00:35:53,607 --> 00:35:55,827
has to make sure you get the best one.
986
00:35:56,227 --> 00:35:57,807
Then we have, um… so that's
987
00:35:57,807 --> 00:35:59,077
the offline evaluation.
988
00:35:59,087 --> 00:36:01,687
Then during runtime, when you go to
989
00:36:01,687 --> 00:36:03,497
the runtime, you want to make sure
990
00:36:03,527 --> 00:36:06,317
that you do overtime evaluation,
991
00:36:06,367 --> 00:36:07,987
like that is historically we
992
00:36:07,987 --> 00:36:09,917
did it for traditional ML tool.
993
00:36:10,027 --> 00:36:13,257
If the agent is drifting or on the
994
00:36:13,257 --> 00:36:16,007
behavior and stuff over time, do I
995
00:36:16,017 --> 00:36:18,277
have the right metrics to track that?
996
00:36:18,527 --> 00:36:22,497
And also, um, for the agent itself
997
00:36:22,607 --> 00:36:24,807
as it's, uh, making the decisions.
998
00:36:24,817 --> 00:36:26,227
So for example, for us, like
999
00:36:26,417 --> 00:36:28,487
we, we see agent is not really
1000
00:36:28,487 --> 00:36:30,657
a black box of input and output.
1001
00:36:31,177 --> 00:36:32,427
The input goes in.
1002
00:36:33,167 --> 00:36:34,017
Think of a node.
1003
00:36:34,107 --> 00:36:36,037
It breaks it down to a series of
1004
00:36:36,147 --> 00:36:38,957
steps, and then for every step, it
1005
00:36:38,957 --> 00:36:42,457
decides to do a number of calling.
1006
00:36:43,027 --> 00:36:45,237
At every step in that node, you need
1007
00:36:45,237 --> 00:36:47,497
to be there to do the evaluation.
1008
00:36:48,427 --> 00:36:50,517
So there is evaluation in the node,
1009
00:36:50,617 --> 00:36:52,507
there is evaluation over time, and
1010
00:36:52,537 --> 00:36:54,747
evaluation at build time, which is most
1011
00:36:55,027 --> 00:36:57,257
of the time offline, to make sure that
1012
00:36:57,257 --> 00:36:59,307
you have all the metrics monitoring
1013
00:36:59,317 --> 00:37:02,447
that you need to monitor in order
1014
00:37:02,477 --> 00:37:05,037
to make sure your control is working
1015
00:37:05,997 --> 00:37:06,687
Yeah, yeah.
1016
00:37:07,067 --> 00:37:08,387
And that is also hard, right?
1017
00:37:08,387 --> 00:37:09,807
In the sense, you know, many times
1018
00:37:10,167 --> 00:37:11,847
you don't know what good looks like.
1019
00:37:11,847 --> 00:37:13,017
You know, it's, it's sort of like
1020
00:37:13,017 --> 00:37:14,517
what is the ground truth, you know?
1021
00:37:14,647 --> 00:37:16,597
Is, is… Uh, and that's, that's the hard
1022
00:37:16,597 --> 00:37:18,137
part that, you know, industry is facing
1023
00:37:18,137 --> 00:37:19,877
right now in terms of how effectively
1024
00:37:20,147 --> 00:37:22,367
you want to evaluate the, an AI system.
1025
00:37:22,819 --> 00:37:23,429
Exactly.
1026
00:37:23,859 --> 00:37:26,269
But also the goal for evaluation.
1027
00:37:26,269 --> 00:37:28,089
Sometimes the goal for evaluation is
1028
00:37:28,089 --> 00:37:31,029
to make sure that, um, the controls are
1029
00:37:31,029 --> 00:37:33,539
implemented correctly, like guardrails,
1030
00:37:33,549 --> 00:37:35,639
but sometimes the goal is optimization.
1031
00:37:35,909 --> 00:37:37,719
Like, you like to optimize your cost.
1032
00:37:38,209 --> 00:37:40,329
You like to optimize your prompt, right?
1033
00:37:40,429 --> 00:37:42,229
And so it's important to understand
1034
00:37:42,229 --> 00:37:44,489
what is the goal for that evaluation too
1035
00:37:45,123 --> 00:37:45,403
Yeah.
1036
00:37:45,823 --> 00:37:47,163
So, so I guess, you know, when you
1037
00:37:47,163 --> 00:37:49,413
have seen, uh, agentic journeys of your
1038
00:37:49,433 --> 00:37:51,433
customers or, or sort of other teams
1039
00:37:51,433 --> 00:37:53,583
that you spoke to, what, what's, what
1040
00:37:53,593 --> 00:37:56,383
have you seen, you know, things breaking?
1041
00:37:56,463 --> 00:37:58,123
You know, when an enterprise goes from,
1042
00:37:58,123 --> 00:37:59,773
say, let's say a handful of agents to,
1043
00:37:59,773 --> 00:38:02,393
like, maybe 100-plus agents in production,
1044
00:38:02,403 --> 00:38:04,303
you know, across different departments.
1045
00:38:04,303 --> 00:38:05,223
You know, where, where have you
1046
00:38:05,223 --> 00:38:08,273
seen arise, you know, agents running
1047
00:38:08,273 --> 00:38:09,903
in production and misbehaving?
1048
00:38:10,339 --> 00:38:10,739
Yeah.
1049
00:38:11,109 --> 00:38:14,569
Uh, you know, we frequently run
1050
00:38:14,569 --> 00:38:16,489
customer advisory boards, and in the
1051
00:38:16,489 --> 00:38:18,909
last one I asked a similar question.
1052
00:38:18,909 --> 00:38:21,369
I'm like, "What is your top challenge
1053
00:38:21,603 --> 00:38:21,893
Yeah
1054
00:38:21,999 --> 00:38:23,069
it comes to agent
1055
00:38:23,229 --> 00:38:24,859
adoption?" And guess what?
1056
00:38:24,889 --> 00:38:26,069
We already talked about that.
1057
00:38:26,119 --> 00:38:28,799
The number one is accountability.
1058
00:38:29,299 --> 00:38:30,229
Who is accountable?
1059
00:38:30,819 --> 00:38:31,159
Mm-hmm.
1060
00:38:31,299 --> 00:38:33,069
Especially because the leaders
1061
00:38:33,069 --> 00:38:34,919
that we talk to, they're usually
1062
00:38:34,919 --> 00:38:37,099
responsible for their piece of the work.
1063
00:38:37,789 --> 00:38:41,959
And here we are talking about multiple
1064
00:38:42,019 --> 00:38:45,469
departments within the business making the
1065
00:38:45,469 --> 00:38:49,369
call, but also the exposure to external
1066
00:38:49,409 --> 00:38:51,439
entities like model provider, agent
1067
00:38:51,439 --> 00:38:53,309
provider, the tool provider, the person
1068
00:38:53,309 --> 00:38:55,599
that makes a decision to surface it to the
1069
00:38:55,599 --> 00:38:58,919
user or the bad- actor, the user itself.
1070
00:39:00,149 --> 00:39:02,279
And, and, and that- that's really the
1071
00:39:02,279 --> 00:39:05,209
number one, um, problem that they keep
1072
00:39:05,249 --> 00:39:08,429
citing because this, this is not something
1073
00:39:08,429 --> 00:39:11,019
that you just solve with the technology.
1074
00:39:11,019 --> 00:39:13,009
You can… The thing that you can do as
1075
00:39:13,009 --> 00:39:16,619
an enterprise is tracing instrumentation
1076
00:39:16,639 --> 00:39:19,679
to put together all the instrumentation
1077
00:39:19,679 --> 00:39:21,529
so you can go trace back and figure
1078
00:39:21,529 --> 00:39:23,089
out where it was stemmed from.
1079
00:39:23,349 --> 00:39:24,849
But then enforcing it to
1080
00:39:24,889 --> 00:39:27,679
third party requires external
1081
00:39:27,679 --> 00:39:29,739
regulation to catch up, right?
1082
00:39:30,159 --> 00:39:32,819
Um, so it's not a one person, one
1083
00:39:32,819 --> 00:39:36,349
company, get it done and move on.
1084
00:39:36,459 --> 00:39:36,729
Yeah
1085
00:39:37,239 --> 00:39:37,719
makes sense.
1086
00:39:37,939 --> 00:39:38,949
Let's take some relevant
1087
00:39:38,949 --> 00:39:39,809
audience questions.
1088
00:39:39,819 --> 00:39:41,439
There's a question from Kenneth Roy.
1089
00:39:41,829 --> 00:39:43,389
You know, what would be a good governance
1090
00:39:43,409 --> 00:39:45,389
model organizational chart across
1091
00:39:45,429 --> 00:39:48,309
enterprise companies when many departments
1092
00:39:48,309 --> 00:39:50,189
and functions that are individually trying
1093
00:39:50,189 --> 00:39:52,019
to adopt AI agents into their workflows?
1094
00:39:52,019 --> 00:39:52,819
Is there a responsible
1095
00:39:52,899 --> 00:39:53,789
position or committee?
1096
00:39:54,139 --> 00:39:55,109
So it sort of aligns with
1097
00:39:55,219 --> 00:39:56,439
your accountability question,
1098
00:39:56,439 --> 00:39:57,559
but any thoughts on that?
1099
00:39:57,827 --> 00:39:58,207
Yeah.
1100
00:39:58,377 --> 00:40:00,507
Usually the way that we see it is we
1101
00:40:00,507 --> 00:40:02,837
have the line of defense within the
1102
00:40:02,837 --> 00:40:04,517
enterprise and the line of business.
1103
00:40:04,877 --> 00:40:06,547
Line of defense, the first line of
1104
00:40:06,547 --> 00:40:08,507
defense is obviously the business owner.
1105
00:40:08,507 --> 00:40:10,167
The second line of defense is risk
1106
00:40:10,167 --> 00:40:11,907
and compliance officers, and the third
1107
00:40:11,907 --> 00:40:13,407
line of defense is the audit team.
1108
00:40:13,977 --> 00:40:15,937
Those… The line of defense is
1109
00:40:15,937 --> 00:40:18,027
usually they need to have access
1110
00:40:18,027 --> 00:40:20,087
to the entire AI estate, no matter
1111
00:40:20,137 --> 00:40:21,797
which department you are sitting in.
1112
00:40:22,107 --> 00:40:24,477
So if you are part of the business
1113
00:40:24,477 --> 00:40:27,707
side, obviously you're subject to the,
1114
00:40:27,997 --> 00:40:29,857
um, guardrails and controls that the
1115
00:40:29,857 --> 00:40:32,007
risk and compliance is exposing to.
1116
00:40:32,487 --> 00:40:35,077
Um, but when you are, uh, um…
1117
00:40:36,297 --> 00:40:38,297
Yeah, and then on the business side,
1118
00:40:38,647 --> 00:40:41,557
you just really need to make sure that
1119
00:40:41,557 --> 00:40:44,137
you have the right control plane to
1120
00:40:44,147 --> 00:40:47,187
control your realm of the business versus
1121
00:40:47,197 --> 00:40:49,947
360 entire AI estate of the company
1122
00:40:51,787 --> 00:40:52,097
Got it.
1123
00:40:52,127 --> 00:40:52,447
Got it.
1124
00:40:52,727 --> 00:40:54,647
And so maybe a, a relevant follow-up,
1125
00:40:54,647 --> 00:40:56,967
you know, Satvik Parasa asks, "What
1126
00:40:56,967 --> 00:40:58,767
type of metrics would convince you
1127
00:40:58,767 --> 00:41:00,727
that an organization's AI governance
1128
00:41:00,737 --> 00:41:02,977
program is actually effective rather
1129
00:41:02,977 --> 00:41:04,277
than just compliant?" You know, what
1130
00:41:04,287 --> 00:41:06,297
does-- what defines good governance?
1131
00:41:07,063 --> 00:41:08,833
I, I love this question because most
1132
00:41:08,833 --> 00:41:10,413
of the time people think about AI
1133
00:41:10,413 --> 00:41:13,273
assets use case mitigated by risk
1134
00:41:13,293 --> 00:41:14,873
control metric, and we are done.
1135
00:41:15,403 --> 00:41:16,883
But if you notice, like, there was
1136
00:41:16,903 --> 00:41:19,683
one last piece of the puzzle, metric
1137
00:41:19,713 --> 00:41:21,763
connected to business objective.
1138
00:41:22,113 --> 00:41:25,083
So in the governance graph, you
1139
00:41:25,083 --> 00:41:28,483
can trace back from AI assets all
1140
00:41:28,483 --> 00:41:30,263
the way to business initiatives.
1141
00:41:30,343 --> 00:41:31,533
Because at the end of the day,
1142
00:41:31,553 --> 00:41:33,723
we need to hold AI accountable
1143
00:41:33,733 --> 00:41:35,473
to deliver on the outcome that it
1144
00:41:35,473 --> 00:41:38,443
was promised to deliver, right?
1145
00:41:38,893 --> 00:41:42,343
Um, and this is, this is your direct segue
1146
00:41:42,383 --> 00:41:46,233
of AI assets to business, um, initiatives.
1147
00:41:46,523 --> 00:41:47,763
It gives you two things.
1148
00:41:47,863 --> 00:41:50,373
One, obviously over time gener-
1149
00:41:50,383 --> 00:41:53,373
general layer at a, on a monthly basis
1150
00:41:53,373 --> 00:41:55,513
or weekly basis, you have access to
1151
00:41:55,513 --> 00:41:57,873
the dashboards that can show you the
1152
00:41:57,883 --> 00:41:59,873
effectiveness of those AI assets.
1153
00:41:59,873 --> 00:42:01,433
But the second thing that it also
1154
00:42:01,443 --> 00:42:04,033
gives you, if when you have controls
1155
00:42:04,033 --> 00:42:07,173
in place and something wrong happens,
1156
00:42:07,213 --> 00:42:10,013
like breach of control, you can trace it
1157
00:42:10,023 --> 00:42:13,443
back exactly to what KPIs was on breach.
1158
00:42:13,683 --> 00:42:15,173
What is the impact to my
1159
00:42:15,173 --> 00:42:17,163
financials and business
1160
00:42:17,503 --> 00:42:17,763
Right
1161
00:42:18,453 --> 00:42:20,393
from a metric that was breached
1162
00:42:20,473 --> 00:42:22,073
in a tracing dashboard?
1163
00:42:22,353 --> 00:42:23,493
And I think that's the beauty
1164
00:42:23,493 --> 00:42:25,033
of the governance graph.
1165
00:42:25,667 --> 00:42:26,647
Yeah, makes sense.
1166
00:42:27,017 --> 00:42:29,477
And, and so I guess, uh, back
1167
00:42:29,477 --> 00:42:30,897
to that sort of accountability
1168
00:42:30,897 --> 00:42:32,257
and self-attestation, right?
1169
00:42:32,287 --> 00:42:33,797
At, at what point do you think
1170
00:42:33,807 --> 00:42:36,667
companies can't rely on self-attestation
1171
00:42:36,667 --> 00:42:38,087
anymore and need an independent
1172
00:42:38,087 --> 00:42:40,927
assurance for higher risk AI systems?
1173
00:42:43,787 --> 00:42:45,457
I wish I had an answer for this.
1174
00:42:46,207 --> 00:42:47,837
You know, recently I had a
1175
00:42:47,837 --> 00:42:50,527
conversation with a startup founder.
1176
00:42:50,717 --> 00:42:54,347
Um, he used to be a lawyer, and
1177
00:42:54,347 --> 00:42:56,257
now he has a company that does
1178
00:42:56,567 --> 00:42:58,387
this exactly for the companies.
1179
00:42:58,597 --> 00:43:01,547
So you make an AI decision, um,
1180
00:43:01,577 --> 00:43:05,087
at this point of time, you pick a
1181
00:43:05,147 --> 00:43:08,627
third-party AI provider, model provider.
1182
00:43:09,597 --> 00:43:13,057
The startup would run some tests.
1183
00:43:13,557 --> 00:43:15,907
They would legally document all of
1184
00:43:16,287 --> 00:43:19,307
this to show that as an evidence
1185
00:43:19,337 --> 00:43:21,897
that at that point of time you
1186
00:43:21,897 --> 00:43:23,887
considered this option, you evaluate
1187
00:43:23,887 --> 00:43:25,687
it and you protect it yourself.
1188
00:43:26,557 --> 00:43:28,757
The regulation is evolving rapidly.
1189
00:43:28,917 --> 00:43:30,747
Nobody knows in the course how
1190
00:43:30,747 --> 00:43:33,567
AI is gonna be held accountable.
1191
00:43:34,297 --> 00:43:36,267
There is lots of movements in
1192
00:43:36,277 --> 00:43:38,467
the market in terms of startups,
1193
00:43:38,487 --> 00:43:41,817
insurance providers, but it has not
1194
00:43:41,817 --> 00:43:44,297
yet emerged into a single unified.
1195
00:43:44,597 --> 00:43:46,277
And, and just think about that at the
1196
00:43:46,307 --> 00:43:48,787
global layer, it's even more complicated
1197
00:43:48,787 --> 00:43:50,957
because every company… every country
1198
00:43:51,157 --> 00:43:52,887
has their own regulations when it
1199
00:43:52,917 --> 00:43:55,467
comes into enforcement of these things
1200
00:43:56,323 --> 00:43:57,123
Yeah, absolutely.
1201
00:43:57,493 --> 00:44:00,343
And now it's, it's actually has become
1202
00:44:00,343 --> 00:44:02,173
a best practice, not just a regulatory
1203
00:44:02,173 --> 00:44:03,573
shield for most companies, right?
1204
00:44:03,583 --> 00:44:05,023
Essentially, as you articulated,
1205
00:44:05,023 --> 00:44:06,783
you know, it helps you go from
1206
00:44:07,263 --> 00:44:07,553
Faster
1207
00:44:07,633 --> 00:44:10,523
APIs and become your agent native.
1208
00:44:10,583 --> 00:44:11,453
Yeah, that makes sense.
1209
00:44:11,883 --> 00:44:13,553
Uh, so there's also this whole
1210
00:44:13,573 --> 00:44:15,593
sort of debate versus human in
1211
00:44:15,593 --> 00:44:18,323
the loop versus human in the loop.
1212
00:44:18,333 --> 00:44:19,583
You know, when
1213
00:44:19,583 --> 00:44:19,893
does it become-
1214
00:44:20,019 --> 00:44:21,119
what's the new thing?
1215
00:44:21,299 --> 00:44:21,929
Human in the lead
1216
00:44:22,755 --> 00:44:23,595
Human in the lead, yeah.
1217
00:44:23,595 --> 00:44:25,225
So what's your take on that?
1218
00:44:25,225 --> 00:44:27,685
You know, uh, uh, where do you
1219
00:44:27,685 --> 00:44:28,905
suggest, you know, human in the
1220
00:44:28,905 --> 00:44:30,275
loop versus not human in the loop?
1221
00:44:30,829 --> 00:44:32,239
Well, we talked about human,
1222
00:44:32,489 --> 00:44:34,979
um, becoming an orchestrator.
1223
00:44:35,749 --> 00:44:37,359
Once you are an orchestrator,
1224
00:44:37,429 --> 00:44:38,909
you are really the lead, right?
1225
00:44:40,239 --> 00:44:42,979
That's why I like human in the lead as
1226
00:44:42,979 --> 00:44:44,969
a good metaphor where it says, "Oh, keep
1227
00:44:44,969 --> 00:44:47,539
human in the loop. Make sure that there
1228
00:44:47,539 --> 00:44:49,629
is human in the loop to have some sort of
1229
00:44:49,629 --> 00:44:51,719
oversight." No, like human is in the lead
1230
00:44:52,559 --> 00:44:52,839
Yeah.
1231
00:44:52,949 --> 00:44:53,489
Makes sense.
1232
00:44:53,559 --> 00:44:53,999
Makes sense.
1233
00:44:54,449 --> 00:44:57,559
So, so I guess, uh, you see, uh,
1234
00:44:58,409 --> 00:44:59,969
the categories of control plane
1235
00:44:59,969 --> 00:45:01,609
and governance kind of converging?
1236
00:45:01,609 --> 00:45:02,739
Or do you see, like, you know, it
1237
00:45:02,749 --> 00:45:05,189
becomes another, like, control plane
1238
00:45:05,239 --> 00:45:06,499
kind of becoming a foundational
1239
00:45:06,509 --> 00:45:09,379
layer, at the AI infrastructure stack?
1240
00:45:09,379 --> 00:45:10,049
You know, where do you
1241
00:45:10,049 --> 00:45:10,999
see five years from now?
1242
00:45:12,237 --> 00:45:14,217
Well, control point to me is just a tool
1243
00:45:14,597 --> 00:45:16,657
to implement what you need to govern
1244
00:45:17,247 --> 00:45:17,687
Mm-hmm.
1245
00:45:17,807 --> 00:45:18,327
control.
1246
00:45:18,997 --> 00:45:21,657
Um, you can call it control point,
1247
00:45:21,687 --> 00:45:23,217
you can call it control tower, you
1248
00:45:23,217 --> 00:45:25,227
can call it the governance console,
1249
00:45:25,537 --> 00:45:27,097
you can call it a dashboard.
1250
00:45:28,217 --> 00:45:30,807
So, so, uh, to me it's really the,
1251
00:45:30,847 --> 00:45:33,277
the concept of what it's delivering,
1252
00:45:33,317 --> 00:45:35,287
which is the four thing that we
1253
00:45:35,287 --> 00:45:37,157
talked about, the flywheel of control.
1254
00:45:37,247 --> 00:45:40,557
Defining controls, implementing controls,
1255
00:45:40,587 --> 00:45:42,417
enforcing them, and tracking them.
1256
00:45:43,157 --> 00:45:45,917
No matter how it makes sense for your
1257
00:45:45,937 --> 00:45:47,727
enterprise to implement that, this, these
1258
00:45:47,737 --> 00:45:49,747
are really the four things that I don't
1259
00:45:49,747 --> 00:45:52,277
see changing over the next five years.
1260
00:45:52,627 --> 00:45:54,497
The nature of that will be changed,
1261
00:45:54,497 --> 00:45:57,077
but GRC has been around for 20 years
1262
00:45:57,117 --> 00:45:59,597
or more, uh, and it's gonna continue to
1263
00:45:59,597 --> 00:46:01,757
be relevant for many more years to come
1264
00:46:02,627 --> 00:46:02,967
Yeah.
1265
00:46:03,577 --> 00:46:05,107
So, so when it comes to enforcing
1266
00:46:05,107 --> 00:46:06,357
controls, there's actually an interesting
1267
00:46:06,357 --> 00:46:08,197
question from Gurpreet that, "How
1268
00:46:08,197 --> 00:46:10,087
do the enterprise customers scale
1269
00:46:10,087 --> 00:46:11,507
agentic controls without pushing the
1270
00:46:11,507 --> 00:46:13,617
work to individual engineers? We are
1271
00:46:13,617 --> 00:46:15,397
trying to manage the steering files,
1272
00:46:15,397 --> 00:46:16,687
and it has become a headache." You
1273
00:46:16,687 --> 00:46:18,697
know, essentially a lot of the agentic
1274
00:46:18,747 --> 00:46:20,107
controls are being pushed to engineers
1275
00:46:20,107 --> 00:46:21,867
to encode into steering, you know,
1276
00:46:22,267 --> 00:46:23,777
steering files and, you know, it's…
1277
00:46:24,047 --> 00:46:25,837
And Gurpreet is not certainly liking it.
1278
00:46:27,107 --> 00:46:27,557
Yeah.
1279
00:46:27,587 --> 00:46:29,947
So, so one of the things that, uh, we
1280
00:46:29,947 --> 00:46:32,567
recently released is we call it use case
1281
00:46:32,587 --> 00:46:35,337
onboarding optimization or risk advisor.
1282
00:46:35,747 --> 00:46:38,287
Um, so instead of someone going and,
1283
00:46:38,327 --> 00:46:41,007
um, actually saying that, "Hey, I
1284
00:46:41,007 --> 00:46:43,227
would like officially submitting a
1285
00:46:43,227 --> 00:46:45,237
use case," and then a human comes
1286
00:46:45,237 --> 00:46:47,597
in and reviews the stuff, uh,
1287
00:46:47,627 --> 00:46:49,777
we are automating a lot of that.
1288
00:46:49,787 --> 00:46:51,787
So like use case similarity analysis
1289
00:46:51,787 --> 00:46:53,957
that we talked about automatically
1290
00:46:53,957 --> 00:46:57,337
looks into are there relevant use cases?
1291
00:46:57,347 --> 00:46:59,617
If there is one approved, move forward.
1292
00:46:59,837 --> 00:47:01,607
And through that we were able to,
1293
00:47:01,677 --> 00:47:04,997
um, in Q1 get about 55% efficiency.
1294
00:47:05,337 --> 00:47:07,427
So if you're a developer, you don't
1295
00:47:07,427 --> 00:47:09,657
wanna deal with any of this, right?
1296
00:47:10,117 --> 00:47:11,027
You're a builder.
1297
00:47:11,077 --> 00:47:12,817
You like to build the things and move
1298
00:47:12,827 --> 00:47:15,427
forward, but also you don't wanna be
1299
00:47:15,427 --> 00:47:17,707
accountable to figuring out what is the
1300
00:47:17,707 --> 00:47:19,377
guardrail that you need to implement
1301
00:47:19,377 --> 00:47:21,197
or thinking about, oh, when it goes
1302
00:47:21,197 --> 00:47:23,317
wrong, like, what do I need to do?
1303
00:47:23,327 --> 00:47:24,497
Obviously, you need to think
1304
00:47:24,507 --> 00:47:26,707
about it, but then there is a
1305
00:47:26,737 --> 00:47:28,787
structure… Just think about it.
1306
00:47:28,847 --> 00:47:30,457
What if there is a structure that
1307
00:47:30,457 --> 00:47:32,867
tells you exactly what needs to be
1308
00:47:32,877 --> 00:47:35,867
implemented and help you to implement it?
1309
00:47:36,147 --> 00:47:37,727
Does it make it more difficult for
1310
00:47:37,747 --> 00:47:39,627
you, or make the life much easier?
1311
00:47:39,677 --> 00:47:40,567
Because at the end of the
1312
00:47:40,567 --> 00:47:43,297
day, you're not accountable.
1313
00:47:43,347 --> 00:47:45,847
Someone else did the risk analysis
1314
00:47:45,857 --> 00:47:47,277
and told you you're just accountable
1315
00:47:47,277 --> 00:47:48,357
to implement that guardrail.
1316
00:47:48,817 --> 00:47:51,167
So the goal of all of this is to make
1317
00:47:51,197 --> 00:47:53,697
the life much easier for the builders,
1318
00:47:53,717 --> 00:47:55,757
not necessarily complicate things.
1319
00:47:55,937 --> 00:47:57,037
And that's why, like, we are
1320
00:47:57,037 --> 00:47:59,757
packaging, uh, the capabilities that
1321
00:47:59,757 --> 00:48:02,147
we are designing as MCP servers, as
1322
00:48:02,157 --> 00:48:04,117
skills that can be surfaced versus
1323
00:48:04,117 --> 00:48:06,007
a product that you need to go to the
1324
00:48:06,257 --> 00:48:08,547
interface and, uh, report things.
1325
00:48:09,411 --> 00:48:10,301
Yeah, maybe a shameless
1326
00:48:10,311 --> 00:48:11,071
plug here, Gurpreet.
1327
00:48:11,071 --> 00:48:12,701
So there's a policy engine that we
1328
00:48:12,701 --> 00:48:13,781
are building at Fiddler where you
1329
00:48:13,781 --> 00:48:15,301
can encode all of this at an admin
1330
00:48:15,311 --> 00:48:17,191
level so that you don't have to deal
1331
00:48:17,191 --> 00:48:18,791
with all the multiple steering files.
1332
00:48:18,821 --> 00:48:20,191
But, you know, check us
1333
00:48:20,191 --> 00:48:21,411
out and we can chat later.
1334
00:48:21,701 --> 00:48:22,811
So let's actually go into a
1335
00:48:22,811 --> 00:48:24,361
rapid fire section, Maryam,
1336
00:48:24,391 --> 00:48:25,651
I think as we round this out.
1337
00:48:25,671 --> 00:48:28,461
Uh, so, you know, short answers,
1338
00:48:28,491 --> 00:48:30,721
you know, sharp takes, uh, uh,
1339
00:48:30,751 --> 00:48:32,421
traditional ML or generative AI?
1340
00:48:33,731 --> 00:48:34,361
Both
1341
00:48:35,203 --> 00:48:35,233
Okay.
1342
00:48:36,733 --> 00:48:37,353
spoken like a
1343
00:48:37,353 --> 00:48:38,163
true enterprise
1344
00:48:38,225 --> 00:48:38,535
Yeah.
1345
00:48:38,725 --> 00:48:39,045
well,
1346
00:48:39,163 --> 00:48:39,463
veteran.
1347
00:48:39,535 --> 00:48:40,145
is the thing.
1348
00:48:40,285 --> 00:48:42,515
I used to work for Lyft, and my
1349
00:48:42,515 --> 00:48:44,485
job was to put the right bike at
1350
00:48:44,485 --> 00:48:46,115
the right place at the right time.
1351
00:48:46,125 --> 00:48:48,755
This is a classic predictive
1352
00:48:48,867 --> 00:48:49,167
Yeah.
1353
00:48:49,545 --> 00:48:51,265
challenge that requires
1354
00:48:51,275 --> 00:48:52,515
traditional ML. Come on.
1355
00:48:52,545 --> 00:48:53,985
Let's, Let's, not mix them up
1356
00:48:54,237 --> 00:48:54,867
absolutely.
1357
00:48:54,907 --> 00:48:55,257
Awesome.
1358
00:48:55,307 --> 00:48:56,237
Yeah, great to see that.
1359
00:48:56,587 --> 00:48:58,427
Copilots or autonomous agents?
1360
00:48:58,447 --> 00:48:59,357
I think we kind of spoke
1361
00:48:59,357 --> 00:49:00,047
about that, but yeah.
1362
00:49:00,077 --> 00:49:01,407
You wanna give your hot take
1363
00:49:02,593 --> 00:49:03,653
Oh, Copilot, is it the
1364
00:49:03,653 --> 00:49:05,373
Microsoft Copilot or Copilot?
1365
00:49:05,821 --> 00:49:07,401
meaning human in the loop copilot,
1366
00:49:07,571 --> 00:49:09,091
sort of copilot where a human is
1367
00:49:09,091 --> 00:49:09,571
looking at it,
1368
00:49:09,637 --> 00:49:10,557
okay, so
1369
00:49:10,851 --> 00:49:10,871
Yeah
1370
00:49:11,137 --> 00:49:12,617
human in the loop versus
1371
00:49:12,627 --> 00:49:14,457
total 100% automation.
1372
00:49:14,547 --> 00:49:16,257
I would say it depends on the use case
1373
00:49:16,741 --> 00:49:17,061
Got it.
1374
00:49:18,021 --> 00:49:20,171
powerful agent or many specialized agents?
1375
00:49:24,525 --> 00:49:26,355
I would say depends on the use case.
1376
00:49:26,805 --> 00:49:29,025
Sometimes the cost of orchestration
1377
00:49:29,365 --> 00:49:30,945
across the agents can be
1378
00:49:30,945 --> 00:49:32,915
higher than one Uber agent
1379
00:49:33,505 --> 00:49:34,885
Okay, here is a hard question.
1380
00:49:34,925 --> 00:49:36,845
Frontier models or small
1381
00:49:36,855 --> 00:49:37,655
specialized models?
1382
00:49:38,465 --> 00:49:40,175
Well, frontier model is still
1383
00:49:40,175 --> 00:49:42,005
good when you want to burn cash.
1384
00:49:43,125 --> 00:49:44,555
But if your use case doesn't
1385
00:49:44,555 --> 00:49:47,225
require that accuracy, small models
1386
00:49:47,499 --> 00:49:47,759
Yeah.
1387
00:49:48,639 --> 00:49:50,099
Open source or proprietary models
1388
00:49:51,821 --> 00:49:54,261
I'm a fan of open source, so I,
1389
00:49:54,301 --> 00:49:57,321
I, on this one I would go open.
1390
00:49:57,951 --> 00:50:00,121
actually, I'm, I'm a fan of open source
1391
00:50:00,121 --> 00:50:02,001
to a point that, like, you can look it up.
1392
00:50:02,011 --> 00:50:03,531
I even designed a board game
1393
00:50:03,561 --> 00:50:04,751
and I made it open source.
1394
00:50:04,781 --> 00:50:06,871
You can print your own to that
1395
00:50:06,871 --> 00:50:08,581
level, so not even code, yeah.
1396
00:50:08,679 --> 00:50:09,319
What is it called?
1397
00:50:09,811 --> 00:50:11,811
It's called Entanglion.
1398
00:50:11,811 --> 00:50:12,861
It is, uh, a, uh, an open
1399
00:50:12,861 --> 00:50:14,971
source game to teach people how
1400
00:50:14,971 --> 00:50:17,031
to program quantum computers
1401
00:50:17,795 --> 00:50:18,255
Super.
1402
00:50:18,325 --> 00:50:19,115
We'll check that out.
1403
00:50:19,315 --> 00:50:21,785
We'll put, put a link in the video later.
1404
00:50:22,195 --> 00:50:25,375
Um, centralized or federated AI governance
1405
00:50:29,891 --> 00:50:31,131
It depends on the industry
1406
00:50:31,683 --> 00:50:31,933
Yeah.
1407
00:50:32,753 --> 00:50:32,953
Yeah.
1408
00:50:33,443 --> 00:50:34,793
Observability or enforcement,
1409
00:50:34,803 --> 00:50:36,423
which is harder in your opinion?
1410
00:50:44,117 --> 00:50:46,867
Enforcement requires observability.
1411
00:50:47,219 --> 00:50:47,489
Yeah
1412
00:50:47,817 --> 00:50:49,497
If you don't have observability,
1413
00:50:49,887 --> 00:50:51,257
you don't know what you're enforcing
1414
00:50:51,909 --> 00:50:52,509
Makes sense.
1415
00:50:52,979 --> 00:50:54,479
Uh, bigger risk, what is a
1416
00:50:54,479 --> 00:50:56,099
bigger risk, a malicious agent
1417
00:50:56,149 --> 00:50:57,949
or an over-permissioned agent?
1418
00:51:01,859 --> 00:51:02,679
It depends on what the
1419
00:51:02,679 --> 00:51:03,889
agent is trying to achieve
1420
00:51:06,413 --> 00:51:07,893
What is more important, model
1421
00:51:07,893 --> 00:51:09,613
accuracy or system reliability?
1422
00:51:11,395 --> 00:51:12,575
System reliability
1423
00:51:13,135 --> 00:51:14,545
And harness as the work.
1424
00:51:15,125 --> 00:51:16,755
Uh, most overhyped agentic
1425
00:51:16,765 --> 00:51:17,755
use case that you have come
1426
00:51:17,755 --> 00:51:18,845
across in the last few months
1427
00:51:19,797 --> 00:51:20,837
most hyped
1428
00:51:28,579 --> 00:51:30,859
You know, there are lots of statement
1429
00:51:30,859 --> 00:51:33,439
that are being made that, like, agents
1430
00:51:33,439 --> 00:51:38,039
can do almost everything which, uh, in
1431
00:51:38,039 --> 00:51:40,569
an experimentation, sure, but in the
1432
00:51:40,579 --> 00:51:42,789
reality I would say it's applicable to
1433
00:51:42,789 --> 00:51:46,059
almost all of the agents, um, um, if
1434
00:51:46,059 --> 00:51:47,899
they are not designed for production.
1435
00:51:48,319 --> 00:51:51,589
Um, it's so easy to build a an agent in
1436
00:51:51,589 --> 00:51:54,299
five minutes that does rocket science,
1437
00:51:54,819 --> 00:51:56,599
but is it appli- is it practical?
1438
00:51:56,629 --> 00:51:57,369
Can I use it?
1439
00:51:58,319 --> 00:52:00,569
It's, it's nine- it's 90% more,
1440
00:52:00,659 --> 00:52:02,259
uh, difficult to operationalize it.
1441
00:52:02,329 --> 00:52:02,409
You know,
1442
00:52:02,595 --> 00:52:03,305
Exactly.
1443
00:52:03,749 --> 00:52:04,019
right?
1444
00:52:04,025 --> 00:52:04,635
Exactly
1445
00:52:04,949 --> 00:52:05,189
Yeah.
1446
00:52:05,579 --> 00:52:07,739
Most underrated enterprise AI use case
1447
00:52:13,555 --> 00:52:15,505
Well, the most popular one is still
1448
00:52:15,545 --> 00:52:17,565
content grounded question and answering,
1449
00:52:17,685 --> 00:52:19,115
which is a simple question and
1450
00:52:19,125 --> 00:52:21,655
answer, but still there are lots of
1451
00:52:21,665 --> 00:52:23,735
efficiencies that enterprises can gain
1452
00:52:24,707 --> 00:52:25,247
Oh, oh.
1453
00:52:26,087 --> 00:52:28,407
one metric every AI leader should track
1454
00:52:35,599 --> 00:52:37,859
That metric is, do I have all the
1455
00:52:37,859 --> 00:52:40,219
right metrics that I need to track?
1456
00:52:40,967 --> 00:52:41,497
That's great.
1457
00:52:41,987 --> 00:52:42,617
Very well said.
1458
00:52:43,087 --> 00:52:44,827
Uh, what should an AI agent
1459
00:52:44,837 --> 00:52:45,787
never be allowed to do
1460
00:52:45,787 --> 00:52:47,117
autonomously, in your opinion?
1461
00:52:50,047 --> 00:52:51,217
Never say never or
1462
00:52:52,333 --> 00:52:54,293
Well, there are things like, um,
1463
00:52:54,493 --> 00:52:56,533
the foundations like AI should
1464
00:52:56,533 --> 00:52:59,903
never harm human no matter what.
1465
00:53:00,361 --> 00:53:00,691
Mm-hmm.
1466
00:53:00,793 --> 00:53:01,833
So there are a series of
1467
00:53:01,833 --> 00:53:04,463
foundations that no matter what,
1468
00:53:04,513 --> 00:53:06,073
and it's not just applicable to
1469
00:53:06,073 --> 00:53:07,943
agents, it should not be broken
1470
00:53:08,835 --> 00:53:09,105
Yeah.
1471
00:53:09,725 --> 00:53:11,855
finally, what AI belief have
1472
00:53:11,855 --> 00:53:12,845
you changed your mind about?
1473
00:53:16,053 --> 00:53:18,733
What AI belief has changed my mind
1474
00:53:18,929 --> 00:53:20,219
like have you… Like maybe you had a
1475
00:53:20,219 --> 00:53:22,539
certain belief and now it has changed,
1476
00:53:22,559 --> 00:53:24,869
you know, given happening within AI
1477
00:53:26,477 --> 00:53:27,937
Well, I'm an AI fan.
1478
00:53:28,287 --> 00:53:28,627
Mm-hmm.
1479
00:53:28,737 --> 00:53:31,437
I spent the last 20 years working on
1480
00:53:31,447 --> 00:53:34,297
AI, so I'm pretty biased that AI is
1481
00:53:34,535 --> 00:53:34,855
dreams
1482
00:53:34,877 --> 00:53:35,477
cool.
1483
00:53:36,087 --> 00:53:38,007
But, but I would say that the thing
1484
00:53:38,017 --> 00:53:40,697
that I keep observing in the changing
1485
00:53:40,697 --> 00:53:43,427
people is at the beginning they were
1486
00:53:43,437 --> 00:53:46,597
blindly trusting AI, and now with
1487
00:53:46,607 --> 00:53:50,427
all the educations happening um, for
1488
00:53:50,727 --> 00:53:52,657
not just enterprise leaders, like
1489
00:53:52,777 --> 00:53:55,607
outside enterprise, the schools, uh,
1490
00:53:55,627 --> 00:53:57,807
kids interacting with that, like,
1491
00:53:57,817 --> 00:54:01,927
um, parents interacting with it, um,
1492
00:54:02,027 --> 00:54:04,017
people on the street interacting with
1493
00:54:04,017 --> 00:54:06,597
it, they are more aware of what can
1494
00:54:06,597 --> 00:54:09,247
go wrong and limitations, which is,
1495
00:54:09,247 --> 00:54:10,977
which is the right direction to go.
1496
00:54:10,997 --> 00:54:13,177
But historically, like three years
1497
00:54:13,177 --> 00:54:14,997
ago, they were like, "Oh, just
1498
00:54:15,007 --> 00:54:18,137
trust AI no matter what." Yeah
1499
00:54:19,107 --> 00:54:19,387
Awesome.
1500
00:54:20,177 --> 00:54:21,807
great, uh, way to end the
1501
00:54:21,807 --> 00:54:22,997
conversation here, Maryam.
1502
00:54:23,007 --> 00:54:25,117
Thank you so much for joining us and,
1503
00:54:25,117 --> 00:54:27,307
uh, uh, you know, and, you know, sort
1504
00:54:27,307 --> 00:54:28,887
of sharing your valuable thoughts.
1505
00:54:29,515 --> 00:54:30,915
Thanks for having me, Krishna
1506
00:54:31,477 --> 00:54:31,837
Awesome.
1507
00:54:32,117 --> 00:54:32,777
Thank you so much.
1508
00:54:33,027 --> 00:54:33,687
Thank you, folks.
1509
00:54:33,687 --> 00:54:35,137
Uh, we'll come back with, uh,
1510
00:54:35,177 --> 00:54:37,257
another great guest in a few weeks.
1511
00:54:37,267 --> 00:54:39,167
And until then, you know, have a nice day.