TRANSCRIPT 🔗
WEBVTT
00:00:00.000 --> 00:00:02.267
If you're architecting the agentic
00:00:02.267 --> 00:00:05.000
workflow poorly and you're not optimizing
00:00:05.000 --> 00:00:06.364
model use, how much data you pass
00:00:06.364 --> 00:00:07.727
in between agents and everything,
00:00:07.727 --> 00:00:09.187
the cents on a run could
00:00:09.187 --> 00:00:11.071
be dollars on a run, and that's
00:00:11.071 --> 00:00:12.985
exponential when you're doing hundreds
00:00:12.985 --> 00:00:15.000
of thousands of runs of these workflows.
00:00:15.000 --> 00:00:16.322
And then you reach a point
00:00:16.322 --> 00:00:18.000
where it's an "oh, shoot" moment.
00:00:18.000 --> 00:00:20.000
This is actually getting quite large,
00:00:20.000 --> 00:00:21.920
and we've got a finite pool of resources
00:00:21.920 --> 00:00:23.889
from an enterprise perspective, and we've
00:00:23.889 --> 00:00:25.995
got to really think thoughtfully about how
00:00:25.995 --> 00:00:28.000
we allocate those across the enterprise.
00:00:28.000 --> 00:00:30.383
Have there been surprises? Has your team
00:00:30.383 --> 00:00:32.588
come back to you and been like, "Wow,
00:00:32.588 --> 00:00:33.310
we discovered something
00:00:33.310 --> 00:00:34.000
we didn't anticipate"?
00:00:34.000 --> 00:00:36.176
I think the surprise is not a surprise,
00:00:36.176 --> 00:00:38.451
but I think it's the general consensus
00:00:38.451 --> 00:00:40.306
that nobody has it figured out.
00:00:53.000 --> 00:00:56.187
Yo, yo, yo, this is MK, and back with
00:00:56.187 --> 00:00:59.545
another episode of Ready, Set, Runtime.
00:00:59.545 --> 00:01:02.727
Today's topic: what is a token worth?
00:01:02.727 --> 00:01:05.242
And I'm joined by Susannah Warfield
00:01:05.242 --> 00:01:07.182
and Nick Colyer from AHEAD.
00:01:07.182 --> 00:01:09.000
Susannah, would you introduce yourself?
00:01:09.000 --> 00:01:10.000
Absolutely, MK.
00:01:10.000 --> 00:01:12.518
I'm Susannah Warfield. I lead the Value
00:01:12.518 --> 00:01:14.583
Architecture practice for AHEAD.
00:01:14.583 --> 00:01:16.682
So I'm backed by a team of consultants,
00:01:16.682 --> 00:01:18.889
and our responsibility is really to help,
00:01:18.889 --> 00:01:19.801
you know, our clients
00:01:19.801 --> 00:01:20.800
connect the investments
00:01:20.800 --> 00:01:22.267
they're making in IT to the value
00:01:22.267 --> 00:01:24.000
that it generates for the organization.
00:01:24.000 --> 00:01:25.923
And so this topic today is obviously front
00:01:25.923 --> 00:01:27.800
and center and near and dear to my heart,
00:01:27.800 --> 00:01:29.000
so thank you for having me.
00:01:29.000 --> 00:01:30.618
Great to be on, MK.
00:01:30.618 --> 00:01:33.684
Thank you for hosting us today here.
00:01:33.684 --> 00:01:36.429
Nick Colyer, VP of AI Solutions at AHEAD.
00:01:36.429 --> 00:01:38.295
Been at AHEAD for 12 years,
00:01:38.295 --> 00:01:40.231
building out private clouds,
00:01:40.231 --> 00:01:41.395
then it was the AWS practice,
00:01:41.395 --> 00:01:43.000
the Azure practice, the Google practice.
00:01:43.000 --> 00:01:44.588
Feels like we've been
00:01:44.588 --> 00:01:47.083
on this rollercoaster ride of new
00:01:47.083 --> 00:01:49.285
technology waves, built managed services,
00:01:49.285 --> 00:01:51.111
FinOps there, very relevant today.
00:01:51.111 --> 00:01:53.497
It's kinda getting rebuilt with the token
00:01:53.497 --> 00:01:55.476
discussion that we're having here.
00:01:55.476 --> 00:01:56.599
But yeah, my role now,
00:01:56.599 --> 00:01:57.619
I lead AI solutions.
00:01:57.619 --> 00:01:58.995
I spend a lot of time with large
00:01:58.995 --> 00:02:00.500
enterprise figuring out what works,
00:02:00.500 --> 00:02:01.500
what doesn't work.
00:02:01.500 --> 00:02:03.199
That's an ever-changing story,
00:02:03.199 --> 00:02:05.238
as I'm sure everybody knows as well.
00:02:05.238 --> 00:02:06.429
And partner with Susannah
00:02:06.429 --> 00:02:07.381
a lot on just, yeah,
00:02:07.381 --> 00:02:09.431
how do we get value out of all these AI
00:02:09.431 --> 00:02:11.429
solutions, AI SDLC, agentic platforms,
00:02:11.429 --> 00:02:13.586
building agents, you name it.
00:02:13.586 --> 00:02:15.000
So glad to be here.
00:02:15.000 --> 00:02:17.143
Yeah. I'm excited about this conversation.
00:02:17.143 --> 00:02:20.217
You know, I worked most of my career for
00:02:20.217 --> 00:02:23.214
banks, investment banks on Wall Street,
00:02:23.214 --> 00:02:24.607
and I like the new name
00:02:24.607 --> 00:02:26.000
of your team, Susannah.
00:02:26.000 --> 00:02:28.857
This idea of value creation,
00:02:28.857 --> 00:02:31.892
value management, that is the fundamental
00:02:31.892 --> 00:02:35.000
way we think of investment on Wall Street,
00:02:35.000 --> 00:02:36.997
is there needs to be a clear communication
00:02:36.997 --> 00:02:38.947
of what the value is that we're creating.
00:02:38.947 --> 00:02:41.000
And I think reminding people of that all
00:02:41.000 --> 00:02:42.146
the time is critical 'cause
00:02:42.146 --> 00:02:43.250
otherwise sometimes we get
00:02:43.250 --> 00:02:45.263
lost doing things that don't end up
00:02:45.263 --> 00:02:47.333
creating the value that we hope for.
00:02:47.333 --> 00:02:50.060
But to start out, Nick, when you think
00:02:50.060 --> 00:02:52.500
about the spend that you're seeing
00:02:52.500 --> 00:02:54.886
clients do or the work we're doing
00:02:54.886 --> 00:02:57.273
internally at AHEAD for ourselves,
00:02:57.273 --> 00:03:00.522
are you seeing results that you feel
00:03:00.522 --> 00:03:03.500
really good about, where you feel
00:03:03.500 --> 00:03:06.125
like the investment in an AI solution is
00:03:06.125 --> 00:03:08.750
resulting in a return that's meaningful,
00:03:08.750 --> 00:03:10.651
and are people starting to realize
00:03:10.651 --> 00:03:13.000
and recognize how to measure those things?
00:03:13.000 --> 00:03:14.526
Yeah, I think a couple
00:03:14.526 --> 00:03:16.538
of things are important here.
00:03:16.538
Load full transcript
--> 00:03:19.021
First of all, there's sheer productivity
00:03:19.021 --> 00:03:20.882
benefits from using the tools,
00:03:20.882 --> 00:03:22.693
and probably the hardest one to measure
00:03:22.693 --> 00:03:24.412
because everybody's using these tools
00:03:24.412 --> 00:03:25.620
in different ways.
00:03:25.620 --> 00:03:27.500
We've got literacy programs.
00:03:27.500 --> 00:03:29.250
The level of prompting people
00:03:29.250 --> 00:03:31.000
use differs across the board.
00:03:31.000 --> 00:03:32.761
You know, we can see some where people
00:03:32.761 --> 00:03:34.059
just chat in their thoughts,
00:03:34.059 --> 00:03:34.963
and others are people
00:03:34.963 --> 00:03:36.471
who have a well-architected prompt,
00:03:36.471 --> 00:03:38.800
and I think that's just gonna take
00:03:38.800 --> 00:03:40.238
time to work through.
00:03:40.238 --> 00:03:41.373
But I think the productivity
00:03:41.373 --> 00:03:42.143
benefits are there.
00:03:42.143 --> 00:03:43.747
If you tried to take these tools
00:03:43.747 --> 00:03:45.000
away from somebody today,
00:03:45.000 --> 00:03:46.900
people would be up in arms as to how much
00:03:46.900 --> 00:03:48.800
more work would be on their plate because
00:03:48.800 --> 00:03:50.278
people are doing a lot more.
00:03:50.278 --> 00:03:52.620
But in terms of the value, when I think
00:03:52.620 --> 00:03:54.722
of our clients and agentic-like use
00:03:54.722 --> 00:03:56.432
cases for the business versus
00:03:56.432 --> 00:03:58.438
the productivity of an individual,
00:03:58.438 --> 00:04:00.486
those are the ones that require a lot
00:04:00.486 --> 00:04:02.368
more scrutiny, a lot more thought,
00:04:02.368 --> 00:04:03.214
a lot of, "Hey,
00:04:03.214 --> 00:04:04.737
let's go to the whiteboard.
00:04:04.737 --> 00:04:05.500
Let's deconstruct this workflow,
00:04:05.500 --> 00:04:06.000
how you did it today.
00:04:06.000 --> 00:04:08.227
Let's not just automate it the way
00:04:08.227 --> 00:04:10.455
you did it with agents now instead
00:04:10.455 --> 00:04:11.883
of code to automate, but actually
00:04:11.883 --> 00:04:13.182
rethink the business process."
00:04:13.182 --> 00:04:15.625
And that's where we see a lot of benefits.
00:04:15.625 --> 00:04:17.098
Whether it's in healthcare,
00:04:17.098 --> 00:04:19.062
in finance, some of the other areas,
00:04:19.062 --> 00:04:20.315
we've seen a lot of use cases
00:04:20.315 --> 00:04:22.000
where we can measure the workflow time.
00:04:22.000 --> 00:04:25.033
Okay, this took, you know, 20 minutes with
00:04:25.033 --> 00:04:27.778
X handoffs or three days or six weeks,
00:04:27.778 --> 00:04:29.210
and now it takes, you know,
00:04:29.210 --> 00:04:30.217
minutes or seconds,
00:04:30.217 --> 00:04:32.219
and we can start to get that cost per run.
00:04:32.219 --> 00:04:33.696
So it differs across the board.
00:04:33.696 --> 00:04:35.320
I would say at the same time, there's
00:04:35.320 --> 00:04:36.944
this experimentation phase that we've
00:04:36.944 --> 00:04:38.997
all been through and a lot of our clients
00:04:38.997 --> 00:04:41.000
are still going through, is still there.
00:04:41.000 --> 00:04:42.918
It's been tapered back a little bit
00:04:42.918 --> 00:04:45.000
as people are asking the question now,
00:04:45.000 --> 00:04:46.301
which I think is where you're
00:04:46.301 --> 00:04:47.692
coming from there, MK, is like,
00:04:47.692 --> 00:04:49.819
what is the value of that? But I think
00:04:49.819 --> 00:04:51.667
the experimentation will continue
00:04:51.667 --> 00:04:53.916
as people learn and just need to get more
00:04:53.916 --> 00:04:56.000
fluent in these tools as time goes on.
00:04:56.000 --> 00:04:58.564
So all across the board, I think, is the
00:04:58.564 --> 00:05:01.000
answer to your question at the moment.
00:05:01.000 --> 00:05:02.429
I think experimentation is interesting.
00:05:02.429 --> 00:05:03.841
I've been thinking a lot
00:05:03.841 --> 00:05:05.312
about R&D and innovation,
00:05:05.312 --> 00:05:07.835
and that we need to assign and allow
00:05:07.835 --> 00:05:10.357
for budgets to be used against that,
00:05:10.357 --> 00:05:13.550
because it may be hard to measure the ROI,
00:05:13.550 --> 00:05:16.667
and I don't think it should be an endless
00:05:16.667 --> 00:05:18.526
budget, but I do think when you're
00:05:18.526 --> 00:05:20.769
in these critical technology transitions,
00:05:20.769 --> 00:05:23.440
that you allow your team some space
00:05:23.440 --> 00:05:26.417
to discover what the opportunities are.
00:05:26.417 --> 00:05:28.776
Susannah, I like when Nick sort
00:05:28.776 --> 00:05:31.364
of differentiated between personal
00:05:31.364 --> 00:05:33.983
productivity and enterprise productivity,
00:05:33.983 --> 00:05:36.667
and when I look back, let's say 18 months,
00:05:36.667 --> 00:05:38.027
I feel like the boards
00:05:38.027 --> 00:05:40.625
and the executive-level suite were saying,
00:05:40.625 --> 00:05:42.500
"We've got to roll out AI."
00:05:42.500 --> 00:05:44.896
And I think the consequence of that was
00:05:44.896 --> 00:05:47.353
that we rolled out lots of RAG solutions
00:05:47.353 --> 00:05:49.176
and in some cases, agent solutions,
00:05:49.176 --> 00:05:51.364
and in many cases accelerated development.
00:05:51.364 --> 00:05:53.109
What do you think people
00:05:53.109 --> 00:05:55.000
are asking for now, right?
00:05:55.000 --> 00:05:57.232
Like, now that we've sort
00:05:57.232 --> 00:06:00.000
of gotten our feet wet with AI,
00:06:00.000 --> 00:06:01.778
how do you view what the request
00:06:01.778 --> 00:06:03.333
from the executive level is,
00:06:03.333 --> 00:06:05.607
and how do you see it impacting, let's
00:06:05.607 --> 00:06:08.000
say, cost management and value creation?
00:06:08.000 --> 00:06:10.087
Yeah, you know, it's so interesting
00:06:10.087 --> 00:06:12.174
to me how this has kind of followed
00:06:12.174 --> 00:06:13.888
the hype cycle just like cloud did,
00:06:13.888 -->
00:06:15.357
where you have a lot of people
00:06:15.357 --> 00:06:17.199
experimenting and spinning up,
00:06:17.199 --> 00:06:19.286
you know, workloads and, you know,
00:06:19.286 --> 00:06:20.741
kind of being in control of their own
00:06:20.741 --> 00:06:22.000
destiny from a cost perspective.
00:06:22.000 --> 00:06:23.452
And then you reach a point
00:06:23.452 --> 00:06:25.294
where it's an "oh, shoot" moment.
00:06:25.294 --> 00:06:27.059
This is actually getting quite large,
00:06:27.059 --> 00:06:29.217
and we've got a finite pool of resources
00:06:29.217 --> 00:06:31.429
from an enterprise perspective, and we've
00:06:31.429 --> 00:06:33.710
got to really think thoughtfully about how
00:06:33.710 --> 00:06:35.882
we allocate those across the enterprise.
00:06:35.882 --> 00:06:38.164
And so we've kind of reached this tipping
00:06:38.164 --> 00:06:40.000
point where instead of, you know,
00:06:40.000 --> 00:06:41.029
we need to get ahead,
00:06:41.029 --> 00:06:42.500
we've got to be experimenting,
00:06:42.500 --> 00:06:44.486
we've got to keep up with our competition,
00:06:44.486 --> 00:06:46.000
it's, okay, how are we using it?
00:06:46.000 --> 00:06:48.250
What value are we actually getting
00:06:48.250 --> 00:06:50.500
for the cost that we're, you know,
00:06:50.500 --> 00:06:52.150
allocating towards AI? And I absolutely
00:06:52.150 --> 00:06:53.250
agree with what Nick said.
00:06:53.250 --> 00:06:55.059
We've kind of got two camps
00:06:55.059 --> 00:06:56.667
where we have, you know,
00:06:56.667 --> 00:06:58.333
defined business processes
00:06:58.333 --> 00:07:00.000
that are easy to attribute
00:07:00.000 --> 00:07:02.000
value to each and every step before
00:07:02.000 --> 00:07:04.000
and after the implementation of AI.
00:07:04.000 --> 00:07:05.448
And then you've got kind
00:07:05.448 --> 00:07:07.500
of the big and broad productivity.
00:07:07.500 --> 00:07:09.133
I think about, you know,
00:07:09.133 --> 00:07:11.786
giving AI to sales teams as an example.
00:07:11.786 --> 00:07:14.292
They can create proposals faster. They can
00:07:14.292 --> 00:07:16.500
create better proposals, potentially.
00:07:16.500 --> 00:07:18.205
But when you're measuring
00:07:18.205 --> 00:07:19.500
ultimately revenue,
00:07:19.500 --> 00:07:21.173
there are lots of other things
00:07:21.173 --> 00:07:23.125
that are influencing those numbers,
00:07:23.125 --> 00:07:24.180
you know, market conditions
00:07:24.180 --> 00:07:25.000
and things like that.
00:07:25.000 --> 00:07:27.790
So how do you actually siphon off the
00:07:27.790 --> 00:07:30.882
value that's directly attributable to AI?
00:07:30.882 --> 00:07:32.680
And I think that's what boards
00:07:32.680 --> 00:07:34.118
are starting to ask for,
00:07:34.118 --> 00:07:35.435
and that's really hard, right?
00:07:35.435 --> 00:07:37.367
There's a lot of sort of allocation and
00:07:37.367 --> 00:07:39.348
metadata and things that you have to use
00:07:39.348 --> 00:07:41.288
to get to that answer. And Nick
00:07:41.288 --> 00:07:43.667
and I have these conversations weekly.
00:07:43.667 --> 00:07:45.660
People are asking for it,
00:07:45.660 --> 00:07:48.529
and we've got to help, first of all,
00:07:48.529 --> 00:07:50.026
kind of create the visibility
00:07:50.026 --> 00:07:51.522
into the spend and then start
00:07:51.522 --> 00:07:53.303
to tick and tie it back to the value.
00:07:53.303 --> 00:07:55.278
So we're kind of in that inflection point
00:07:55.278 --> 00:07:57.729
right now where I think it's getting much
00:07:57.729 --> 00:08:00.000
more scrutiny than it has in the past.
00:08:00.000 --> 00:08:03.500
It's interesting that we all agree it's
00:08:03.500 --> 00:08:05.750
hard to measure the value
00:08:05.750 --> 00:08:08.000
of personal productivity.
00:08:08.000 --> 00:08:11.410
At the same time, we all agree that if
00:08:11.410 --> 00:08:15.000
we took it away, it would be a disaster.
00:08:15.000 --> 00:08:17.613
And I just think we have to think
00:08:17.613 --> 00:08:20.385
about that story for the enterprise
00:08:20.385 --> 00:08:23.187
because I agree with Nick, like, the focus
00:08:23.187 --> 00:08:25.588
needs to be on enterprise solutions.
00:08:25.588 --> 00:08:27.059
Like, I love personal productivity,
00:08:27.059 --> 00:08:29.209
but it's just not a big enough
00:08:29.209 --> 00:08:31.000
story at the board level.
00:08:31.000 --> 00:08:33.909
And all that being said, like we said,
00:08:33.909 --> 00:08:35.679
it's become such a valuable
00:08:35.679 --> 00:08:37.188
part of our day-to-day.
00:08:37.188 --> 00:08:39.278
Nick, I feel like you're at clients
00:08:39.278 --> 00:08:41.667
where you're starting to do project work
00:08:41.667 --> 00:08:44.546
where there is an expectation of, let's
00:08:44.546 --> 00:08:47.500
say, hard realization of a value of ROI.
00:08:47.500 --> 00:08:50.048
What are those projects like?
00:08:50.048 --> 00:08:52.333
How does the enterprise...
00:08:52.333 --> 00:08:54.000
Is it patient about it?
00:08:54.000 --> 00:08:56.000
Is it being generous about it?
00:08:56.000 --> 00:08:59.304
Like, where is everybody as it relates
00:08:59.304 --> 00:09:02.000
to the urgency of seeing value?
00:09:02.000 --> 00:09:03.534
Yeah, I think, well, actually,
00:09:03.534 --> 00:09:05.682
before I add that, I just want to dovetail
00:09:05.682 --> 00:09:06.534
something Susannah said,
00:09:06.534 --> 00:09:07.955
which I think connects to your question.
00:09:07.955 --> 00:09:09.759
Like, did we learn anything
00:09:09.759 --> 00:09:12.500
from when we went to cloud and, you know,
00:09:12.500 --> 00:09:15.119
I remember the tag and the visibility
00:09:15.119 --> 00:09:17.667
conversation on-prem to then the tag
00:09:17.667 --> 00:09:19.718
and visibility in the cloud,
00:09:19.718 --> 00:09:22.429
and now people are going nuts, right?
00:09:22.429 --> 00:09:24.040
Everybody's got stuff everywhere
00:09:24.040 --> 00:09:25.652
and same question's being asked.
00:09:25.652 --> 00:09:27.228
So I think, to your question,
00:09:27.228 --> 00:09:28.478
it's one thing to plan,
00:09:28.478 --> 00:09:29.987
and I'll talk more about the clients
00:09:29.987 --> 00:09:31.579
here and where they're seeing results,
00:09:31.579 --> 00:09:34.474
but if you aren't able to measure it,
00:09:34.474 --> 00:09:36.447
it's very hard to then go back and tell
00:09:36.447 --> 00:09:38.571
that story without having to, you know, go
00:09:38.571 --> 00:09:40.560
back and do a bunch of engineering rework
00:09:40.560 --> 00:09:42.500
and figure out where your spend's going,
00:09:42.500 --> 00:09:44.063
or are these tokens for business
00:09:44.063 --> 00:09:45.333
unit A or business unit B,
00:09:45.333 --> 00:09:46.333
and what project was it for?
00:09:46.333 --> 00:09:47.557
So I think that part
00:09:47.557 --> 00:09:49.636
that Susannah mentioned is really,
00:09:49.636 --> 00:09:51.925
really important from the value story.
00:09:51.925 --> 00:09:53.611
In terms of clients, though,
00:09:53.611 --> 00:09:55.216
I'll use one in healthcare,
00:09:55.216 --> 00:09:57.000
you know, one of the examples.
00:09:57.000 --> 00:09:58.917
I won't give too much information, but on
00:09:58.917 --> 00:10:00.833
your phone, you've got your medical card,
00:10:00.833 --> 00:10:02.554
for example, and there's a whole
00:10:02.554 --> 00:10:04.167
bunch of processes that happen
00:10:04.167 --> 00:10:06.164
behind the scenes with PBMs and other
00:10:06.164 --> 00:10:08.000
processes in the medical industry.
00:10:08.000 --> 00:10:10.162
And one of the clients we're working with,
00:10:10.162 --> 00:10:11.500
you know, we could measure
00:10:11.500 --> 00:10:14.093
this time of passing things through and
00:10:14.093 --> 00:10:16.818
human-in-the-loop checks prior to agents,
00:10:16.818 --> 00:10:18.773
and just an extensive amount
00:10:18.773 --> 00:10:20.938
of work that people were doing.
00:10:20.938 --> 00:10:22.799
And we could take that work,
00:10:22.799 --> 00:10:24.062
which is just very,
00:10:24.062 --> 00:10:25.667
very administrative and very measurable,
00:10:25.667 --> 00:10:28.112
and say, take it down to now we know
00:10:28.112 --> 00:10:30.762
this run takes 19 seconds or, you know,
00:10:30.762 --> 00:10:32.631
up to three minutes, and then there's
00:10:32.631 --> 00:10:34.500
still a human-in-the-loop checkpoint.
00:10:34.500 --> 00:10:36.157
But a whole bunch of other
00:10:36.157 --> 00:10:37.941
mundane stuff is being done.
00:10:37.941 --> 00:10:39.761
And so, again, hours become minutes
00:10:39.761 --> 00:10:41.944
or seconds, you know, and that translates.
00:10:41.944 --> 00:10:44.268
It's very easy to measure the impact
00:10:44.268 --> 00:10:46.333
of that and the hiring of people
00:10:46.333 --> 00:10:48.365
in that area. So those are good,
00:10:48.365 --> 00:10:50.333
you know, efficiency use cases,
00:10:50.333 --> 00:10:52.667
and I think they're the easier ones
00:10:52.667 --> 00:10:55.000
now for the businesses to go after.
00:10:55.000 --> 00:10:56.856
We've got another financial client,
00:10:56.856 --> 00:10:57.333
you know,
00:10:57.333 --> 00:10:59.471
connecting with Fiserv and a bunch
00:10:59.471 --> 00:11:01.923
of other components, that are reworking
00:11:01.923 --> 00:11:04.282
that entire business workflow as well,
00:11:04.282 --> 00:11:06.579
and again, putting agents in the mix.
00:11:06.579 --> 00:11:08.046
The interesting thing as we design
00:11:08.046 --> 00:11:09.211
and architect these things,
00:11:09.211 --> 00:11:10.922
it's not like one agent goes
00:11:10.922 --> 00:11:13.000
in and replaces the whole process.
00:11:13.000 --> 00:11:14.258
You have to start to think
00:11:14.258 --> 00:11:15.952
of agents that take this component,
00:11:15.952 --> 00:11:17.597
and this one's an evaluator-critic
00:11:17.597 --> 00:11:19.048
on the step the other one did,
00:11:19.048 --> 00:11:20.592
and this one's going to transact
00:11:20.592 --> 00:11:21.750
and execute in a system.
00:11:21.750 --> 00:11:23.500
This one might be doing research, right?
00:11:23.500 --> 00:11:24.950
There's all sorts of patterns
00:11:24.950 --> 00:11:26.000
that we see evolving,
00:11:26.000 --> 00:11:28.054
and I think as the businesses get
00:11:28.054 --> 00:11:30.294
better and better at those patterns,
00:11:30.294 --> 00:11:33.224
they're able to now look at a use case
00:11:33.224 --> 00:11:36.308
and roll that out much more effectively.
00:11:36.308 --> 00:11:37.245
So I think those
00:11:37.245 --> 00:11:38.769
efficiency ones are there.
00:11:38.769 --> 00:11:40.670
In terms of the revenue-generating ones,
00:11:40.670 --> 00:11:42.619
I mean, there's all sorts of things from,
00:11:42.619 --> 00:11:44.048
"Hey, I've got customer service bots."
00:11:44.048 --> 00:11:45.655
I don't think you can visit
00:11:45.655 --> 00:11:47.500
a retail website today without,
00:11:47.500 --> 00:11:49.904
you know, some name for some cool AI bot
00:11:49.904 --> 00:11:52.429
in the bottom-right corner of your screen.
00:11:52.429 --> 00:11:55.414
And so it'll be interesting to see
00:11:55.414 --> 00:11:58.750
the results of those as things evolve.
00:11:58.750 --> 00:12:00.856
You know, is that really replacing
00:12:00.856 --> 00:12:03.333
human time, or is that really, you know,
00:12:03.333 --> 00:12:05.230
advancing sales now that wouldn't
00:12:05.230 --> 00:12:06.667
have happened previously?
00:12:06.667 --> 00:12:09.224
So those ones are pretty interesting
00:12:09.224 --> 00:12:11.000
as we go through as well.
00:12:11.000 --> 00:12:14.220
Nick, when I think about the new SDLC,
00:12:14.220 --> 00:12:16.000
the AI-DLC, the XDLC,
00:12:16.000 --> 00:12:17.333
whatever flavor you like,
00:12:17.333 --> 00:12:19.952
one of the things we learned in Agile
00:12:19.952 --> 00:12:22.500
is keeping the stories small, right?
00:12:22.500 --> 00:12:24.000
Let's say just in custom
00:12:24.000 --> 00:12:25.312
software development.
00:12:25.312 --> 00:12:27.783
And when you were talking about agent
00:12:27.783 --> 00:12:30.455
tasks, it reminded me of sort of keeping
00:12:30.455 --> 00:12:33.234
them small, and then having agents
00:12:33.234 --> 00:12:36.667
that are very task-focused, very specific.
00:12:36.667 --> 00:12:38.908
Is that similar to how we used
00:12:38.908 --> 00:12:41.000
to think in the Agile world?
00:12:41.000 --> 00:12:43.798
Is it intentional to keep an agent's
00:12:43.798 --> 00:12:46.364
span of control or purview small,
00:12:46.364 --> 00:12:48.079
or is that just really a reflection
00:12:48.079 --> 00:12:48.864
of where we are,
00:12:48.864 --> 00:12:50.333
and we'll see that evolve over time?
00:12:50.333 --> 00:12:51.768
I think it's important
00:12:51.768 --> 00:12:53.333
for a couple of reasons,
00:12:53.333 --> 00:12:56.071
both a security and a cost reason,
00:12:56.071 --> 00:12:58.562
in itself, because by keeping it small
00:12:58.562 --> 00:13:01.053
in terms of the purpose of that agent,
00:13:01.053 --> 00:13:02.420
you're able to adjust the models
00:13:02.420 --> 00:13:04.000
as new models come out for that task.
00:13:04.000 --> 00:13:06.418
Like, text extraction needs
00:13:06.418 --> 00:13:10.000
a very specialized language model versus
00:13:10.000 --> 00:13:11.891
something that's doing orchestration
00:13:11.891 --> 00:13:13.571
or research or planning upfront,
00:13:13.571 --> 00:13:14.548
maybe needs a large
00:13:14.548 --> 00:13:15.833
language model, you know,
00:13:15.833 --> 00:13:17.657
like Opus or one of the other
00:13:17.657 --> 00:13:19.167
ones that are out there.
00:13:19.167 --> 00:13:21.726
So I think that's important in itself
00:13:21.726 --> 00:13:24.286
because it allows you to make changes
00:13:24.286 --> 00:13:27.300
more effectively throughout the workflow.
00:13:27.300 --> 00:13:29.697
I do think there's a case of, "Hey,
00:13:29.697 --> 00:13:32.368
we have a model at the top that spawns,
00:13:32.368 --> 00:13:34.474
you know, an agent at the front,
00:13:34.474 --> 00:13:36.185
that spawns agents with different
00:13:36.185 --> 00:13:38.000
models and has different criteria."
00:13:38.000 --> 00:13:39.333
There's this ongoing debate
00:13:39.333 --> 00:13:41.121
in the industry about level one through
00:13:41.121 --> 00:13:43.000
level four that people may have heard of.
00:13:43.000 --> 00:13:44.906
Now, if you go to Anthropic's
00:13:44.906 --> 00:13:46.944
website and some of the others,
00:13:46.944 --> 00:13:49.017
they talk extensively about agent loops
00:13:49.017 --> 00:13:51.250
and agents being able to spawn sub-agents.
00:13:51.250 --> 00:13:52.883
Level three is where I would
00:13:52.883 --> 00:13:54.750
say a lot of our clients are at,
00:13:54.750 --> 00:13:56.471
and some are gonna stay there.
00:13:56.471 --> 00:13:57.594
They're saying, "Hey,
00:13:57.594 --> 00:13:59.412
for the safety of this particular,
00:13:59.412 --> 00:14:01.471
you know, agent and its use case
00:14:01.471 --> 00:14:03.529
or multiple agents and use case,
00:14:03.529 --> 00:14:05.427
I wanna put a deterministic wrapper
00:14:05.427 --> 00:14:07.000
around all these sub-agents."
00:14:07.000 --> 00:14:10.278
They go, "I want some more control there.
00:14:10.278 --> 00:14:11.163
I want to enforce
00:14:11.163 --> 00:14:12.778
human-in-the-loop checkpoints."
00:14:12.778 --> 00:14:14.323
And so there's this philosophy
00:14:14.323 --> 00:14:15.250
going on of, like,
00:14:15.250 --> 00:14:16.835
how much control do you give
00:14:16.835 --> 00:14:18.250
over to the master agent,
00:14:18.250 --> 00:14:20.695
the chief-of-staff agent for the agents,
00:14:20.695 --> 00:14:22.895
right? Depending how you look at it.
00:14:22.895 --> 00:14:24.014
And how much do you build
00:14:24.014 --> 00:14:25.222
more deterministic wrappers
00:14:25.222 --> 00:14:27.000
in and more code around that.
00:14:27.000 --> 00:14:29.057
So I think that'll continue. But I think
00:14:29.057 --> 00:14:30.909
back to your question, I think there
00:14:30.909 --> 00:14:33.475
is value in sub-components and being
00:14:33.475 --> 00:14:36.042
very modularized and self-contained,
00:14:36.042 --> 00:14:37.537
as long as it's easy to build. If you're
00:14:37.537 --> 00:14:38.958
over-engineering something, forget it.
00:14:38.958 --> 00:14:40.392
Now you got, you know,
00:14:40.392 --> 00:14:43.000
10,000 boxes you've gotta go and manage.
00:14:43.000 --> 00:14:44.595
But if you do it in a really smart
00:14:44.595 --> 00:14:46.190
way and a very easy-to-manage way,
00:14:46.190 --> 00:14:47.857
then I think it's really effective.
00:14:47.857 --> 00:14:49.904
And, you know, Susannah and I talk all
00:14:49.904 --> 00:14:52.059
the time on just, you know, whether it's
00:14:52.059 --> 00:14:54.213
LLM gateways and kill switches and,
00:14:54.213 --> 00:14:56.429
you know, models and cost per token,
00:14:56.429 --> 00:14:59.279
those things can all be massaged at a more
00:14:59.279 --> 00:15:02.062
granular level if you take that approach.
00:15:02.062 --> 00:15:04.625
Susannah, you know, we're starting to see,
00:15:04.625 --> 00:15:06.354
like Nick is describing, the pivot
00:15:06.354 --> 00:15:08.235
that you talked about earlier, right?
00:15:08.235 --> 00:15:09.534
This idea that we're using
00:15:09.534 --> 00:15:11.333
the tools for personal productivity,
00:15:11.333 --> 00:15:14.212
and now we're seeing Nick describe
00:15:14.212 --> 00:15:16.667
it being used where there can
00:15:16.667 --> 00:15:19.295
be measurable evaluation of what's
00:15:19.295 --> 00:15:22.000
happening, so speed and efficiency.
00:15:22.000 --> 00:15:24.373
I know that at AHEAD we look
00:15:24.373 --> 00:15:27.000
at our own internal use of the,
00:15:27.000 --> 00:15:29.967
let's say, RAG models, whether we're
00:15:29.967 --> 00:15:33.182
using Claude or Glean as an abstraction
00:15:33.182 --> 00:15:36.320
layer to other models, and we're talking
00:15:36.320 -->
00:15:39.615
to our teammates about how we're using it.
00:15:39.615 --> 00:15:41.821
Like, you know, did MK use
00:15:41.821 --> 00:15:45.385
more than was expected this month or less?
00:15:45.385 --> 00:15:48.862
What are you seeing in terms
00:15:48.862 --> 00:15:53.333
of the standardization of measuring,
00:15:53.333 --> 00:15:55.167
like, team use of just the
00:15:55.167 --> 00:15:57.000
straight-up RAG solutions?
00:15:57.000 --> 00:15:59.905
And do you think our philosophy will
00:15:59.905 --> 00:16:03.214
change based on our expectation of higher
00:16:03.214 --> 00:16:05.066
use and hopefully a reduction
00:16:05.066 --> 00:16:06.471
in cost of the models?
00:16:06.471 --> 00:16:08.361
Like, are you starting to feel
00:16:08.361 --> 00:16:10.000
the push and pull of that?
00:16:10.000 --> 00:16:11.047
We are, I think,
00:16:11.047 --> 00:16:12.812
and you mentioned this, MK,
00:16:12.812 --> 00:16:14.810
we've got kind of an internal
00:16:14.810 --> 00:16:16.600
project running right now.
00:16:16.600 --> 00:16:19.000
We've got a multitude of tools.
00:16:19.000 --> 00:16:20.667
We've got everybody using them.
00:16:20.667 --> 00:16:23.270
We've got use cases hitting multiple tools
00:16:23.270 --> 00:16:25.750
in some cases. So all of this hinges on,
00:16:25.750 --> 00:16:27.698
first and foremost, kind of centralizing
00:16:27.698 --> 00:16:29.500
all of that data so that we can start
00:16:29.500 --> 00:16:32.314
to get our arms wrapped around what are
00:16:32.314 --> 00:16:35.200
we really consuming by department or by,
00:16:35.200 --> 00:16:36.488
you know, use case family,
00:16:36.488 --> 00:16:37.429
whatever it may be.
00:16:37.429 --> 00:16:39.988
So I'm hearing not only internally
00:16:39.988 --> 00:16:43.000
but when I meet externally with clients,
00:16:43.000 --> 00:16:45.505
you know, lots of talk about different
00:16:45.505 --> 00:16:48.077
observability tools and how do we start
00:16:48.077 --> 00:16:49.559
to really measure
00:16:49.559 --> 00:16:52.000
and optimize those outcomes.
00:16:52.000 --> 00:16:54.372
Candidly, I haven't seen like one
00:16:54.372 --> 00:16:56.600
silver-bullet, lights-out tool.
00:16:56.600 --> 00:16:57.745
There are more coming
00:16:57.745 --> 00:16:59.000
out each and every day.
00:16:59.000 --> 00:17:01.696
We've got a handful running through
00:17:01.696 --> 00:17:04.545
our kind of scoring rubric currently,
00:17:04.545 --> 00:17:06.621
and suspect we'll have some good,
00:17:06.621 --> 00:17:07.188
you know,
00:17:07.188 --> 00:17:10.110
lines of sight and inputs on that shortly.
00:17:10.110 --> 00:17:12.059
But yeah, I think, you know,
00:17:12.059 --> 00:17:14.005
being able to first and foremost see
00:17:14.005 --> 00:17:15.789
it and then understand, you know,
00:17:15.789 --> 00:17:17.655
which models are being selected and what
00:17:17.655 --> 00:17:19.474
each department's kind of using it for,
00:17:19.474 --> 00:17:20.474
that's step one.
00:17:20.474 --> 00:17:22.621
And really where a lot of the clients
00:17:22.621 --> 00:17:25.000
that we interact with anyway from a value
00:17:25.000 --> 00:17:25.610
architecture team,
00:17:25.610 --> 00:17:27.000
they're asking those very same questions.
00:17:27.000 --> 00:17:29.331
I love step one. I think step one
00:17:29.331 --> 00:17:31.944
informs it in a really important way.
00:17:31.944 --> 00:17:33.506
And when I meet with clients
00:17:33.506 --> 00:17:35.625
sometimes and I talk about governance,
00:17:35.625 --> 00:17:38.814
I feel like they're expecting me
00:17:38.814 --> 00:17:43.000
to introduce a new idea that I don't have.
00:17:43.000 --> 00:17:44.575
I have the same idea,
00:17:44.575 --> 00:17:46.750
which is how are we using it?
00:17:46.750 --> 00:17:49.000
Are we seeing value in how we're using it?
00:17:49.000 --> 00:17:50.164
Can we, like you said,
00:17:50.164 --> 00:17:52.333
measure the department use, the role use?
00:17:52.333 --> 00:17:54.588
You know, that's a lot of information,
00:17:54.588 --> 00:17:56.842
and it's such an easy sentence to say,
00:17:56.842 --> 00:17:58.421
you know, we should measure it.
00:17:58.421 --> 00:18:00.230
But I think there's so much value
00:18:00.230 --> 00:18:01.875
in understanding how our teams
00:18:01.875 --> 00:18:04.658
are using it now, especially if you see an
00:18:04.658 --> 00:18:07.308
operational team that seems to be using,
00:18:07.308 --> 00:18:08.718
let's say, more tokens
00:18:08.718 --> 00:18:10.000
than everybody else.
00:18:10.000 --> 00:18:11.907
That's a reason to investigate
00:18:11.907 --> 00:18:13.750
proactively to find solutions
00:18:13.750 --> 00:18:15.571
to help manage their cost, because
00:18:15.571 --> 00:18:17.500
they're getting something out of it,
00:18:17.500 --> 00:18:20.000
and we've got to find out what that is.
00:18:20.000 --> 00:18:22.571
Yeah. You could have a department that
00:18:22.571 --> 00:18:25.278
has 10x the consumption of another team,
00:18:25.278 --> 00:18:26.471
but they could be providing
00:18:26.471 --> 00:18:27.222
tremendous value,
00:18:27.222 --> 00:18:29.683
and if you're looking at just cost in a
00:18:29.683 --> 00:18:32.333
silo, you're gonna miss that full picture,
00:18:32.333 --> 00:18:33.617
which is really what we're
00:18:33.617 --> 00:18:35.000
trying to solve here, right?
00:18:35.000 --> 00:18:37.028
Like, what is the correct amount
00:18:37.028 --> 00:18:38.929
of consumption to get whatever
00:18:38.929 --> 00:18:41.824
outcome we're trying to deliver? But we've
00:18:41.824 --> 00:18:44.375
got to figure out a really consistent
00:18:44.375 --> 00:18:46.000
and meaningful way to connect those dots.
00:18:46.000 --> 00:18:48.222
The reverse is also true.
00:18:48.222 --> 00:18:50.749
I hear people talk about the things
00:18:50.749 --> 00:18:53.636
they're doing for personal productivity,
00:18:53.636 --> 00:18:56.449
and frequently they use more complex
00:18:56.449 --> 00:18:58.636
language to describe a
very,
00:18:58.636 --> 00:19:00.000
very simple outcome.
00:19:00.000 --> 00:19:01.752
And in some of those cases,
00:19:01.752 --> 00:19:04.412
their token use, I think, is way too high
00:19:04.412 --> 00:19:06.323
for the value that they're creating
00:19:06.323 --> 00:19:08.125
because it's sort of unnecessary.
00:19:08.125 --> 00:19:10.357
It's like, oh, what is the expression?
00:19:10.357 --> 00:19:12.561
I don't know. I mean, you know,
00:19:12.561 --> 00:19:15.263
everything doesn't need a hammer, and,
00:19:15.263 --> 00:19:16.336
you know, sometimes people
00:19:16.336 --> 00:19:17.368
are just hammering stuff,
00:19:17.368 --> 00:19:18.532
and you're like, "Okay,
00:19:18.532 --> 00:19:20.000
there's another way at this."
00:19:20.000 --> 00:19:23.319
So this for me pivots to the next
00:19:23.319 --> 00:19:25.833
part of the conversation,
00:19:25.833 --> 00:19:30.385
which is we approve project work.
00:19:30.385 --> 00:19:32.634
When I propose, let's say, a custom
00:19:32.634 --> 00:19:35.333
technology solution at an investment bank,
00:19:35.333 --> 00:19:37.897
there's the initial cost to, let's say,
00:19:37.897 --> 00:19:40.000
roll out the MVP and, let's say,
00:19:40.000 --> 00:19:42.440
a phase one cost, and maybe I'm even
00:19:42.440 --> 00:19:45.286
costing out what it is to manage features,
00:19:45.286 --> 00:19:47.439
bug fix, maintenance over the course of,
00:19:47.439 --> 00:19:49.000
let's say, a three-year plan.
00:19:49.000 --> 00:19:52.246
But those trailing costs are modest
00:19:52.246 --> 00:19:55.769
compared to the initial down payments,
00:19:55.769 --> 00:19:56.843
let's say. Now,
00:19:56.843 --> 00:19:58.846
if I'm rolling out something
00:19:58.846 --> 00:20:02.089
that has token cost associated with it,
00:20:02.089 --> 00:20:05.000
I think the TCO is more complicated
00:20:05.000 --> 00:20:06.636
because I've gotta understand
00:20:06.636 --> 00:20:08.667
what the average token cost will be,
00:20:08.667 --> 00:20:11.962
and then I have to consider if the AI
00:20:11.962 --> 00:20:14.545
drifts or whatever the issues
00:20:14.545 --> 00:20:18.403
or challenges start to occur as it relates
00:20:18.403 --> 00:20:21.250
to the AI inside our solutions.
00:20:21.250 --> 00:20:23.704
You know, we have to understand
00:20:23.704 --> 00:20:26.000
what those ongoing costs are.
00:20:26.000 --> 00:20:28.209
And I feel like it's early
00:20:28.209 --> 00:20:30.588
to be able to estimate that.
00:20:30.588 --> 00:20:31.912
But Nick, you're probably
00:20:31.912 --> 00:20:33.235
seeing that on the daily.
00:20:33.235 --> 00:20:34.505
And so what are you seeing
00:20:34.505 --> 00:20:35.385
people talk about?
00:20:35.385 --> 00:20:37.399
I think the biggest thing here
00:20:37.399 --> 00:20:39.615
is the difference between clients
00:20:39.615 --> 00:20:42.839
and teams doing this well and teams
00:20:42.839 --> 00:20:46.154
that are doing it poorly is growing.
00:20:46.154 --> 00:20:48.034
And the teams that
are doing
00:20:48.034 --> 00:20:50.250
it exceptionally well and looking
00:20:50.250 --> 00:20:51.865
at token costs, they're looking
00:20:51.865 --> 00:20:54.000
at a business workflow at cents on a run.
00:20:54.000 --> 00:20:56.114
So when you start to translate it from,
00:20:56.114 --> 00:20:57.632
okay, here's the token cost,
00:20:57.632 --> 00:20:59.205
but here's what it actually costs
00:20:59.205 --> 00:21:00.588
to create that business value
00:21:00.588 --> 00:21:02.327
that Susannah talks about and can
00:21:02.327 --> 00:21:04.118
work with the business to measure,
00:21:04.118 --> 00:21:05.500
becomes a different conversation.
00:21:05.500 --> 00:21:07.807
If you're architecting the agentic
00:21:07.807 --> 00:21:10.588
workflow poorly and you're not optimizing
00:21:10.588 --> 00:21:12.684
model use and, you know, how much data
00:21:12.684 --> 00:21:15.000
you pass in between agents and everything,
00:21:15.000 --> 00:21:16.091
the cents on a run could
00:21:16.091 --> 00:21:17.000
be dollars on a run.
00:21:17.000 --> 00:21:19.333
And that's exponential when you're doing,
00:21:19.333 --> 00:21:21.599
you know, hundreds of thousands
00:21:21.599 --> 00:21:23.571
of runs of these workflows.
00:21:23.571 --> 00:21:25.254
So that's the sort of gravity,
00:21:25.254 --> 00:21:27.105
I think, that we're dealing with.
00:21:27.105 --> 00:21:28.684
And I'll use another example.
00:21:28.684 --> 00:21:30.278
Even just one of our developers
00:21:30.278 --> 00:21:31.923
that builds a significant amount
00:21:31.923 --> 00:21:34.397
of our agentic platform had a good
00:21:34.397 --> 00:21:36.579
example of, hey, we were using
00:21:36.579 --> 00:21:38.266
Sonnet for the code development
00:21:38.266 --> 00:21:40.333
for a particular use case for a while.
00:21:40.333 --> 00:21:42.867
Now we've gone to one of the Grok
00:21:42.867 --> 00:21:45.938
models at one quarter of the token cost.
00:21:45.938 --> 00:21:48.131
Same run, but now it's one quarter.
00:21:48.131 --> 00:21:50.325
So instead of, I think it's 18,000,
00:21:50.325 --> 00:21:51.907
20,000 a month they were spending
00:21:51.907 --> 00:21:53.250
on some of this development,
00:21:53.250 --> 00:21:54.400
again, the value's there.
00:21:54.400 --> 00:21:56.333
We know where we're going in this project,
00:21:56.333 --> 00:21:58.677
but that's now only $5,000 a month,
00:21:58.677 --> 00:22:00.217
you know, give or take.
00:22:00.217 --> 00:22:02.345
So that is, I think, a big part of it.
00:22:02.345 --> 00:22:03.913
And so people may be saying,
00:22:03.913 --> 00:22:05.931
"Okay, well, do I wanna invest in FinOps
00:22:05.931 --> 00:22:08.000
capabilities today and optimize them up?"
00:22:08.000 --> 00:22:09.765
Absolutely. Because if you get
00:22:09.765 --> 00:22:12.000
these right, that's just one employee,
00:22:12.000 --> 00:22:14.178
and you start to advance this again
00:22:14.178 --> 00:22:16.667
from a productivity, from a development,
00:22:16.667 --> 00:22:19.128
or from an agentic run cycle, that's
00:22:19.128 --> 00:22:22.000
gonna allow you to optimize over and over.
00:22:22.000 --> 00:22:23.250
And these things are only
00:22:23.250 --> 00:22:24.500
gonna get better as well.
00:22:24.500 --> 00:22:26.082
The model capabilities and the cost
00:22:26.082 --> 00:22:27.619
per run for what we're doing today
00:22:27.619 --> 00:22:29.517
is already less than what we would spend,
00:22:29.517 --> 00:22:30.952
you know, last year doing this.
00:22:30.952 --> 00:22:32.619
Now there's questions about data center
00:22:32.619 --> 00:22:34.286
capacity, and do token costs go back up
00:22:34.286 --> 00:22:36.236
and everything. But we're still
00:22:36.236 --> 00:22:38.438
in a very early stage of the cycle,
00:22:38.438 --> 00:22:40.401
and we can expect the costs
00:22:40.401 --> 00:22:42.000
to start to come down.
00:22:42.000 --> 00:22:45.000
But your ability to actually affect
00:22:45.000 --> 00:22:47.281
those by making changes to the workflow
00:22:47.281 --> 00:22:49.444
and the models and your suppliers of,
00:22:49.444 --> 00:22:51.494
you know, consumption here,
00:22:51.494 --> 00:22:54.000
that's gonna be a big part of it.
00:22:54.000 --> 00:22:56.375
There's an impression that the current
00:22:56.375 --> 00:22:58.750
cost structure of tokens is subsidized
00:22:58.750 --> 00:23:00.909
by the frontier labs, right?
00:23:00.909 --> 00:23:05.580
That we're paying a lower cost than maybe
00:23:05.580 --> 00:23:10.250
the actual cost to, let's say, Anthropic.
00:23:10.250 --> 00:23:12.000
What do you hear about that, Nick?
00:23:12.000 --> 00:23:13.395
Are people concerned
00:23:13.395 --> 00:23:15.000
about their token cost?
00:23:15.000 --> 00:23:16.562
Yeah. At least I've input
00:23:16.562 --> 00:23:18.438
another thought there as well,
00:23:18.438 --> 00:23:20.457
as people are also concerned they're
00:23:20.457 --> 00:23:22.812
not gonna have availability right now too.
00:23:22.812 --> 00:23:23.889
Like, there's already,
00:23:23.889 --> 00:23:25.357
you know, in the hyperscalers,
00:23:25.357 --> 00:23:27.933
certain regions where you can't run your
00:23:27.933 --> 00:23:30.444
models because they're out of capacity,
00:23:30.444 --> 00:23:32.375
or even just compute in certain regions.
00:23:32.375 --> 00:23:34.016
So there is... You know,
00:23:34.016 --> 00:23:36.750
I feel like it's less of a concern there
00:23:36.750 --> 00:23:38.033
and more about the capacity.
00:23:38.033 --> 00:23:39.500
Now, that depends on the client.
00:23:39.500 --> 00:23:41.783
Some clients are already well into this,
00:23:41.783 --> 00:23:43.438
and they're worried about it.
00:23:43.438 --> 00:23:45.015
Others are just getting started
00:23:45.015 --> 00:23:47.000
and building four-node clusters on-prem
00:23:47.000 --> 00:23:48.731
just to get experience and make sure
00:23:48.731 --> 00:23:50.750
they know they've got some capacity there,
00:23:50.750 --> 00:23:51.985
and they can
obviously scale
00:23:51.985 --> 00:23:53.000
up in the hyperscalers.
00:23:53.000 --> 00:23:56.359
So are they subsidizing it? I'm sure
00:23:56.359 --> 00:24:00.278
in cases, and trying to get people hooked,
00:24:00.278 --> 00:24:02.037
you know, on their services. You know,
00:24:02.037 --> 00:24:03.611
everybody wants the platform play.
00:24:03.611 --> 00:24:05.480
I mean, Anthropic's got their IPO
00:24:05.480 --> 00:24:07.632
coming out. You know, the question is:
00:24:07.632 --> 00:24:09.783
how big is that moat that they've got?
00:24:09.783 --> 00:24:11.765
Well, they've got fantastic models,
00:24:11.765 --> 00:24:14.021
but is everybody gonna go there
00:24:14.021 --> 00:24:16.786
for the platform, or are they gonna go
00:24:16.786 --> 00:24:18.803
to a Microsoft Foundry, an AWS Bedrock,
00:24:18.803 --> 00:24:20.200
Gemini Enterprise platform,
00:24:20.200 --> 00:24:21.868
the Gleans of the world, right, depending
00:24:21.868 --> 00:24:23.333
on the scenario we're talking about?
00:24:23.333 --> 00:24:25.931
And that's gonna allow that abstraction
00:24:25.931 --> 00:24:28.462
of models and ability to switch models
00:24:28.462 --> 00:24:30.593
out more effectively. Cursor, GrokBot,
00:24:30.593 --> 00:24:32.500
for example, they allow, you know,
00:24:32.500 --> 00:24:33.594
multiple model selection
00:24:33.594 --> 00:24:34.688
from inside those tools.
00:24:34.688 --> 00:24:36.939
So I just think, at least
00:24:36.939 --> 00:24:40.000
when I advise an enterprise today,
00:24:40.000 --> 00:24:42.671
flexibility right now is better without
00:24:42.671 --> 00:24:45.000
over-engineering that flexibility.
00:24:45.000 --> 00:24:46.852
'Cause there's something to be said
00:24:46.852 --> 00:24:48.333
by just going to a platform,
00:24:48.333 --> 00:24:49.269
building your agents,
00:24:49.269 --> 00:24:51.053
running it there and using their models,
00:24:51.053 --> 00:24:52.209
and they give you all
00:24:52.209 --> 00:24:53.421
the help and guidance.
00:24:53.421 --> 00:24:55.334
But I think once people get educated
00:24:55.334 --> 00:24:57.353
and understand that it's not that hard
00:24:57.353 --> 00:24:59.557
to write something in LangGraph and put
00:24:59.557 --> 00:25:01.875
it on any platform and switch your models
00:25:01.875 --> 00:25:03.937
out, then the light bulb goes off,
00:25:03.937 --> 00:25:06.364
and people just become more comfortable,
00:25:06.364 --> 00:25:08.000
I think, is the big thing.
00:25:08.000 --> 00:25:10.925
Susannah, I've been thinking, as Nick
00:25:10.925 --> 00:25:14.167
was talking, about the spreadsheet models
00:25:14.167 --> 00:25:18.708
that your team creates and how good we've
00:25:18.708 --> 00:25:22.917
got at evaluating the cost, let's say,
00:25:22.917 --> 00:25:25.179
of cloud over the years. How much
00:25:25.179 --> 00:25:27.647
effort right now are you putting in,
00:25:27.647 --> 00:25:29.881
and I like to use the word, to innovate
00:25:29.881 --> 00:25:32.000
around these financial models, right?
00:25:32.000 --> 00:25:35.290
This is a new way. It's rare in some
00:25:35.290 --> 00:25:38.214
cases that you get to, you know,
00:25:38.214 --> 00:25:41.000
discover a new way of demonstrating value.
00:25:41.000 --> 00:25:42.000
Yeah.
00:25:42.000 --> 00:25:44.636
And I know that I've seen your teams
00:25:44.636 --> 00:25:47.273
talk about this in various meetings.
00:25:47.273 --> 00:25:49.443
But what is it like keeping up with
00:25:49.443 --> 00:25:51.800
this part of what we're talking about?
00:25:51.800 --> 00:25:53.769
Because it's starting to be the critical
00:25:53.769 --> 00:25:55.000
part of the conversation.
00:25:55.000 --> 00:25:56.481
What does it cost? Could it cost less?
00:25:56.481 --> 00:25:58.000
Do we have transparency and visibility?
00:25:58.000 --> 00:26:00.184
Yeah. We're out of spreadsheets,
00:26:00.184 --> 00:26:02.368
so that's the great news, right?
00:26:02.368 --> 00:26:04.806
So yeah, I mean, we've leveraged our own
00:26:04.806 --> 00:26:07.000
tools to kind of push the boundaries
00:26:07.000 --> 00:26:09.000
of user interface and all that good stuff.
00:26:09.000 --> 00:26:10.154
So we've got some really sleek
00:26:10.154 --> 00:26:11.000
dashboards to do this.
00:26:11.000 --> 00:26:12.748
But this is the fun part,
00:26:12.748 --> 00:26:14.077
right? So you know,
00:26:14.077 --> 00:26:16.926
the team is responsible for ultimately
00:26:16.926 --> 00:26:20.000
aggregating all of our clients' IT spend,
00:26:20.000 --> 00:26:20.952
and as I mentioned,
00:26:20.952 --> 00:26:22.105
connecting it to value.
00:26:22.105 --> 00:26:24.462
This is where the puck is going,
00:26:24.462 --> 00:26:27.333
right? So this is a new spend category.
00:26:27.333 --> 00:26:29.602
We're seeing our clients, you know,
00:26:29.602 --> 00:26:32.000
bring it back just the way cloud did,
00:26:32.000 --> 00:26:33.427
bringing it back into kind
00:26:33.427 --> 00:26:35.294
of a centralized FinOps department
00:26:35.294 --> 00:26:37.474
so that you can manage it and manage
00:26:37.474 --> 00:26:39.412
the ripple effect of that spend,
00:26:39.412 --> 00:26:41.386
be it security, network considerations,
00:26:41.386 --> 00:26:42.500
those sorts of things.
00:26:42.500 --> 00:26:44.917
Being able to tell this story
00:26:44.917 --> 00:26:47.667
and help clients articulate value
00:26:47.667 --> 00:26:49.925
and the attribution and all of that,
00:26:49.925 --> 00:26:51.304
at the end of the day,
00:26:51.304 --> 00:26:52.678
the team is really problem-solvers,
00:26:52.678 --> 00:26:54.130
right? Like, that's what we wanna do.
00:26:54.130 --> 00:26:55.923
We take robust datasets and morph
00:26:55.923 --> 00:26:56.864
them into the storylines
00:26:56.864 --> 00:26:58.000
that our clients are telling.
00:26:58.000 --> 00:27:01.044
And so being able to be where we are in
00:27:01.044 --> 00:27:04.167
this life cycle and kinda help architect
00:27:04.167 --> 00:27:06.316
that for the clients is the
fun part.
00:27:06.316 --> 00:27:08.541
So we're enjoying it and candidly hoping
00:27:08.541 --> 00:27:10.600
that we can get to the answer faster,
00:27:10.600 --> 00:27:13.000
for obvious reasons. But it's been great.
00:27:13.000 --> 00:27:16.175
So, A, how different is what you're doing
00:27:16.175 --> 00:27:19.118
from what you learned when you started
00:27:19.118 --> 00:27:20.118
doing cloud cost analysis?
00:27:20.118 --> 00:27:22.350
And B, are there lessons learned
00:27:22.350 --> 00:27:25.000
from that that you're starting to use,
00:27:25.000 --> 00:27:26.070
or is it really a whole
00:27:26.070 --> 00:27:27.000
new way of thinking?
00:27:27.000 --> 00:27:29.186
The fundamentals are the same, right?
00:27:29.186 --> 00:27:31.667
You've got the same kinda core components.
00:27:31.667 --> 00:27:35.000
You've got the same why behind, you know,
00:27:35.000 --> 00:27:36.618
consumption and the use cases
00:27:36.618 --> 00:27:38.235
that our clients are driving.
00:27:38.235 --> 00:27:39.992
I think the difference here
00:27:39.992 --> 00:27:42.333
is that instead of having, you know,
00:27:42.333 --> 00:27:44.292
a handful of variables and things
00:27:44.292 --> 00:27:46.250
to kinda articulate a story with,
00:27:46.250 --> 00:27:47.688
we've got things coming
00:27:47.688 --> 00:27:49.062
at us from a multitude
00:27:49.062 --> 00:27:51.040
of directions and trying to parse
00:27:51.040 --> 00:27:53.077
out what's meaningful, what's not,
00:27:53.077 --> 00:27:55.509
what do we, you know, weight more heavily
00:27:55.509 --> 00:27:58.000
than another particular piece of criteria.
00:27:58.000 --> 00:28:01.207
Like, that's been the interesting part.
00:28:01.207 --> 00:28:04.167
But from a modeling perspective and,
00:28:04.167 --> 00:28:06.458
you know, total cost of ownership
00:28:06.458 --> 00:28:07.917
to value realization,
00:28:07.917 --> 00:28:09.953
all the fundamentals are the same.
00:28:09.953 --> 00:28:11.750
This is just the latest story.
00:28:11.750 --> 00:28:13.500
So I feel like we're very equipped,
00:28:13.500 --> 00:28:16.285
and I think the market is generally very
00:28:16.285 --> 00:28:19.000
well equipped to kinda figure this out.
00:28:19.000 --> 00:28:20.355
We're just, you know,
00:28:20.355 --> 00:28:23.000
where we are in the life cycle today, so.
00:28:23.000 --> 00:28:26.387
Yeah. You know, as we continue to learn
00:28:26.387 --> 00:28:29.688
about this, have there been surprises?
00:28:29.688 --> 00:28:31.158
Has your team come back
00:28:31.158 --> 00:28:32.500
to you and been like,
00:28:32.500 --> 00:28:34.392
"Wow, we discovered something we didn't
00:28:34.392 --> 00:28:36.429
anticipate"? Or, you know, as the maturity
00:28:36.429 --> 00:28:38.629
around the things we've just discussed
00:28:38.629 --> 00:28:40.308
around cloud, on-prem FinOps,
00:28:40.308 --> 00:28:42.534
has that really informed it so that there
00:28:42.534 --> 00:28:44.000
are really fewer surprises?
00:28:44.000 --> 00:28:47.333
I think the surprise is not a surprise,
00:28:47.333 --> 00:28:49.903
but I think it's the general consensus
00:28:49.903 --> 00:28:52.000
that nobody has it figured out.
00:28:52.000 --> 00:28:54.039
I think just many of our clients
00:28:54.039 --> 00:28:56.333
have financial information in silos.
00:28:56.333 --> 00:28:57.667
You've got different pockets
00:28:57.667 --> 00:28:59.000
of the organization managing
00:28:59.000 --> 00:29:01.398
their own budgets and forecasts
00:29:01.398 --> 00:29:03.100
and value realization.
00:29:04.333 --> 00:29:07.552
I think bringing it in and centralizing
00:29:07.552 --> 00:29:09.615
it is where everybody is.
00:29:09.615 --> 00:29:12.787
The surprise there, I think, is, you know,
00:29:12.787 --> 00:29:15.882
there's a disconnect sometimes between...
00:29:15.882 --> 00:29:17.596
And this is an interesting
00:29:17.596 --> 00:29:19.706
point and one that we see often.
00:29:19.706 --> 00:29:22.336
When you think about legacy financial
00:29:22.336 --> 00:29:25.250
planning and analytics teams, FP&A teams,
00:29:25.250 --> 00:29:27.375
they're really good at the numbers, right?
00:29:27.375 --> 00:29:29.718
And they're able to kind of articulate
00:29:29.718 --> 00:29:32.000
and create financial walks and do all
00:29:32.000 --> 00:29:33.774
the things that are necessary to talk
00:29:33.774 --> 00:29:35.357
about the health of the business.
00:29:35.357 --> 00:29:38.317
IT finance is kind of a whole different
00:29:38.317 --> 00:29:41.429
beast because you need to have that skill
00:29:41.429 --> 00:29:44.440
set, and then you need to also speak IT.
00:29:44.440 --> 00:29:47.000
And so when you start interjecting
00:29:47.000 --> 00:29:49.679
AI and the speed at which all of those
00:29:49.679 --> 00:29:52.500
capabilities and technology is evolving,
00:29:52.500 --> 00:29:54.244
that learning curve is really,
00:29:54.244 --> 00:29:55.000
really steep.
00:29:55.000 --> 00:29:56.875
So I think the surprise
00:29:56.875 --> 00:29:58.750
is we've got the volume
00:29:58.750 --> 00:30:00.303
coming in and the asks
00:30:00.303 --> 00:30:01.786
coming into the team.
00:30:01.786 --> 00:30:04.173
We're having to manage that workload and
00:30:04.173 --> 00:30:06.500
be students of the game and really stay
00:30:06.500 --> 00:30:08.250
up on the latest and greatest technology
00:30:08.250 --> 00:30:10.000
so that we can articulate those stories.
00:30:10.000 --> 00:30:12.203
So I think that the acceleration
00:30:12.203 --> 00:30:14.062
and the speed has been kind
00:30:14.062 --> 00:30:16.000
of the biggest aha moment for me.
00:30:16.000 --> 00:30:17.867
I think we knew it was coming.
00:30:17.867 --> 00:30:19.394
I don't think we quite realized
00:30:19.394 --> 00:30:20.625
how fast it would evolve,
00:30:20.625 --> 00:30:22.083
and I think every day I'm like,
00:30:22.083 --> 00:30:23.542
"Oh, gosh, I've got to go study
00:30:23.542 --> 00:30:24.961
up on XYZ 'cause I don't fully
00:30:24.961
--> 00:30:26.429
understand that component yet."
00:30:26.429 --> 00:30:28.143
So I think our clients
00:30:28.143 --> 00:30:30.714
are struggling with that as well.
00:30:30.714 --> 00:30:33.176
And that's gonna be the challenge,
00:30:33.176 --> 00:30:36.000
is keeping those FP&A teams up to speed
00:30:36.000 --> 00:30:37.838
on what the data is and ultimately
00:30:37.838 --> 00:30:40.000
the stories that they're trying to tell.
00:30:40.000 --> 00:30:42.692
Nick, just to extend from that,
00:30:42.692 --> 00:30:44.951
you know, you've helped AHEAD manage
00:30:44.951 --> 00:30:47.083
through lots of innovation cycles.
00:30:47.083 --> 00:30:48.787
You described, you know,
00:30:48.787 --> 00:30:50.278
the different clouds,
00:30:50.278 --> 00:30:52.222
the life cycle of how people moved
00:30:52.222 --> 00:30:54.167
to the cloud, stayed in the cloud,
00:30:54.167 --> 00:30:56.551
moved back to on-prem or thought
00:30:56.551 --> 00:30:58.636
about multi-cloud solutions.
00:30:58.636 --> 00:31:01.017
What do you see as similarities
00:31:01.017 --> 00:31:04.167
in this new innovation cycle that relates
00:31:04.167 --> 00:31:06.243
back to your experience when people
00:31:06.243 --> 00:31:08.438
were starting to invest in the cloud,
00:31:08.438 --> 00:31:10.357
or are they just wildly different?
00:31:10.357 --> 00:31:12.490
Some similarities. I mean, I think
00:31:12.490 --> 00:31:15.000
Susannah mentioned the speed of this one
00:31:15.000 --> 00:31:17.727
is the biggest difference. I mean, while
00:31:17.727 --> 00:31:20.250
I thought cloud was fast at the time,
00:31:20.250 --> 00:31:22.399
there was a lot to learn at the time
00:31:22.399 --> 00:31:24.250
there. It was really, you know,
00:31:24.250 --> 00:31:25.716
the trio of hyperscalers that came out,
00:31:25.716 --> 00:31:27.182
and you look back and you're like, "Ah,
00:31:27.182 --> 00:31:28.245
that's just a grid of services,
00:31:28.245 --> 00:31:29.000
and it's pretty easy."
00:31:29.000 --> 00:31:30.648
And it's so commoditized now.
00:31:30.648 --> 00:31:32.750
We take it for granted and, you know,
00:31:32.750 --> 00:31:34.955
hopefully in a few years we'll be looking
00:31:34.955 --> 00:31:37.105
back at AI, and we know all our services
00:31:37.105 --> 00:31:39.018
and platforms and models and everything
00:31:39.018 --> 00:31:40.882
there. I think what's different, yeah,
00:31:40.882 --> 00:31:43.294
with how we've approached it at AHEAD,
00:31:43.294 --> 00:31:45.833
and I see this at other clients as well,
00:31:45.833 --> 00:31:48.317
is I think when people established
00:31:48.317 --> 00:31:50.800
their cloud CoEs, that's something
00:31:50.800 --> 00:31:52.615
that is being replicated well with AI,
00:31:52.615 --> 00:31:54.000
but even more deliberate now.
00:31:54.000 --> 00:31:56.903
Like at AHEAD, we have an AI hive,
00:31:56.903 --> 00:31:58.611
call it the beehive.
00:31:58.611 --> 00:32:01.303
Everything's bee-themed. I'm sure you can
00:32:01.303 --> 00:32:03.667
get it. But we were very deliberate:
00:32:03.667 --> 00:32:05.984
this can't just be another side
00:32:05.984 --> 00:32:08.077
job for somebody to take on.
00:32:08.077 --> 00:32:09.491
You know, I think cloud
00:32:09.491 --> 00:32:11.765
was the infrastructure team side job.
00:32:11.765 --> 00:32:13.406
It's kind of evolved into platform
00:32:13.406 --> 00:32:15.000
engineering and other components.
00:32:15.000 --> 00:32:17.345
But AI is just, there's so much to learn,
00:32:17.345 --> 00:32:18.889
there's so much to consume.
00:32:18.889 --> 00:32:19.965
I think over the years
00:32:19.965 --> 00:32:21.042
I've been studying it,
00:32:21.042 --> 00:32:21.792
to Susannah's point,
00:32:21.792 --> 00:32:22.917
saying we never stop studying.
00:32:22.917 --> 00:32:24.458
You know, I thought YouTube
00:32:24.458 --> 00:32:26.000
videos at the weekend were,
00:32:26.000 --> 00:32:27.000
you know, things during cloud,
00:32:27.000 --> 00:32:29.000
and it was a little bit, but now there's
00:32:29.000 --> 00:32:31.000
just so much to read, so much to digest.
00:32:31.000 --> 00:32:32.918
Using AI tools to give you the news
00:32:32.918 --> 00:32:35.000
in a digestible way, what's important,
00:32:35.000 --> 00:32:36.571
what's important for clients we're
00:32:36.571 --> 00:32:38.235
working on, what's relevant to them,
00:32:38.235 --> 00:32:39.977
what's going on in the industry
00:32:39.977 --> 00:32:42.000
verticals that's so specialized now,
00:32:42.000 --> 00:32:43.927
and then what is more, you know,
00:32:43.927 --> 00:32:45.312
horizontal across them.
00:32:45.312 --> 00:32:47.589
So I do think that's the other key
00:32:47.589 --> 00:32:50.000
component that's probably different,
00:32:50.000 --> 00:32:52.019
is the platform teams
00:32:52.019 --> 00:32:55.000
have to understand the business
00:32:55.000 --> 00:32:56.905
use cases and how to translate
00:32:56.905 --> 00:32:59.000
workflows into agentic workflows.
00:32:59.000 --> 00:33:01.136
It's not just the app development team.
00:33:01.136 --> 00:33:02.943
They need to do as well. They've got
00:33:02.943 --> 00:33:05.000
to learn to code and develop agents well.
00:33:05.000 --> 00:33:06.275
The common pattern I would
00:33:06.275 --> 00:33:07.500
say I'm seeing, you know,
00:33:07.500 --> 00:33:09.354
both at AHEAD and in our clients
00:33:09.354 --> 00:33:10.571
is you have... Again,
00:33:10.571 --> 00:33:12.174
a lot of people have ideas of things
00:33:12.174 --> 00:33:14.000
they want for AI, but what is a use case?
00:33:14.000 --> 00:33:14.650
And that goes back
00:33:14.650 --> 00:33:15.625
to that value conversation,
00:33:15.625 --> 00:33:17.188
and that's where a consultant,
00:33:17.188 --> 00:33:19.164
whether that's kind of an AHEAD consultant
00:33:19.164 --> 00:33:21.000
or a consultant inside your business...
00:33:21.000 --> 00:33:23.986
At AHEAD, we have consultants that work
00:33:23.986 --> 00:33:26.667
to go take those ideas and map them
00:33:26.667 --> 00:33:28.219
into real use cases and articulate
00:33:28.219 --> 00:33:30.000
the value of them with Susannah's team.
00:33:30.000 --> 00:33:31.790
Then you've got to engineer
00:33:31.790 --> 00:33:33.846
those onto an agentic platform,
00:33:33.846 --> 00:33:35.569
and that's where forward-deployed
00:33:35.569 --> 00:33:36.875
engineers really come in.
00:33:36.875 --> 00:33:39.062
So buddying them up with the business.
00:33:39.062 --> 00:33:40.835
Again, they're not just coding
00:33:40.835 --> 00:33:42.667
infrastructure on the back end.
00:33:42.667 --> 00:33:44.301
They really have to understand
00:33:44.301 --> 00:33:45.500
that workflow to think
00:33:45.500 --> 00:33:47.000
about the right architecture for it.
00:33:47.000 --> 00:33:48.708
And so I think that's
00:33:48.708 --> 00:33:50.417
probably the big one.
00:33:50.417 --> 00:33:51.747
You know, I worked with some
00:33:51.747 --> 00:33:53.125
clients recently, and in fact
00:33:53.125 --> 00:33:54.451
just had a conversation with one
00:33:54.451 --> 00:33:56.190
of the CTOs yesterday, and they were like,
00:33:56.190 --> 00:33:57.871
"It can't just be X person's job
00:33:57.871 --> 00:33:59.762
on top of the job they already have,
00:33:59.762 --> 00:34:00.797
and no one's thinking about all
00:34:00.797 --> 00:34:02.000
the people this person has to hire."
00:34:02.000 --> 00:34:03.220
And I think it's
00:34:03.220 --> 00:34:05.278
gonna create a lot of jobs.
00:34:05.278 --> 00:34:06.369
A lot of people
00:34:06.369 --> 00:34:08.333
have to learn and train up.
00:34:08.333 --> 00:34:09.510
But forward-deployed engineers
00:34:09.510 --> 00:34:10.882
and consultants that can understand
00:34:10.882 --> 00:34:12.482
the business and translate it into
00:34:12.482 --> 00:34:14.412
something working with a forward-deployed
00:34:14.412 --> 00:34:16.112
engineer can go and execute against
00:34:16.112 --> 00:34:18.056
and measure those results and refine it,
00:34:18.056 --> 00:34:20.041
I think is critical right now,
00:34:20.041 --> 00:34:21.562
and I think that's back
00:34:21.562 --> 00:34:23.809
to where we've seen people doing it well
00:34:23.809 --> 00:34:26.000
and where people aren't moving as fast.
00:34:26.000 --> 00:34:27.753
The ones that are doing it well have
00:34:27.753 --> 00:34:29.750
figured out some of the FinOps processes,
00:34:29.750 --> 00:34:31.611
have figured out how to measure,
00:34:31.611 --> 00:34:33.529
figured out how to map use cases.
00:34:33.529 --> 00:34:35.040
They've got intake processes, because
00:34:35.040 --> 00:34:36.591
you don't have enough forward-deployed
00:34:36.591 --> 00:34:37.578
engineer capacity to support
00:34:37.578 --> 00:34:38.636
the entire business right now.
00:34:38.636 --> 00:34:41.667
You know, if we went to every department,
00:34:41.667 --> 00:34:43.286
everybody's got 50 ideas, we would
00:34:43.286 --> 00:34:45.000
never have the engineering capacity.
00:34:45.000 --> 00:34:47.319
So which ones add the most value
00:34:47.319 --> 00:34:50.000
and are technically feasible as well?
00:34:50.000 --> 00:34:53.214
Yeah, I think a lot about this idea
00:34:53.214 --> 00:34:56.429
of citizen engineer or vibe coding,
00:34:56.429 --> 00:34:58.564
and I'm a big fan of people
00:34:58.564 --> 00:35:00.938
who have expertise should kind
00:35:00.938 --> 00:35:03.783
of stay in their lane. And so I think
00:35:03.783 --> 00:35:06.091
we have to also be intentional
00:35:06.091 --> 00:35:09.000
about where our business users should be.
00:35:09.000 --> 00:35:11.886
Like, do I think business users should
00:35:11.886 --> 00:35:15.000
be coding? No, but do I think they should
00:35:15.000 --> 00:35:17.328
be building UIs to demonstrate
00:35:17.328 --> 00:35:19.500
the functionality they want?
00:35:19.500 --> 00:35:21.485
Yes. Like, do I think product
00:35:21.485 --> 00:35:23.333
owners on the business side
00:35:23.333 --> 00:35:25.293
have a better toolkit now to illustrate
00:35:25.293 --> 00:35:27.353
and demonstrate what's important to them?
00:35:27.353 --> 00:35:28.851
Yes, but I don't think
00:35:28.851 --> 00:35:30.417
they have the skill set
00:35:30.417 --> 00:35:33.242
to build an enterprise solution that meets
00:35:33.242 --> 00:35:36.000
the regulatory and security requirements.
00:35:36.000 --> 00:35:38.182
Are you seeing pushback on that?
00:35:38.182 --> 00:35:40.429
And the other part of that question
00:35:40.429 --> 00:35:43.125
is really like it's a cost analysis thing.
00:35:43.125 --> 00:35:45.870
If I'm CapEx-ing software delivery,
00:35:45.870 --> 00:35:48.929
I can't CapEx probably a business user.
00:35:48.929 --> 00:35:51.086
It probably would get kind
00:35:51.086 --> 00:35:53.077
of flagged a little bit.
00:35:53.077 --> 00:35:56.592
But are you seeing people wanna change
00:35:56.592 --> 00:36:00.200
those rules and expand the group to do,
00:36:00.200 --> 00:36:01.200
you know, vibe coding?
00:36:01.200 --> 00:36:04.000
What are you seeing in the real world?
00:36:04.000 --> 00:36:06.178
Yeah, I mean, I'll say, I'll let Susannah
00:36:06.178 --> 00:36:08.250
jump in here too, but I would say there
00:36:08.250 --> 00:36:10.859
is a desire for both low-code, no-code,
00:36:10.859 --> 00:36:13.333
and pro-code platforms and workflows.
00:36:13.333 --> 00:36:15.519
And I think the other thing that goes
00:36:15.519 --> 00:36:17.941
along with that is sandbox to production.
00:36:17.941 --> 00:36:20.000
I mean, that's gotta be the workflow.
00:36:20.000 --> 00:36:22.925
Sandbox, go to town, figure out your idea,
00:36:22.925 --> 00:36:24.667
play with it, build a UI,
00:36:24.667 --> 00:36:27.267
stitch together a workflow, put in some
00:36:27.267 --> 00:36:30.000
dummy data, figure out what you wanna do.
00:36:30.000 --> 00:36:31.916
But the minute you're like, "Okay,
00:36:31.916 --> 00:36:33.889
I've got something that's real now.
00:36:33.889 --> 00:36:35.000
I've experimented with it.
00:36:35.000 -->
00:36:37.021
I wanna take this," whether it's
00:36:37.021 --> 00:36:39.231
into something like Copilot Studio,
00:36:39.231 --> 00:36:42.058
Glean, whatever the product set might be,
00:36:42.058 --> 00:36:44.333
whether it's pro code on Foundry,
00:36:44.333 --> 00:36:45.370
Bedrock, you know,
00:36:45.370 --> 00:36:47.500
Anthropic directly on their platform,
00:36:47.500 --> 00:36:50.233
that's where people just need that process
00:36:50.233 --> 00:36:52.250
to take it all the way through.
00:36:52.250 --> 00:36:53.942
But once you get out of sandbox,
00:36:53.942 --> 00:36:55.000
there is risk there.
00:36:55.000 --> 00:36:57.500
There's gotta be gates, and agents gotta
00:36:57.500 --> 00:36:59.293
go from sandbox to production through
00:36:59.293 --> 00:37:01.087
all the checkpoints that it needs to.
00:37:01.087 --> 00:37:02.551
Some of those can be agent checkpoints
00:37:02.551 --> 00:37:04.130
that are checking people are doing things
00:37:04.130 --> 00:37:05.565
right, but the final human-in-the-loop
00:37:05.565 --> 00:37:07.000
checkpoint to go to production is key.
00:37:07.000 --> 00:37:08.871
And then it's just a case of,
00:37:08.871 --> 00:37:11.000
okay, you wanna build this agent,
00:37:11.000 --> 00:37:13.358
and an agent's gonna run autonomously
00:37:13.358 --> 00:37:15.333
or on triggers, whatever it is.
00:37:15.333 --> 00:37:17.846
But who ultimately is accountable for that
00:37:17.846 --> 00:37:20.238
agent? I'm a big believer in every agent
00:37:20.238 --> 00:37:21.132
has a human counterpart
00:37:21.132 --> 00:37:22.143
that's accountable for it.
00:37:22.143 --> 00:37:24.048
You know, it's fine when you've got five
00:37:24.048 --> 00:37:26.000
to 10 in the environment, but when you've
00:37:26.000 --> 00:37:27.739
got hundreds, thousands, 10,000 running,
00:37:27.739 --> 00:37:29.000
who is accountable for those?
00:37:29.000 --> 00:37:31.533
And that comes back to what's your agent
00:37:31.533 --> 00:37:33.750
registry? What's running out there?
00:37:33.750 --> 00:37:35.945
And then that dovetails back into,
00:37:35.945 --> 00:37:38.333
without an agent registry and knowing
00:37:38.333 --> 00:37:40.112
what you've got and tagged,
00:37:40.112 --> 00:37:42.812
how can the financial office do their job
00:37:42.812 --> 00:37:43.877
correctly and security
00:37:43.877 --> 00:37:44.991
do their job correctly?
00:37:44.991 --> 00:37:46.501
Susannah, I expect you know
00:37:46.501 --> 00:37:48.235
that I'm leading to this, which
00:37:48.235 --> 00:37:50.118
is we treat the cost of agents different
00:37:50.118 --> 00:37:52.000
than we treat the cost of people, right?
00:37:52.000 --> 00:37:53.688
They have sort of different
00:37:53.688 --> 00:37:55.250
balance-sheet treatments.
00:37:55.250 --> 00:37:57.250
You know, the cost of agents is always an
00:37:57.250 --> 00:37:59.250
operational cost, and the cost of people,
00:37:59.250 --> 00:38:00.894
depending on what they're doing,
00:38:00.894 --> 00:38:03.000
can be capitalized for what they deliver.
00:38:03.000 --> 00:38:05.625
But what are you seeing as, like,
00:38:05.625 --> 00:38:08.901
clients asking to measure their human
00:38:08.901 --> 00:38:12.000
capital versus their agent capital?
00:38:12.000 --> 00:38:12.870
You know, are people
00:38:12.870 --> 00:38:14.000
starting to think, "Oh no,
00:38:14.000 --> 00:38:17.558
it would be a good idea for these users
00:38:17.558 --> 00:38:20.294
to maybe build more software,"
00:38:20.294 --> 00:38:21.716
maybe they previously hadn't.
00:38:21.716 --> 00:38:22.941
But to me, this starts...
00:38:22.941 --> 00:38:24.441
You know, I think I'm mixing
00:38:24.441 --> 00:38:26.316
my chocolate with my peanut butter.
00:38:26.316 --> 00:38:28.116
It gets very confusing for me.
00:38:28.116 --> 00:38:29.737
What are you seeing as sort
00:38:29.737 --> 00:38:32.153
of being the active approach
00:38:32.153 --> 00:38:35.000
to separating those costs or not?
00:38:35.000 --> 00:38:36.312
I, from my personal experience,
00:38:36.312 --> 00:38:37.750
don't think we're quite there yet.
00:38:37.750 --> 00:38:39.884
I think we're still trying to get our arms
00:38:39.884 --> 00:38:41.765
wrapped around what that cost profile
00:38:41.765 --> 00:38:44.109
is in aggregate versus trying to figure
00:38:44.109 --> 00:38:46.333
out kind of the ideal CapEx/OpEx mix,
00:38:46.333 --> 00:38:49.463
which takes us back to kind of the overall
00:38:49.463 --> 00:38:52.368
governance theme. But Nick, you may see
00:38:52.368 --> 00:38:53.650
something different given the clients
00:38:53.650 --> 00:38:55.000
that you interact with most frequently.
00:38:55.000 --> 00:38:58.117
Yep, pretty similar. I mean, I still
00:38:58.117 --> 00:39:01.667
think today agents are a cost of software
00:39:01.667 --> 00:39:04.350
and tooling to perform an outcome today,
00:39:04.350 --> 00:39:06.765
and human costs are human costs and,
00:39:06.765 --> 00:39:08.476
you know, they're treated differently.
00:39:08.476 --> 00:39:10.278
You've got people and culture teams, HR,
00:39:10.278 --> 00:39:11.552
however you kind of phrase
00:39:11.552 --> 00:39:12.778
them in the organization.
00:39:12.778 --> 00:39:14.768
There's just a lot of differences
00:39:14.768 --> 00:39:17.000
that come with an agent and a person.
00:39:17.000 --> 00:39:18.413
So I don't think we should
00:39:18.413 --> 00:39:19.500
treat them the same.
00:39:19.500 --> 00:39:21.923
That may be a controversial opinion,
00:39:21.923 --> 00:39:24.211
but that's generally how I see it.
00:39:24.211 --> 00:39:27.085
I think what is going to change
00:39:27.085 --> 00:39:30.238
is the rise of bots versus agents,
00:39:30.238 --> 00:39:31.246
which are still essentially
00:39:31.246 --> 00:39:32.143
agents under the covers,
00:39:32.143 --> 00:39:33.800
but the way they're triggered
00:39:33.800 --> 00:39:35.000
and the way they run.
00:39:35.000 --> 00:39:37.526
But now you're starting to treat
00:39:37.526 -->
00:39:40.526
a bot to take on a function or a role,
00:39:40.526 --> 00:39:42.201
now that bot may have sub-functions
00:39:42.201 --> 00:39:44.211
and agents and models and everything else,
00:39:44.211 --> 00:39:45.613
and tools and connectors
00:39:45.613 --> 00:39:47.250
that it needs to do its job.
00:39:47.250 --> 00:39:49.750
But I still think at the end of the day,
00:39:49.750 --> 00:39:51.885
while it may get treated more like a name
00:39:51.885 --> 00:39:53.500
of an employee in the workflow,
00:39:53.500 --> 00:39:54.650
you're still gonna know
00:39:54.650 --> 00:39:56.000
it's an AI employee, right?
00:39:56.000 --> 00:39:59.636
Or AI agent in this case, or AI bot.
00:39:59.636 --> 00:40:01.206
But it's ultimately still
00:40:01.206 --> 00:40:03.214
a cost of delivering a workflow.
00:40:03.214 --> 00:40:05.116
It's software and tooling and, you know,
00:40:05.116 --> 00:40:06.875
infrastructure at the end of the day,
00:40:06.875 --> 00:40:08.750
and I think it still comes back to that.
00:40:08.750 --> 00:40:09.390
I think you just gotta
00:40:09.390 --> 00:40:10.000
tag it appropriately.
00:40:10.000 --> 00:40:12.500
For the last few minutes here,
00:40:12.500 --> 00:40:15.375
I wanna talk about on-prem versus
00:40:15.375 --> 00:40:17.727
the cloud for AI workloads,
00:40:17.727 --> 00:40:20.475
and what are the key things
00:40:20.475 --> 00:40:23.833
that our clients should consider.
00:40:23.833 --> 00:40:26.000
Susannah, do you wanna start for us?
00:40:26.000 --> 00:40:28.400
I think from my perspective,
00:40:28.400 --> 00:40:30.978
we're seeing highly regulated
00:40:30.978 --> 00:40:34.000
clients have on-prem AI workloads.
00:40:34.000 --> 00:40:36.667
And that's getting a little more complex
00:40:36.667 --> 00:40:38.567
with the supply chain considerations
00:40:38.567 --> 00:40:40.625
that are amplifying each and every day.
00:40:40.625 --> 00:40:43.163
But I think, you know, for the folks
00:40:43.163 --> 00:40:46.053
that are experimenting and kinda dabbling
00:40:46.053 --> 00:40:48.200
in it or early in the cycle, we're seeing
00:40:48.200 --> 00:40:50.294
most of those workloads be in the cloud.
00:40:50.294 --> 00:40:51.817
Those that are more advanced,
00:40:51.817 --> 00:40:53.235
we are seeing those clients
00:40:53.235 --> 00:40:55.507
reach a tipping point where it does make
00:40:55.507 --> 00:40:57.778
economic sense to bring it back on-prem.
00:40:57.778 --> 00:40:59.925
So I think, you know, most of our
00:40:59.925 --> 00:41:02.462
financial clients are doing the latter.
00:41:02.462 --> 00:41:05.000
But Nick, what would you...
00:41:05.000 --> 00:41:07.056
Yeah. I would agree, and I think it's
00:41:07.056 --> 00:41:09.000
smart to have a dual strategy here,
00:41:09.000 --> 00:41:12.417
a hybrid strategy. I don't think going
00:41:12.417 --> 00:41:15.833
all in on one hyperscaler or AI-native
00:41:15.833 --> 00:41:16.944
platform alone makes sense.
00:41:16.944 --> 00:41:18.935
Doesn't mean it's not your primary,
00:41:18.935 --> 00:41:21.154
but I think you need options right now.
00:41:21.154 --> 00:41:23.411
I think learning how portable
00:41:23.411 --> 00:41:25.357
an agent is is important.
00:41:25.357 --> 00:41:26.944
Ensuring that you have a
00:41:26.944 --> 00:41:28.929
model-agnostic layer, I think,
00:41:28.929 --> 00:41:30.988
is very important on how you do that,
00:41:30.988 --> 00:41:33.158
whether it's LLM gateways or using some
00:41:33.158 --> 00:41:34.974
of the abstraction tools, et cetera,
00:41:34.974 --> 00:41:36.286
I think is very important.
00:41:36.286 --> 00:41:38.581
And I think it's gonna give flexibility
00:41:38.581 --> 00:41:41.053
to all the open-weight models that are out
00:41:41.053 --> 00:41:43.052
there too. There's no reason you always
00:41:43.052 --> 00:41:45.000
have to pay the premium for a frontier
00:41:45.000 --> 00:41:46.952
model from the big providers.
00:41:46.952 --> 00:41:49.375
So I think a dual strategy is smart.
00:41:49.375 --> 00:41:51.717
I think if you're gonna be experimenting
00:41:51.717 --> 00:41:54.000
and maybe you build on Claude platform,
00:41:54.000 --> 00:41:55.250
Microsoft, Google, Amazon, whatever,
00:41:55.250 --> 00:41:57.168
go and build a cluster in your data
00:41:57.168 --> 00:41:59.250
center as well, and make sure you know
00:41:59.250 --> 00:42:01.003
how to just deploy through that life
00:42:01.003 --> 00:42:03.000
cycle of building and deploying an agent.
00:42:03.000 --> 00:42:04.285
Great, I can deploy
00:42:04.285 --> 00:42:06.111
it here in the hyperscaler.
00:42:06.111 --> 00:42:07.540
I can deploy it directly
00:42:07.540 --> 00:42:08.611
in my data center.
00:42:08.611 --> 00:42:10.563
It has access to my toolbox with my tools,
00:42:10.563 --> 00:42:12.143
connectors, and everything I need.
00:42:12.143 --> 00:42:14.521
I figured all of that out. I've got
00:42:14.521 --> 00:42:17.308
my observability and everything baked in.
00:42:17.308 --> 00:42:19.974
And I think once people build
00:42:19.974 --> 00:42:22.273
that and have that ready,
00:42:22.273 --> 00:42:23.915
then it makes the building
00:42:23.915 --> 00:42:26.000
agents conversation a lot easier.
00:42:26.000 --> 00:42:27.389
What I hear sometimes is like,
00:42:27.389 --> 00:42:28.824
"Well, where should I build it?
00:42:28.824 --> 00:42:30.476
And do we have a platform ready?
00:42:30.476 --> 00:42:31.857
Oh, I don't... I can't build.
00:42:31.857 --> 00:42:33.810
I can't take the use case to production."
00:42:33.810 --> 00:42:34.839
And the barrier is usually
00:42:34.839 --> 00:42:35.750
financial and security.
00:42:35.750 --> 00:42:37.250
We need a financial lens
00:42:37.250 --> 00:42:39.250
on it and a security lens on it.
00:42:39.250 --> 00:42:40.631
So I think getting those
00:42:40.631 --> 00:42:42.300
platforms right is important.
00:42:42.300 --> 00:42:44.211
But don't make it the two-year
00:42:44.211 --> 00:42:46.250
project to get a hybrid platform
00:42:46.250 -->
00:42:48.482
off the ground that you never make
00:42:48.482 --> 00:42:50.714
any progress on building an agent.
00:42:50.714 --> 00:42:52.619
But yeah, I think it's smart.
00:42:52.619 --> 00:42:54.015
I think on-prem makes total sense,
00:42:54.015 --> 00:42:55.000
along with hyperscalers.
00:42:55.000 --> 00:42:57.037
I agree. I think a lot of clients
00:42:57.037 --> 00:42:59.444
are looking at it workload by workload,
00:42:59.444 --> 00:43:01.000
which I think is the right way to do it.
00:43:01.000 --> 00:43:03.395
But it all hinges on you have to
00:43:03.395 --> 00:43:05.789
have your kind of value and cost
00:43:05.789 --> 00:43:07.656
hypothesis on the front end, so you've
00:43:07.656 --> 00:43:09.474
gotta kinda back yourself up to that,
00:43:09.474 --> 00:43:11.671
what is that intake process? And then
00:43:11.671 --> 00:43:13.750
you have to continuously, you know,
00:43:13.750 --> 00:43:15.870
snap a line and measure progress
00:43:15.870 --> 00:43:17.857
against the entire life cycle,
00:43:17.857 --> 00:43:20.000
'cause you can deviate very, very quickly.
00:43:20.000 --> 00:43:22.681
And again, you may have selected on-prem
00:43:22.681 --> 00:43:25.294
for a particular use case at the start,
00:43:25.294 --> 00:43:26.765
and something changes,
00:43:26.765 --> 00:43:28.235
and you need to pivot.
00:43:28.235 --> 00:43:30.532
So having that governance and that process
00:43:30.532 --> 00:43:32.500
and having the financial work stream
00:43:32.500 --> 00:43:34.750
running in parallel with the execution
00:43:34.750 --> 00:43:37.000
of that use case is gonna be critical.
00:43:37.000 --> 00:43:38.400
Really appreciate both of you.
00:43:38.400 --> 00:43:40.419
I think this conversation is one
00:43:40.419 --> 00:43:42.059
of the most important ones
00:43:42.059 --> 00:43:44.092
for people to be having right now. It's
00:43:44.092 --> 00:43:46.176
the combination of technology and value,
00:43:46.176 --> 00:43:48.518
which is the business that we're in.
00:43:48.518 --> 00:43:51.250
So thank you for giving me the time today,
00:43:51.250 --> 00:43:53.791
and I look forward to reconnecting
00:43:53.791 --> 00:43:55.286
again in the future.
00:43:55.286 --> 00:43:57.000
Susannah, Nick, thank you so much.
00:43:57.000 --> 00:44:00.043
This has been Ready, Set, Runtime,
00:44:00.043 --> 00:44:01.923
a podcast from AHEAD.
00:44:01.923 --> 00:44:05.150
I'm MK, and I'll see you next time. Thank
00:44:05.150 --> 00:44:08.378
you for listening to Ready, Set, Runtime.
00:44:09.444 --> 00:44:10.822
If something in today's
00:44:10.822 --> 00:44:12.500
conversation sparked an idea
00:44:12.500 --> 00:44:15.179
or a question about your own environment,
00:44:15.179 --> 00:44:17.857
AHEAD's team would love to hear from you.
00:44:17.857 --> 00:44:19.893
Reach out anytime at ahead.com,
00:44:19.893 --> 00:44:21.667
and if you haven't already,
00:44:21.667 --> 00:44:24.000
follow the show wherever you're tuning in.
00:44:24.000 --> 00:44:26.875
The views shared on this podcast are those
00:44:26.875 --> 00:44:31.068
of the speakers and don't necessarily
00:44:31.068 --> 00:44:35.600
reflect those of AHEAD. Until next time.