WEBVTT
1
00:00:00.000 --> 00:00:04.000
Jeff McMillan, the former head of AI at Morgan Stanley.
2
00:00:04.000 --> 00:00:08.640
Jeff believes we are at the tipping point where companies are either going to learn how to
3
00:00:09.280 --> 00:00:15.200
adopt AI and they're going to thrive or they're basically going to be on a slow death march
4
00:00:15.200 --> 00:00:19.440
because of their inability to adapt. What we've done is we've distilled down
5
00:00:19.440 --> 00:00:25.840
some of his best ideas from the conversation and talked about why he was so excited about what
6
00:00:25.840 --> 00:00:31.840
was happening that he actually left Morgan Stanley to go work with other companies on helping them
7
00:00:31.840 --> 00:00:33.840
make this transition.
8
00:00:41.200 --> 00:00:48.080
You've got the most unlikely AI background in a lot of ways. You graduated West Point,
9
00:00:48.080 --> 00:00:51.280
you worked in the Army, I have a degree in economics if I'm not mistaken,
10
00:00:52.080 --> 00:00:56.480
and that led you to running AI at Morgan Stanley.
11
00:00:56.480 --> 00:01:00.480
If you don't mind just giving us a minute or two on that journey, it would be great just to
12
00:01:00.480 --> 00:01:03.680
orient everybody. So first of all Keith, thanks for having me.
13
00:01:04.400 --> 00:01:07.520
And second of all, I have no idea how I wound up where I am right now.
14
00:01:09.040 --> 00:01:13.440
As you mentioned, I was a young lieutenant who had gotten out of the military.
15
00:01:14.160 --> 00:01:18.640
And if you're familiar with veterans who are transitioning, there's lots of skills and
16
00:01:18.640 --> 00:01:22.800
leadership that you get from military, but you don't really know how to do much of anything.
17
00:01:22.800 --> 00:01:28.320
And I think like many veterans, I struggled. I actually bounced around two or three jobs
18
00:01:28.320 --> 00:01:36.400
that I was neither good at nor enjoyed until I found my first job back at Morgan Stanley,
19
00:01:36.400 --> 00:01:42.720
which was an internal consulting group. And my first project was to clean up the customer
20
00:01:42.720 --> 00:01:48.720
information of Morgan Stanley's customer accounts. And what's funny at the time is I thought I
21
00:01:48.720 --> 00:01:54.720
was actually qualified to do that job when they assigned it to me. Little that I know that
22
00:01:54.720 --> 00:02:02.240
long-term capital would hit about six months later. And my little $2 million project became a
23
00:02:02.240 --> 00:02:07.760
multi-year, multi-million dollar project. And I became, for lack of a better term, an expert in
24
00:02:07.840 --> 00:02:11.920
data quality. And I only mentioned that because, I mean, first of all, that's the stark for me,
25
00:02:11.920 --> 00:02:16.800
but it's really funny because back in the mid-90s, data quality was really important.
26
00:02:17.600 --> 00:02:23.520
And guess what? In 2026, data quality is probably even more important. And I don't think
27
00:02:23.520 --> 00:02:29.120
many organizations have progressed much further than they have in the last 20, 30 years. But
28
00:02:29.120 --> 00:02:35.280
we can talk about that maybe later. You know, my career is sort of a career of the growth
29
00:02:35.280 --> 00:02:40.480
of analytics and data. I worked at Merrill for a while. We used data to understand what clients
30
00:02:40.480 --> 00:02:47.040
were doing and how to drive more value with our research. And then in 2009, I went to work at Morgan
31
00:02:47.040 --> 00:02:53.520
Stanley. I had this very simple idea that we could use data to drive more and affect more
32
00:02:53.520 --> 00:02:58.160
effective targeting to our clients using machine learning algorithms. And this was, you know,
33
00:02:58.160 --> 00:03:03.840
09, so it was sort of the early days of machine learning. And, you know, honestly, that took me
34
00:03:03.840 --> 00:03:10.800
nine years to do. And it wasn't until 2018 that we released the first algorithmically driven
35
00:03:10.800 --> 00:03:16.320
recommendation engine for financial advisors, which we called next best action. You know, I kept
36
00:03:16.320 --> 00:03:22.560
sort of looking back and saying, wow, this, when you put really good content into these algorithms,
37
00:03:22.640 --> 00:03:27.760
and you have really smart employees, in this case, financial advisors, and you give them the
38
00:03:27.760 --> 00:03:33.840
right day at the right time, really good things, really good things happen. And then in March of 22,
39
00:03:33.840 --> 00:03:39.360
so four years ago, almost to the day, myself and a few others leaders at Morgan Stanley were sitting
40
00:03:39.360 --> 00:03:45.520
in a room in Palo Alto, meeting with a bunch of startups. And one of the eight startups that we
41
00:03:45.520 --> 00:03:51.200
met today with a small unknown company called OpenAI, led by at the time this unknown guy named Sam
42
00:03:52.160 --> 00:03:58.240
Altman. And we became the first customer of OpenAI. And we became the first
43
00:03:59.040 --> 00:04:04.880
organization to deploy anything in production. And, you know, what I would say Keith is,
44
00:04:05.680 --> 00:04:11.040
I'm not the smartest engineer in the world. I'm certainly not an engineer at all. But what I've
45
00:04:11.040 --> 00:04:16.480
tried to do, and where I may be, if I've had any, any effect this in career, my career, I sort of
46
00:04:16.480 --> 00:04:21.200
looked at that inflection point between business people trying to solve a problem and
47
00:04:21.200 --> 00:04:25.280
an immersion technologies and connecting those dots in a way that drives business value.
48
00:04:25.280 --> 00:04:30.400
Two years ago, I became the first head of AI at Morgan Stanley, which I served on, you know,
49
00:04:30.400 --> 00:04:37.600
for two years before, before recently leaving the firm to do my own thing. So that's kind of my
50
00:04:37.600 --> 00:04:43.840
life story in five minutes. So just to jump back to 2022, that's interesting. When you became the
51
00:04:43.840 --> 00:04:49.760
first customer, what were they, what were they selling back then? And why did you guys
52
00:04:50.960 --> 00:04:55.040
choose to buy it? I'm not sure they knew what they were selling exactly. I think what they were
53
00:04:55.040 --> 00:05:01.280
looking like many startups, they were looking to develop a partner or partner relationships.
54
00:05:02.560 --> 00:05:10.400
They like many firms were early in their maturity cycle. So, you know, the things that you need to
55
00:05:10.400 --> 00:05:16.640
do to sell to an enterprise customer, you know, entitlements, BYOK, right? All the information
56
00:05:16.640 --> 00:05:21.200
security issues, you know, those were, those were things that, you know, I think they were
57
00:05:21.200 --> 00:05:26.320
grappling with and they were looking for a partner. So I think, I think what were they looking for?
58
00:05:26.320 --> 00:05:30.000
They were looking for a partner. What were they were selling? They were selling GPT-3.
59
00:05:31.360 --> 00:05:35.600
That was the product at the time. And, you know, that was seven months before,
60
00:05:36.560 --> 00:05:43.280
before CHECK GPT hit the, hit the transom. And, you know, we had our CHECK GPT moment,
61
00:05:43.280 --> 00:05:48.400
like back in March, right, four years ago. And, you know, if you think about financial services,
62
00:05:48.400 --> 00:05:54.400
we move a lot of information across a lot of different channels. And it was clear that this technology,
63
00:05:54.400 --> 00:06:01.440
although, again, clearly undefined, uninformed in many ways, would be very helpful. Actually,
64
00:06:01.520 --> 00:06:05.120
the first use case is we spent a lot of time thinking about the first use case. And our first
65
00:06:05.120 --> 00:06:10.480
use case was a simple bot for the financial advisors to help them find, you know, information in real
66
00:06:10.480 --> 00:06:14.880
time as they were talking to clients about any number of a million different topics. Two comments,
67
00:06:14.880 --> 00:06:21.520
one is it ended up being incredibly useful and impactful. And two, it was incredibly challenging
68
00:06:21.520 --> 00:06:27.360
and difficult to actually deploy that, you know, in an enterprise solution, given the nature of
69
00:06:27.440 --> 00:06:31.680
the new technology and just a lack of understanding and the governance and all that other stuff that
70
00:06:31.680 --> 00:06:39.360
was necessary to be successful. Flash forward four years, it's 2026. You just wrote a couple of weeks
71
00:06:39.360 --> 00:06:46.880
ago in article or a framework, I guess, that really breaks down enterprise AI into six layers.
72
00:06:47.760 --> 00:06:55.040
And I'm assuming that that based on the learnings that you had as part of sort of this rollout,
73
00:06:55.040 --> 00:06:58.160
I'm wondering if you don't mind just walking us through those layers for a second,
74
00:06:58.160 --> 00:07:01.040
because it really resonated with me when I read it. It all made a lot of sense.
75
00:07:01.040 --> 00:07:05.600
There were two drivers. One is how do my job is I'm really am trying to just educate people and
76
00:07:05.600 --> 00:07:13.440
raise the literacy of our society, because I think a well educated, responsible group of professionals
77
00:07:13.440 --> 00:07:19.120
in this world will be better fiduciaries of this technology than ones that are not. So my goal is to
78
00:07:19.120 --> 00:07:24.160
hopefully make people smarter to include myself in a world in which this technology is moving very
79
00:07:24.160 --> 00:07:29.600
quickly. That's pointable. And point number two is everyone is focused on two things. They're
80
00:07:29.600 --> 00:07:34.960
focused on vendor or own built solutions, applications, and they're focused on models.
81
00:07:35.600 --> 00:07:41.280
And what in my own personal experience, those are the visible things and they certainly get the hype
82
00:07:41.280 --> 00:07:47.040
that's not where the ultimate long-term value is going to come from. Particularly, if you're only
83
00:07:47.040 --> 00:07:54.000
looking to deploy six bots in your organization, sure go ahead, pick a model or pick two models
84
00:07:54.000 --> 00:07:59.120
and pick three applications and go to work. But what happens when my organization or your organization
85
00:07:59.120 --> 00:08:08.240
has 20 or 200 or 2,000 or 20,000? And I don't believe I'm exaggerating any narrative
86
00:08:08.240 --> 00:08:15.360
reform that in the next 10 years, Fortune 500 firms are going to have tens of thousands of agents
87
00:08:15.440 --> 00:08:20.240
that are going to be talking to each other. And if you believe that to be true, which I fundamentally
88
00:08:20.240 --> 00:08:27.760
believe in the timing we can argue about, you need to have an infrastructure that is scalable
89
00:08:28.640 --> 00:08:34.080
and is able to manage the growth of that infrastructure in a controlled and responsible way.
90
00:08:34.080 --> 00:08:39.440
So what does that mean? That means first and foremost, you need an accessible data layer,
91
00:08:39.440 --> 00:08:45.440
which by the way, I'd probably have done 30 CEO conversations the last two months.
92
00:08:46.640 --> 00:08:52.160
Everyone's data is a problem and it's either a data access problem or it's a data quality problem.
93
00:08:52.160 --> 00:08:56.640
And the point is AI doesn't solve that just by putting it on top. You actually have to do
94
00:08:56.640 --> 00:09:00.400
real stuff to make that work. So there's a, first of all, you got your what I'll call your
95
00:09:00.400 --> 00:09:05.280
foundational layer. Then you have what I'll call your data accessibility later, which is your models,
96
00:09:05.280 --> 00:09:09.440
your rag infrastructure, your knowledge graphs. And again, we could talk a lot about that.
97
00:09:09.440 --> 00:09:14.320
But the point is is that AI does better when the information is structured in a certain way,
98
00:09:15.040 --> 00:09:19.520
just throwing up a bunch of PDFs into a database doesn't necessarily drive value.
99
00:09:20.720 --> 00:09:24.480
The third is what I'll call the governance and control layer, which by the way no one's talking
100
00:09:24.480 --> 00:09:32.320
about. You need a mechanism to entitle the right information, the right people, to essentially
101
00:09:32.560 --> 00:09:39.600
make sure that the model is doing what you want it to do and not doing what you don't want it to do.
102
00:09:39.600 --> 00:09:44.960
And that needs to be very explicit. The fourth is the models, right? And by the way, you can get
103
00:09:44.960 --> 00:09:49.360
one model, two models. Most firms are using more than one right now. But I will tell you,
104
00:09:49.360 --> 00:09:52.640
you're not going to fail because you don't pick the right model. All the models are brilliant.
105
00:09:53.360 --> 00:09:57.840
There's so much more advanced than we as a business star. It's not even worth arguing about.
106
00:09:58.320 --> 00:10:03.440
The fifth layer is the other piece that I don't think we get enough play. And I know we talk about
107
00:10:03.440 --> 00:10:06.960
agents. But what I'd like to talk about is what I'm going to call the
108
00:10:06.960 --> 00:10:14.000
agentic orchestration layer. So if you think about, if you think about the world that we're moving
109
00:10:14.000 --> 00:10:22.400
into, you're going to need this orchestration tool to be able to not only do the individual tasks
110
00:10:22.400 --> 00:10:28.480
that you want the AI to do, but to do those tasks across the organization. And if you have more
111
00:10:28.480 --> 00:10:33.440
than one of these, what I'm calling orchestration layers, you're going to get yourself into a mess
112
00:10:33.440 --> 00:10:38.240
because the compliance tool is not going to talk to the risk tool. And the risk tool is not going
113
00:10:38.240 --> 00:10:44.000
to talk to the financial tool. And we can talk about the right vendors and there's hundreds of
114
00:10:44.000 --> 00:10:48.720
in this space right now. And I'm not even here to sort of argue about that. But what I will say is
115
00:10:49.280 --> 00:10:53.840
that organizations need to be very thoughtful about the angle of agentic layer and building that
116
00:10:53.840 --> 00:10:57.760
in a way that scales. And then finally, you have applications, right? And those applications,
117
00:10:58.320 --> 00:11:02.560
the build versus buy decision is a lot more complicated than used to be. I think people are going
118
00:11:02.560 --> 00:11:09.040
to build a lot more than they used to because the ease of the solutions. But even if you are buying,
119
00:11:09.040 --> 00:11:14.640
you're still going to want the thing that you buy. What are the criterias to fit on that agentic layer?
120
00:11:14.640 --> 00:11:17.520
So those are what I'll call the six layers. And there's actually one what I'm going to call
121
00:11:17.520 --> 00:11:22.480
horizontal layer, which is really your monitoring and evaluation observability, right? You're going to
122
00:11:22.480 --> 00:11:29.680
need a tool that essentially is able to sit across all six of these components and essentially
123
00:11:29.680 --> 00:11:36.960
watch what's going on, being able to not understand when things break, understand how you're going
124
00:11:36.960 --> 00:11:44.000
to fix those things, being able to demonstrate that you know why you got the answers that you got.
125
00:11:44.560 --> 00:11:48.960
And whether that's internal needs or regulatory, I don't really care. You still need to be able to
126
00:11:48.960 --> 00:11:54.240
and be able to defend that your system is operating in a way that is consistent with your
127
00:11:54.240 --> 00:11:59.200
expectations. Now the good news there is that you can actually use AI to look at the AI,
128
00:11:59.200 --> 00:12:03.440
which is a pretty powerful capability. But I think the point when I'm talking to my customers
129
00:12:04.480 --> 00:12:09.280
and you know leaders, I'm really emphasizing this six tiered approach and it's not like you just
130
00:12:09.280 --> 00:12:13.520
go out and buy the whole thing, you know, it's pieces and it's messy and whatever. But I think
131
00:12:13.520 --> 00:12:18.640
has a strategy or organizations really need to think about layers and that horizontal piece and
132
00:12:18.640 --> 00:12:25.760
starting to build that out in a structured way. A lot of focus is on that model layer and the press
133
00:12:25.760 --> 00:12:32.240
certainly likes to talk about the model layer. The application layer ultimately, how important is
134
00:12:32.800 --> 00:12:41.840
to pick a vendor that can scale horizontally because it'll create interoperability? What are your
135
00:12:41.840 --> 00:12:46.080
thoughts on? Okay, we're going to go with a, you know, I think it's Sarah and customer service,
136
00:12:46.080 --> 00:12:51.040
right? And they're going to do it and then they're going to go everywhere or do you think it doesn't
137
00:12:51.040 --> 00:12:57.680
matter for where we are today? Well, so first of all, if you're in the application layer,
138
00:12:57.680 --> 00:13:01.520
you're a vendor in the application layer, you're trying to move down the stack. So you're
139
00:13:01.520 --> 00:13:08.320
already built out your agentry framework. If you are one of the data providers, you're trying to
140
00:13:08.320 --> 00:13:12.800
move up the stack and creating an attempt at what? And if you're a cloud services company, you're
141
00:13:12.800 --> 00:13:16.400
trying to do all of it. And that's right. If you're a cloud services company, you're trying to do
142
00:13:16.400 --> 00:13:24.960
all of it. So which by the way, if I'm advising a firm, I'm, I'm, I'm, I am going to tell them to do
143
00:13:24.960 --> 00:13:33.440
that because I don't know how else you're going to compete unless, unless I think what will likely
144
00:13:33.440 --> 00:13:38.400
happen is that there will be n number of models. You're not going to have that many. You're going
145
00:13:38.400 --> 00:13:44.000
to have n number of agentic orchestration layers. No, this could be from Azure, they could be Amazon,
146
00:13:44.000 --> 00:13:49.200
they could be in traffic. I, again, I'm not here to pick the winners. And then what's very likely,
147
00:13:49.200 --> 00:13:55.200
similar to like we have the Apple store and the Google store, I can imagine a world where there's
148
00:13:55.200 --> 00:14:01.200
a handful of agentic players in the space. And then if you are a vendor solution that wants to
149
00:14:01.200 --> 00:14:06.640
build something in a specific vertical, you will just build it in the context of that framework.
150
00:14:06.640 --> 00:14:13.200
So when you come to me as a customer, I'll say, are you, are you certified by the MCP protocol
151
00:14:13.200 --> 00:14:19.920
of XYZ infrastructure? And you will say, yes, I am certified. I meet all the protocol standards
152
00:14:19.920 --> 00:14:24.480
and I work very well in their stack. And if you don't, I won't be able to do business with you.
153
00:14:24.480 --> 00:14:32.480
So I could imagine a world where we have a few number of agentic solutions, a few number of,
154
00:14:32.480 --> 00:14:38.400
of models. And then that there's a lot of mix and match underneath it in terms of the data set
155
00:14:38.480 --> 00:14:44.800
themselves. Maybe the governance gets tied into the agentic frameworks, probably does. But you've
156
00:14:44.800 --> 00:14:50.800
got more flexibility on the applications so long as they are built in a way that's, you know,
157
00:14:50.800 --> 00:14:56.320
structured for that protocol that you've defined. So just for those who aren't familiar, can you
158
00:14:56.320 --> 00:15:04.720
explain the MCP? MCP is model control protocol. It actually came out of Anthropic. And you know,
159
00:15:04.720 --> 00:15:09.120
the way I kind of describe it, if I wrote, actually wrote a piece on this recently as well,
160
00:15:09.120 --> 00:15:14.480
you know, the 1930s, and you can fact check them in this, but there were 27 different gauges of
161
00:15:14.480 --> 00:15:20.640
railways in the United States. So every, by the way, if you want to read about like the AI
162
00:15:20.640 --> 00:15:25.360
Bloom, read about the railways, it's the same, it's almost the same story. It's that the parallels
163
00:15:25.360 --> 00:15:33.600
are actually quite funny in some ways. And it wasn't until industry government regulators got
164
00:15:33.600 --> 00:15:39.280
together and said, this is how wide a railway gauge is going to be. And oh, by the way, we're going
165
00:15:39.280 --> 00:15:42.960
to have time zones in three parts of the country. I mean, at one point, there was like 30 different
166
00:15:42.960 --> 00:15:46.560
time zones, like you could be in Stanford and I could be New York City and it would be, you know,
167
00:15:46.560 --> 00:15:51.920
three hours difference. It wasn't until that, and then they established some safety standards
168
00:15:51.920 --> 00:15:57.600
that you were able to drive to Florida to New York on one trade, right? And that was a massively
169
00:15:57.600 --> 00:16:02.480
disruptive concept because that's when capital formed. And if you think about MCP, think about MCP
170
00:16:02.480 --> 00:16:11.120
as the gauge for AI. It's the language that allows these disparate AI solutions, not just to
171
00:16:11.120 --> 00:16:16.640
talk to each other, but to talk to applications because when you get into an autonomous or semi-autonomous
172
00:16:16.640 --> 00:16:24.640
world, the AI does the thinking, but then it needs to act with machines, databases, applications,
173
00:16:24.640 --> 00:16:34.000
websites to action the work. And think about MCP as the common language that we can all use so
174
00:16:34.000 --> 00:16:42.160
that when I talk to you, I know who you are as an application, I know what you do, I know the
175
00:16:42.160 --> 00:16:48.800
information that you can't provide me, the entitlements that I have access to, maybe even the
176
00:16:48.800 --> 00:16:54.320
financial economics of it over time. And what I'll say is, you know, this is a whole another
177
00:16:54.320 --> 00:17:01.600
conversation, that there is still a tremendous amount of work that needs to be done around these
178
00:17:01.600 --> 00:17:07.600
protocols on an application by application level to make them really effective both within organizations
179
00:17:07.600 --> 00:17:12.640
and outside of organizations. But I think over time you're going to see a closing of standards and
180
00:17:12.640 --> 00:17:17.920
what that comes from governments industries or somebody else. I'm not really sure, but I think
181
00:17:17.920 --> 00:17:22.960
that's going to be super important as we try to really start to scale and commercialize
182
00:17:22.960 --> 00:17:29.520
authentic solutions. You're at this point where you're a leader of a, there's the Morgan
183
00:17:29.520 --> 00:17:36.240
Stanley's of the world and then there's a $150,200 million business and each of those problems
184
00:17:36.240 --> 00:17:40.080
are very different. Sort of take each of those spectrums. If you're looking at this layer,
185
00:17:40.960 --> 00:17:45.760
then you assume based on what you've said, the data, you have to start somewhere foundationally
186
00:17:45.760 --> 00:17:53.200
with data. Where do you go next? Right now, there are no proven players that will work across
187
00:17:53.200 --> 00:18:00.880
these seven stack, right? Now, there's lots of startups with no customers that have built these
188
00:18:00.880 --> 00:18:07.280
solutions. So what do you do? You, you work with the hyper scalers that are in place today,
189
00:18:08.400 --> 00:18:12.640
who are bringing it together, but bringing it together slowly, the Microsoft, to the world,
190
00:18:13.360 --> 00:18:22.080
Google or AWS. If I'm a $100 million business, that's probably what I'm doing. Or I'm working very
191
00:18:22.080 --> 00:18:30.160
closely with one of my data, the snowflakes and the Databricks of the world, right? Or I'm going
192
00:18:30.160 --> 00:18:38.400
to make up that on one of the big SaaS providers, the service, now's, the sales forces,
193
00:18:39.360 --> 00:18:44.400
because by the way, they all have the identical strategy. I'm not here to tell you who's going
194
00:18:44.400 --> 00:18:53.120
to win, right? I don't know. But I think you have to be very thoughtful. And what's unfortunate
195
00:18:53.840 --> 00:18:59.920
is picking three or four options is a bad option, I think. I think you actually have to pick
196
00:19:00.960 --> 00:19:07.600
a provider that you think, even if they're not even if not the A plus player right now,
197
00:19:08.400 --> 00:19:14.640
even if maybe the B or B minus player, that you think that they've got the long term durability,
198
00:19:14.640 --> 00:19:20.880
and they're going to be around in five years. I would much rather pick a perceived legacy player
199
00:19:20.880 --> 00:19:25.200
that I know is going to exist in five years than the single greatest
200
00:19:26.240 --> 00:19:33.040
agentic solution in the marketplace, because one, I don't, I'm very hard to evaluate. I mean,
201
00:19:33.040 --> 00:19:36.480
I get calls all the time, they're like, my thing's the best thing in the world. I mean, I get that
202
00:19:36.480 --> 00:19:41.840
three times a day. It would take me a month to figure out if your thing is the best thing in the world.
203
00:19:41.840 --> 00:19:45.840
I have no idea. And I don't have the time. Well, that was a question before you jump ahead. Like,
204
00:19:45.840 --> 00:19:50.000
let's say it is the best thing in the world. How hard is it when you have an engineer that can do
205
00:19:50.000 --> 00:19:56.160
500,000 lines of code a month for somebody to copy at at this point? If you're a multi-billion dollar
206
00:19:56.160 --> 00:20:01.040
company, yes, I'm not so sure if you're a hundred million dollar company that you really want to be
207
00:20:01.120 --> 00:20:07.040
building, you know, agentic frameworks for yourself. I think where you want to build potentially.
208
00:20:07.040 --> 00:20:12.320
This is an important point. Like, I would, I would, I would recommend to people to buy an
209
00:20:12.320 --> 00:20:17.040
agentic infrastructure. And obviously, they're not going to, I mean, people, I think, I think we're
210
00:20:17.040 --> 00:20:21.200
beyond the point where people will argue that they're going to build their own models themselves. I mean,
211
00:20:21.200 --> 00:20:25.280
I think that's kind of, I mean, there's certain use cases, but putting aside a few of them, for the
212
00:20:25.280 --> 00:20:29.360
most part, I think you can use right now, assuming pricing stays the same. You can use any of the
213
00:20:29.360 --> 00:20:34.240
biggest players. I would similarly suggest that you buy an agentic framework because you're going
214
00:20:34.240 --> 00:20:39.360
to get all of these controls. You're going to get a monitoring framework. Sure, you could build that,
215
00:20:39.360 --> 00:20:44.320
but I don't think that's what you want to build, where I think if I'm a developer, I want to think
216
00:20:44.320 --> 00:20:50.880
about that application layer. And I want to think about building the best client intelligence tool,
217
00:20:50.880 --> 00:20:59.360
the best sales and trading tool, the best, you know, investment banking presentation analysis tool.
218
00:20:59.360 --> 00:21:06.720
Right. I would focus on using your proprietary knowledge and expertise and your proprietary data.
219
00:21:07.680 --> 00:21:14.320
And then leverage a generic model infrastructure and a generic agentic. So that, that sort of
220
00:21:14.320 --> 00:21:19.520
three, four and five spot, the governance controls, the models and the agentic layer. I'd personally,
221
00:21:19.520 --> 00:21:23.520
even on a large firm, I'd probably buy it because it's so commoditized.
222
00:21:23.520 --> 00:21:27.200
When you say agentic layer, just to be clear, like, what are some of those companies? Let's just
223
00:21:27.200 --> 00:21:33.120
make sure everyone understands what those are. I mean, Microsoft is building out a three tiered
224
00:21:34.720 --> 00:21:42.720
agentic layer, companies like Manus, in an agentic orchestration layer, service now used to be
225
00:21:42.720 --> 00:21:49.760
a firm's company, they're becoming an agentic sales force is now becoming. And what I guess,
226
00:21:49.760 --> 00:21:53.600
let me be arguably any big SaaS player at this point. Anyway, they're all becoming, because what,
227
00:21:53.600 --> 00:21:58.720
think about what what I would describe is, okay, we have agents, agents do not just, they don't
228
00:21:58.720 --> 00:22:08.880
just think they act, but an agentic layer is a firm wide AI orchestration capability that is able
229
00:22:08.880 --> 00:22:16.640
to not only manage work within a vertical, but is able to manage work across horizontally
230
00:22:16.640 --> 00:22:22.720
across an organization. Think about that. And the reason I talk about it as a thing, as opposed
231
00:22:22.720 --> 00:22:31.120
to just agents, is if you're the CEO of a multinational corporation, the value that your organization
232
00:22:31.120 --> 00:22:36.320
brings is the intellectual capital that's sitting in little pockets across the organization.
233
00:22:36.880 --> 00:22:42.080
And if you create an orchestration layer that really understands the work that people do,
234
00:22:42.720 --> 00:22:47.440
you will unlock value that you don't even know exists, because you will essentially be able to
235
00:22:47.440 --> 00:22:52.880
start to bring the intellectual capital and the individual decision making of people to bear
236
00:22:52.880 --> 00:22:58.480
across. I mean, you could have agents that could be working 24 hours a seven days a week solving
237
00:22:58.480 --> 00:23:05.440
problems for you with resources outside of your four walls of your business area in 50 other
238
00:23:05.440 --> 00:23:10.640
countries around the world completely independent of you knowing any single person in that group.
239
00:23:10.640 --> 00:23:14.800
Like, that's that's what I'm talking about in an agentic orchestration layer. And I'm not even
240
00:23:14.800 --> 00:23:18.240
talking just about the work. I'm talking about like the intellectual capital, but I'm the work
241
00:23:18.240 --> 00:23:26.560
matters too. So if I'm a hundred five hundred million dollar business, even I get that in practical
242
00:23:26.560 --> 00:23:33.520
terms, if I'm a multi multi billion dollar corporation, I'm sure that there's a head of customer
243
00:23:33.520 --> 00:23:37.360
success. He says, I really want this solution. And as a head of something that says, I really want
244
00:23:37.360 --> 00:23:44.400
this solution. So is that is that vision even possible in the way actual human nature is operating
245
00:23:44.400 --> 00:23:49.200
in these companies? I think now we're talking about why leadership matters so much here. There are
246
00:23:49.200 --> 00:23:56.720
some CEOs that I talk to that understand this and understand that this is an existential opportunity
247
00:23:56.800 --> 00:24:04.000
for their business. Those CEOs are very switched on. And they really understand this vision.
248
00:24:04.000 --> 00:24:10.720
And they are aggressively pushing this concept of an orchestration framework for their organization.
249
00:24:10.720 --> 00:24:15.120
And canily the one this I then I feel very strongly the ones that are able to do this and the ones
250
00:24:15.120 --> 00:24:22.960
that are able to not allow that one special interest to buy the one best CRM system that's in
251
00:24:22.960 --> 00:24:30.720
the marketplace. I mean canily give me give me the B solutions across a hundred different things
252
00:24:30.720 --> 00:24:36.160
that's on the same agenda framework than the A plus solutions that are not a hundred times out
253
00:24:36.160 --> 00:24:42.400
of a hundred because it's the connectivity. And the other point I'll make Keith, there's the
254
00:24:42.400 --> 00:24:50.080
analogy and I apologize from me myself, you know, the world has given us a Ferrari and most of us
255
00:24:50.080 --> 00:24:54.640
are driving circles in a Walmart parking lot. And we don't even have to turn the air conditioner on
256
00:24:55.440 --> 00:25:00.640
and don't know how to play the radio, right? Like everybody wants the best solution.
257
00:25:01.680 --> 00:25:08.880
The most mediocre solution with a good organization that has good frameworks in place and a decent
258
00:25:08.880 --> 00:25:15.360
data set will crush it a hundred percent of the time. You don't need the best model. You don't
259
00:25:15.360 --> 00:25:19.760
need the best agenda framework. What you need is you need smart people who know how to deliver this.
260
00:25:20.720 --> 00:25:25.840
And my point is is that it's more of an organizational management to your point to these issues
261
00:25:25.840 --> 00:25:30.480
of like, well, I want my tool. No, I want my tool. And the ability to, by the way, not only do that,
262
00:25:30.480 --> 00:25:36.400
but also to share capabilities, right? Like you don't need more than one document processing tool
263
00:25:37.200 --> 00:25:41.440
and an organization. You should pick one. Pick the best one you can find. Don't care. And then
264
00:25:41.440 --> 00:25:46.720
make everybody use it as opposed to letting all the businesses. Now, is that the nature of large
265
00:25:46.720 --> 00:25:51.280
multi-billion dollar corporations? Absolutely not. And I think that's the challenge, but that's
266
00:25:51.280 --> 00:25:56.400
where again, that's where I think the CEO matters in this equation so much because if he or she
267
00:25:57.040 --> 00:26:01.280
is really thoughtful about those decisions, I think they're going to have a massive advantage.
268
00:26:01.280 --> 00:26:07.600
I think what you're implicitly saying, if not explicitly, is the solution itself may or may
269
00:26:07.600 --> 00:26:15.120
or may not matter as much as the approach. And I find imagining, though, your CEO, you're
270
00:26:15.120 --> 00:26:20.320
on an investor call, your board's calling, they want to see some action. They want to see movement.
271
00:26:20.320 --> 00:26:25.600
Is the advice, go move, but don't move on 20 different fronts, move with a single partner?
272
00:26:25.600 --> 00:26:30.640
I think, first of all, you have to believe in what I'm saying. You have to believe that a long-term
273
00:26:31.360 --> 00:26:35.280
consolidated orchestration framework with these components matters for your business. I think,
274
00:26:35.280 --> 00:26:40.560
first of all, you have to acknowledge it. Now, you can't wait five years for the mall to be built
275
00:26:40.560 --> 00:26:45.600
and everything. I mean, you got to deliver value. And I think so long as you are doing that
276
00:26:46.320 --> 00:26:50.880
in the context of a broader framework, and if you're going to take shortcuts,
277
00:26:51.840 --> 00:26:56.960
sometimes you will, then you just do it knowing that you're taking a shortcut and there might be
278
00:26:56.960 --> 00:27:02.080
the risk of having to rebuild something or rewire. So I'm not suggesting you wait. And by the way,
279
00:27:02.080 --> 00:27:07.920
I would actually suggest the opposite. I think, you know, if I'm a CEO of a company, I tell them,
280
00:27:07.920 --> 00:27:14.560
take your five biggest, most important strategic initiatives and get in a room and ask yourself,
281
00:27:14.560 --> 00:27:22.240
one, how can we make these things better faster, cheaper? Number two, if in five years we're blown
282
00:27:22.240 --> 00:27:26.160
out, how are we going to get blown out and what are we going to do about it? And then finally,
283
00:27:26.960 --> 00:27:30.960
what could we do tomorrow with GNI that we're not doing today that would improve our whole business
284
00:27:30.960 --> 00:27:35.440
model? I would ask those questions and I'd walk away with five projects and I would just execute
285
00:27:35.440 --> 00:27:39.840
relentlessly. I take my best people out of their day jobs. I get my best technologist.
286
00:27:40.800 --> 00:27:47.120
I would be personally involved if I was a CEO and I would look to drive real material KPI.
287
00:27:47.120 --> 00:27:51.840
And I would just do that in the context of the framework that we're talking about and knowing that
288
00:27:51.840 --> 00:27:55.040
things are going to evolve, there's going to be products that are wouldn't lose. So I don't think
289
00:27:55.040 --> 00:27:59.120
it's one or the other. I think you just have to do it and do it thoughtfully. As opposed to just
290
00:27:59.120 --> 00:28:04.560
letting your organization buy a bunch of products that won't work in three years together because
291
00:28:04.560 --> 00:28:13.200
you haven't been strategic. In 2018, you launched the first AI ML product. And then in
292
00:28:13.200 --> 00:28:22.160
22, you started partnering with OpenAI. What metric did you look at after that 2018
293
00:28:22.960 --> 00:28:28.080
implementation to say, this is a success. And then as you got deeper and deeper in over the last
294
00:28:28.080 --> 00:28:31.440
four years, what were you looking at? You know, listen, everyone's got a different,
295
00:28:31.520 --> 00:28:36.080
everyone's got a different objective, right? In the wealth business, you know, and if you go back
296
00:28:36.080 --> 00:28:40.480
to 15 and 16 when there was a perception that the robot advisors were going to essentially
297
00:28:40.480 --> 00:28:46.640
eliminate all of wealth management. And by the way, we intuitively felt that wasn't the case,
298
00:28:46.640 --> 00:28:51.040
but we didn't have any data. So I had done a lot of work. And by the way, what I'm sharing is all
299
00:28:51.040 --> 00:28:58.880
public information from prior. So I can certainly share it with you. You know, what we found is that
300
00:28:59.120 --> 00:29:06.240
we looked at 300 plus attributes on a multivariate model. And the only thing that really mattered
301
00:29:06.240 --> 00:29:12.720
for success for a happy customer or a successful financial advisor was contact. I mean, I used to
302
00:29:12.720 --> 00:29:17.840
jump with the FAs that it took me three years and 10 PhDs to figure out if you called your customers,
303
00:29:17.840 --> 00:29:23.680
you'd make more money than if you didn't. It's true. And if I walk into any branch,
304
00:29:24.320 --> 00:29:29.840
by this is true for any company. And if I can, if I can see the financial advisors that have
305
00:29:29.840 --> 00:29:34.960
the most outbound phone calls per week, I guarantee six or seven and we're going to have the greatest
306
00:29:34.960 --> 00:29:42.720
and that acquired asset revenue. Because even in 2018, and by the way, in 2026, maybe, you know,
307
00:29:42.720 --> 00:29:47.360
what happened to 28 or 29, I don't know, but today, the most valuable thing that financial advisors
308
00:29:47.360 --> 00:29:52.720
can do in any business is call their clients. But the problem is that they're way down with so many
309
00:29:52.720 --> 00:29:57.600
other things, right? They're managing portfolios. They're moving money. They're doing planning,
310
00:29:57.600 --> 00:30:04.080
right? So one, the strategy is how do you free up capacity? But two, how do you give them the
311
00:30:04.080 --> 00:30:09.520
ability to scale? Because when the Dow drops, you know, 400 points, you can't pick up the phone
312
00:30:09.520 --> 00:30:16.000
and call every client. So you create a digital tool that increases their ability to velocity. And
313
00:30:16.000 --> 00:30:21.840
then what's the measurement? The measurement is one is velocity of contact increasing,
314
00:30:22.720 --> 00:30:29.680
right? Both human and digital. And two, is that driving net acquired asset? Because it's not
315
00:30:29.680 --> 00:30:34.160
about revenue and wealth. It's, I mean, revenue comes and it's not about whether you buy a bond or
316
00:30:34.160 --> 00:30:38.160
sell a bond like that's not where people make money and financial advice. They, they make money by
317
00:30:38.160 --> 00:30:43.360
you staying with them of a long period of time and putting more money in, right? As you grow. And
318
00:30:44.080 --> 00:30:48.880
the only thing that matters is contact. And if because contact matters, then we measure contact.
319
00:30:48.880 --> 00:30:53.520
And if you do that contact, then you'll basically drive more, more assets. So that's really was the
320
00:30:53.520 --> 00:30:58.320
key metric for next best action. And in terms of the other ones, I mean, some of them are efficiency
321
00:30:58.320 --> 00:31:04.560
plays. You know, are you doing a unit of code faster? And then is the question, do you use that
322
00:31:04.560 --> 00:31:13.200
unit to reinvest in more software development or not? Some of them are about how many PowerPoint
323
00:31:13.200 --> 00:31:18.800
or just kind of cash flow models are you're able to do in a day? And is that creasing? And again,
324
00:31:18.800 --> 00:31:26.880
do you use that capacity to develop more solutions for clients or to create more efficiency? A lot of
325
00:31:26.880 --> 00:31:31.600
it's about risk. I don't think that gets enough play. I mean, one thing's great about AI is it
326
00:31:31.600 --> 00:31:38.080
reduces, it reduces the standard deviation of quality and organization and allows you to monitor
327
00:31:38.080 --> 00:31:43.600
stuff at scale. And then finally, which is a part of play is just revenue. Are you producing a new
328
00:31:43.600 --> 00:31:48.480
product, a new service that you weren't producing before? And does the introduction of that service
329
00:31:48.480 --> 00:31:53.120
create or product create incremental revenue for our business? But all these things are measurable,
330
00:31:53.120 --> 00:31:57.440
but can we most organizations really struggle with this kind of stuff? It's like, oh, it got better
331
00:31:57.440 --> 00:32:01.760
and we gave co-pilot everyone and people seemed to like, I mean, that's not really a strategy.
332
00:32:01.760 --> 00:32:07.360
Well, that's interesting because as you were speaking, what I was thinking of is what time frame do
333
00:32:07.360 --> 00:32:12.160
you have to set realistically internally? Because I think a lot of the feedback I get from people
334
00:32:12.160 --> 00:32:18.960
is they did this project. The CEO is looking for accountability, which is something I know you
335
00:32:18.960 --> 00:32:23.600
believe in. And that could be on the cost side to reduce costs. It could be how is it impacting
336
00:32:23.600 --> 00:32:29.360
the business? But there's a time frame by which things have to bake. For sure. And by the way,
337
00:32:29.360 --> 00:32:34.880
I actually have a slide on this that I do. I do, I teach a class for executives and I talk about
338
00:32:34.880 --> 00:32:41.440
it. I think for most organizations, like year one is purely getting your feet underneath you,
339
00:32:41.440 --> 00:32:47.040
giving an enterprise tool to everybody, developing your guardrails and policies, teaching people how
340
00:32:47.040 --> 00:32:55.280
to prompt, and probably getting one or two projects done. That's year one. Year two is then saying
341
00:32:55.280 --> 00:32:59.120
to everybody, okay, every business now that you don't want to have a use of tool on what you do,
342
00:32:59.120 --> 00:33:04.240
a project. So maybe in year two, you get 20 or 30. And then maybe by year three, you're sort of
343
00:33:04.240 --> 00:33:09.600
up and running with material. And because the thing is, it's, it's almost impossible for me to come
344
00:33:09.600 --> 00:33:13.680
to somebody who's never used AI and said, I want you to deploy something. I want you to deploy
345
00:33:13.680 --> 00:33:18.880
it effectively. And I want to know what your KPIs are. Like most people can't even spell AI.
346
00:33:18.880 --> 00:33:24.640
So to say you want material impacting like year one, I think it's like, you know, it's like asking
347
00:33:24.640 --> 00:33:30.720
your, it's asking your, your 10 year old to go dunk a ball, right? They just don't have the skills
348
00:33:30.800 --> 00:33:35.760
yet to do that. They need to mature. So I think people really do need to take a much longer time
349
00:33:35.760 --> 00:33:40.720
frame. And I think it is sort of in the three plus years before, I mean, we were at it for four,
350
00:33:41.680 --> 00:33:45.040
right? And, and you know, I would, again, I don't, I don't work through anymore, but I would like
351
00:33:45.040 --> 00:33:49.280
to think Morgan Stanley is in a good spot. But it didn't happen overnight. And it takes,
352
00:33:49.280 --> 00:33:52.960
and by the way, the biggest challenge is not even tactical. It's just organizational,
353
00:33:52.960 --> 00:33:57.520
educational awareness and training. And how much, if you're looking at a percent of a budget,
354
00:33:57.600 --> 00:34:02.640
how much budget or time would you put behind that? Here's how I respond to that question.
355
00:34:02.640 --> 00:34:07.360
I don't think people should be spending any more technology dollars than they're spending today.
356
00:34:07.360 --> 00:34:12.560
I think what they should be doing is they should look at the technology dollars that they're spending.
357
00:34:14.480 --> 00:34:19.440
And they should be asking themselves, with the advent of generative AI, how does it change my
358
00:34:19.440 --> 00:34:24.960
investment? In many ways, GNI is actually cheaper to design and deploy. That being said,
359
00:34:24.960 --> 00:34:30.000
one thing I will say Keith, and this is important, the piece that's missing in most organizations
360
00:34:31.040 --> 00:34:38.400
is that AI, generative AI, is changing the way we design and develop software. So before,
361
00:34:39.200 --> 00:34:43.040
if you were the business guy, and I'm the tech guy, you would write requirements documents,
362
00:34:43.040 --> 00:34:47.440
and then you'd hand them to me, and I write a technical specification and a user specification
363
00:34:47.440 --> 00:34:52.800
document. And then I would hand it to 20 developers. And then six weeks later, we would build it,
364
00:34:52.800 --> 00:34:55.840
and I'd give it back to you, and you would say, Jeff, this is what I wanted, and I'd say,
365
00:34:55.840 --> 00:34:59.280
well, you didn't explain it to me, right? And then we'd circle through for a month or two,
366
00:34:59.280 --> 00:35:04.880
and then you'd run out of budget, and we would just call it a day, right? In 2026,
367
00:35:05.760 --> 00:35:11.120
you could actually, in some cases, you could completely build that solution on yourself,
368
00:35:11.120 --> 00:35:15.840
like completely build it without any of my involvement. Now, maybe you might need my help,
369
00:35:15.840 --> 00:35:20.160
maybe you won't, but let's just assume you could build it. The problem is, most people in the
370
00:35:20.160 --> 00:35:24.480
business don't have actually the skill to discipline to build that appropriately. They don't know how
371
00:35:24.480 --> 00:35:29.680
to prompt. They certainly don't know how to evaluate software solutions, and they don't have
372
00:35:29.680 --> 00:35:35.280
capacity. So what we've done is we shifted this bubble to the business, but we haven't given them
373
00:35:35.280 --> 00:35:41.360
the training and the skills and the knowledge and the oversight to do that effectively. And so when
374
00:35:41.360 --> 00:35:46.560
you talk about budget, I mean, net net is probably the same dollars, but you're probably if you're
375
00:35:46.560 --> 00:35:51.920
really smart, your cost of software development is going down. So what you probably should be doing,
376
00:35:51.920 --> 00:35:56.880
and by the way, maybe not taking a head out at all, but you're shifting the resource capacity
377
00:35:56.880 --> 00:36:01.680
that has historically been developing software in the back, and you're moving those capacities
378
00:36:01.680 --> 00:36:07.040
closer to the business so that they have the knowledge, capabilities, and they're able to build
379
00:36:07.040 --> 00:36:14.240
this stuff in a responsible way. So I totally agree with this idea that people don't know how to
380
00:36:14.240 --> 00:36:20.240
use the tools. And one thing I've discovered from having these conversations, a lot of companies
381
00:36:20.240 --> 00:36:27.920
are very strict on sort of shadow tools and paying for solutions. So how are people, if their
382
00:36:27.920 --> 00:36:32.800
companies aren't supporting them, if they're sort of banning shadow tools, supposed to develop the
383
00:36:32.800 --> 00:36:39.520
knowledge to be able to use these things when presented opportunities? So I tell everyone you have
384
00:36:39.600 --> 00:36:46.320
to give every employee an end user AI tool. And by the way, you're going to reduce your risk.
385
00:36:46.320 --> 00:36:48.800
People are like, oh, I can't give this, they're going to do something stupid. Well, it gets
386
00:36:48.800 --> 00:36:53.840
what they're already using cloud, and they're probably already uploading customer information.
387
00:36:56.240 --> 00:37:00.880
So you want to reduce that risk, and you want them to do it. The other thing is that by putting it
388
00:37:00.880 --> 00:37:07.440
in your infrastructure, you can monitor it. I mean, very simple, take one tool, and then throw
389
00:37:07.440 --> 00:37:16.000
another tool on top of it. Put your code of conduct in place, and say, you are a compliance officer
390
00:37:16.000 --> 00:37:22.080
at XYZ Corporation, and your responsibility is to monitor the prompting behavior of every
391
00:37:22.080 --> 00:37:26.080
employee that uses this system. So then take every input prompt, put every output prompt,
392
00:37:26.080 --> 00:37:29.760
and then dump it into another model and have it tell you what it thinks, right? I mean, this
393
00:37:29.760 --> 00:37:35.600
actually reduces your risk profile. So I think number one, you have to give, but I think the other
394
00:37:35.600 --> 00:37:40.800
thing is that you cannot think that giving people a tool is going to save you much money.
395
00:37:42.240 --> 00:37:49.440
Because what these generative AI tools do is they go out to tasks, and very few of us in this
396
00:37:49.440 --> 00:37:54.480
world do one task all day long. Now, maybe if you're a developer, maybe if you're in a call center,
397
00:37:54.480 --> 00:38:00.560
right? Those are use cases. But most of them actually do 20 or 30 tasks today. So if you take 50
398
00:38:00.560 --> 00:38:07.280
percent of a task that I do eight minutes a day, even if it's across 50,000 people,
399
00:38:07.280 --> 00:38:11.920
you're never going to see that. But my argument is that these end user tools are not because
400
00:38:11.920 --> 00:38:16.720
they're going to save you money, but because they educate your workforce in the effect of use
401
00:38:16.720 --> 00:38:21.280
of these technologies. And then by doing that, that's like what I'm talking about, you're one,
402
00:38:21.280 --> 00:38:24.560
get people using these tools because the other thing that happens is that they start to say,
403
00:38:25.360 --> 00:38:31.920
geez, if I can do this little thing with an agent, you know, in co-pilot or GPT and enterprise GPT,
404
00:38:32.800 --> 00:38:41.200
or a project in Claude, I could, if I only had an API, and if I could wire that API into this service,
405
00:38:42.000 --> 00:38:47.280
and if I could connect to my CRM system in this way, wow, I could do something really interesting.
406
00:38:47.280 --> 00:38:51.920
And it opens up the aperture. And the point is that unless you know how to do this kind of stuff
407
00:38:51.920 --> 00:38:56.960
on your own, it's very hard to even think that way, which is why when I teach executives,
408
00:38:56.960 --> 00:39:02.480
one of the things I do, we spend two hours building. We actually sit in a room designing and
409
00:39:02.480 --> 00:39:06.640
building solutions because that's the only way to effectively learn. And then it kind of opens up
410
00:39:06.640 --> 00:39:13.600
the opportunity. You had one of the earliest and most controversial titles, just a set of AI
411
00:39:13.600 --> 00:39:18.240
title, which people feel, I posted a lot about this on LinkedIn and people feel very strongly
412
00:39:18.320 --> 00:39:26.160
about this title. So one, I'd like to, to hear what you think of that, the importance of a role
413
00:39:26.160 --> 00:39:31.760
like that in most organizations today. And then interestingly, you had this title, which is very
414
00:39:32.560 --> 00:39:38.160
in demand, in my opinion, and you chose to walk away to go do something else. And so what did you see
415
00:39:38.160 --> 00:39:46.080
that made you want to leave? So, so first of all, I've been in Morgan Stanley since 2009. I mean,
416
00:39:46.080 --> 00:39:53.520
it is, it is home. And I worked for a gentleman named Andy Saperstein, who's the co-president there,
417
00:39:53.520 --> 00:40:02.400
and Candleley is not only was a boss, a mentor, and a friend. I mean, I love Morgan Stanley,
418
00:40:02.400 --> 00:40:07.440
and I love the people there, and have only the utmost respect. I mean, I really mean that, and there's
419
00:40:07.440 --> 00:40:13.520
no hyperbole there. But I would say two things. Number one, when I took the job, and I said this
420
00:40:13.520 --> 00:40:19.040
publicly, I said, when I took the job, if I did the job right, I would be out of a job in two years.
421
00:40:19.840 --> 00:40:26.560
And you know, part of the controversy of these roles is, is they're not permanent roles?
422
00:40:27.440 --> 00:40:33.920
They are temporary roles. They are part, they are, because you need, because people will argue,
423
00:40:33.920 --> 00:40:37.520
they'll say, some people say, well, we have to centralize everything because we don't centralize
424
00:40:37.520 --> 00:40:42.240
everything. People are going to get out of control. And then the business say, no, no, no, we need to
425
00:40:42.240 --> 00:40:47.360
do whatever we want to do. And the answer is yes, you need to do both. You need to have somebody
426
00:40:47.360 --> 00:40:52.960
that is able to put your infrastructure in place, set the standards, establish the guard rails,
427
00:40:53.760 --> 00:40:59.440
put your evaluation frameworks, figure out what it means to like, how do I, how do I submit a
428
00:40:59.440 --> 00:41:04.480
project? What are the steps? Who approves it? What's the, you know, all that kind of stuff? And then
429
00:41:04.480 --> 00:41:08.480
most importantly, you need someone to teach the organization. And then once that's done, you need to
430
00:41:09.280 --> 00:41:14.160
give the, give the organization the ability to add value. So I think number one, I saw myself,
431
00:41:15.280 --> 00:41:20.480
can they work myself at a job? And by the way, I'm, I'm, couldn't be proud of, right? I wasn't trying
432
00:41:20.480 --> 00:41:26.480
to, I wasn't trying to hang on too long. And I'm very proud of where Morgan Stanley is and
433
00:41:26.480 --> 00:41:30.080
the work that we did. And by the way, they're in great shape, great shape. And they're going to,
434
00:41:30.080 --> 00:41:34.560
I think they're going to do great things because the technology is diffused to the right people
435
00:41:34.560 --> 00:41:42.000
in the business to do that. But secondly, I saw an enormous gap in the, the level of awareness
436
00:41:42.000 --> 00:41:46.400
at the senior level. And it's not, you know, it's not about the models, it's even not about the
437
00:41:46.400 --> 00:41:51.120
agenda clear. It's about, do senior executives really understand what this technology can do their
438
00:41:51.120 --> 00:41:56.000
business and drive value? And then lastly, and I don't want to be, I don't want to be too sentimental
439
00:41:56.000 --> 00:42:01.760
here, but I think like this is a once in a generation technology. And how, what a privilege
440
00:42:02.320 --> 00:42:06.880
to be able to be part of the dialogue here, to drive, you know, I mean, I teach a Columbia,
441
00:42:07.920 --> 00:42:13.280
you know, we just had our, we just had our event last week where I had 50 students,
442
00:42:13.280 --> 00:42:17.680
Bill Day, I solutions in three weeks from scratch, not a single one of the technologists, right?
443
00:42:18.320 --> 00:42:22.160
And to be able to sort of be part of this ecosystem where we're thinking about
444
00:42:23.040 --> 00:42:28.320
policy and education, I don't know, what a, what a, what an opportunity. So, you know, for those
445
00:42:28.320 --> 00:42:32.400
reasons, one, my work was done and two, I think there's a lot of work to be done outside.
446
00:42:33.200 --> 00:42:39.360
I decided to, um, to set up my little McMillan AI shop. It's awesome. Thank you so much
447
00:42:39.360 --> 00:42:43.440
for coming on. This was incredible. It can save from having talked to you for a long time now.
448
00:42:43.440 --> 00:42:49.600
Most people should give you a call. Well, keep, you can find me on McMillan AI.com. Is this
449
00:42:49.600 --> 00:42:56.400
like a, is this like an advertisement? McMillan AI.com. And what I would say is there's lots of good
450
00:42:56.400 --> 00:43:02.400
content on there, but there's also a Jeff bot. If you're interested in talking to the AI version
451
00:43:02.400 --> 00:43:10.320
of me, which I love, I don't know, probably four years of content, um, every speech, every framework.
452
00:43:10.960 --> 00:43:14.800
By the way, Keith, you should ask all these questions to Jeff bot and see if you get a better answer.
453
00:43:14.800 --> 00:43:18.800
You probably will. I think that's awesome. I think I will. That's great.
1
00:00:00.000 --> 00:00:04.000
Jeff McMillan, the former head of AI at Morgan Stanley.
2
00:00:04.000 --> 00:00:08.640
Jeff believes we are at the tipping point where companies are either going to learn how to
3
00:00:09.280 --> 00:00:15.200
adopt AI and they're going to thrive or they're basically going to be on a slow death march
4
00:00:15.200 --> 00:00:19.440
because of their inability to adapt. What we've done is we've distilled down
5
00:00:19.440 --> 00:00:25.840
some of his best ideas from the conversation and talked about why he was so excited about what
6
00:00:25.840 --> 00:00:31.840
was happening that he actually left Morgan Stanley to go work with other companies on helping them
7
00:00:31.840 --> 00:00:33.840
make this transition.
8
00:00:41.200 --> 00:00:48.080
You've got the most unlikely AI background in a lot of ways. You graduated West Point,
9
00:00:48.080 --> 00:00:51.280
you worked in the Army, I have a degree in economics if I'm not mistaken,
10
00:00:52.080 --> 00:00:56.480
and that led you to running AI at Morgan Stanley.
11
00:00:56.480 --> 00:01:00.480
If you don't mind just giving us a minute or two on that journey, it would be great just to
12
00:01:00.480 --> 00:01:03.680
orient everybody. So first of all Keith, thanks for having me.
13
00:01:04.400 --> 00:01:07.520
And second of all, I have no idea how I wound up where I am right now.
14
00:01:09.040 --> 00:01:13.440
As you mentioned, I was a young lieutenant who had gotten out of the military.
15
00:01:14.160 --> 00:01:18.640
And if you're familiar with veterans who are transitioning, there's lots of skills and
16
00:01:18.640 --> 00:01:22.800
leadership that you get from military, but you don't really know how to do much of anything.
17
00:01:22.800 --> 00:01:28.320
And I think like many veterans, I struggled. I actually bounced around two or three jobs
18
00:01:28.320 --> 00:01:36.400
that I was neither good at nor enjoyed until I found my first job back at Morgan Stanley,
19
00:01:36.400 --> 00:01:42.720
which was an internal consulting group. And my first project was to clean up the customer
20
00:01:42.720 --> 00:01:48.720
information of Morgan Stanley's customer accounts. And what's funny at the time is I thought I
21
00:01:48.720 --> 00:01:54.720
was actually qualified to do that job when they assigned it to me. Little that I know that
22
00:01:54.720 --> 00:02:02.240
long-term capital would hit about six months later. And my little $2 million project became a
23
00:02:02.240 --> 00:02:07.760
multi-year, multi-million dollar project. And I became, for lack of a better term, an expert in
24
00:02:07.840 --> 00:02:11.920
data quality. And I only mentioned that because, I mean, first of all, that's the stark for me,
25
00:02:11.920 --> 00:02:16.800
but it's really funny because back in the mid-90s, data quality was really important.
26
00:02:17.600 --> 00:02:23.520
And guess what? In 2026, data quality is probably even more important. And I don't think
27
00:02:23.520 --> 00:02:29.120
many organizations have progressed much further than they have in the last 20, 30 years. But
28
00:02:29.120 --> 00:02:35.280
we can talk about that maybe later. You know, my career is sort of a career of the growth
29
00:02:35.280 --> 00:02:40.480
of analytics and data. I worked at Merrill for a while. We used data to understand what clients
30
00:02:40.480 --> 00:02:47.040
were doing and how to drive more value with our research. And then in 2009, I went to work at Morgan
31
00:02:47.040 --> 00:02:53.520
Stanley. I had this very simple idea that we could use data to drive more and affect more
32
00:02:53.520 --> 00:02:58.160
effective targeting to our clients using machine learning algorithms. And this was, you know,
33
00:02:58.160 --> 00:03:03.840
09, so it was sort of the early days of machine learning. And, you know, honestly, that took me
34
00:03:03.840 --> 00:03:10.800
nine years to do. And it wasn't until 2018 that we released the first algorithmically driven
35
00:03:10.800 --> 00:03:16.320
recommendation engine for financial advisors, which we called next best action. You know, I kept
36
00:03:16.320 --> 00:03:22.560
sort of looking back and saying, wow, this, when you put really good content into these algorithms,
37
00:03:22.640 --> 00:03:27.760
and you have really smart employees, in this case, financial advisors, and you give them the
38
00:03:27.760 --> 00:03:33.840
right day at the right time, really good things, really good things happen. And then in March of 22,
39
00:03:33.840 --> 00:03:39.360
so four years ago, almost to the day, myself and a few others leaders at Morgan Stanley were sitting
40
00:03:39.360 --> 00:03:45.520
in a room in Palo Alto, meeting with a bunch of startups. And one of the eight startups that we
41
00:03:45.520 --> 00:03:51.200
met today with a small unknown company called OpenAI, led by at the time this unknown guy named Sam
42
00:03:52.160 --> 00:03:58.240
Altman. And we became the first customer of OpenAI. And we became the first
43
00:03:59.040 --> 00:04:04.880
organization to deploy anything in production. And, you know, what I would say Keith is,
44
00:04:05.680 --> 00:04:11.040
I'm not the smartest engineer in the world. I'm certainly not an engineer at all. But what I've
45
00:04:11.040 --> 00:04:16.480
tried to do, and where I may be, if I've had any, any effect this in career, my career, I sort of
46
00:04:16.480 --> 00:04:21.200
looked at that inflection point between business people trying to solve a problem and
47
00:04:21.200 --> 00:04:25.280
an immersion technologies and connecting those dots in a way that drives business value.
48
00:04:25.280 --> 00:04:30.400
Two years ago, I became the first head of AI at Morgan Stanley, which I served on, you know,
49
00:04:30.400 --> 00:04:37.600
for two years before, before recently leaving the firm to do my own thing. So that's kind of my
50
00:04:37.600 --> 00:04:43.840
life story in five minutes. So just to jump back to 2022, that's interesting. When you became the
51
00:04:43.840 --> 00:04:49.760
first customer, what were they, what were they selling back then? And why did you guys
52
00:04:50.960 --> 00:04:55.040
choose to buy it? I'm not sure they knew what they were selling exactly. I think what they were
53
00:04:55.040 --> 00:05:01.280
looking like many startups, they were looking to develop a partner or partner relationships.
54
00:05:02.560 --> 00:05:10.400
They like many firms were early in their maturity cycle. So, you know, the things that you need to
55
00:05:10.400 --> 00:05:16.640
do to sell to an enterprise customer, you know, entitlements, BYOK, right? All the information
56
00:05:16.640 --> 00:05:21.200
security issues, you know, those were, those were things that, you know, I think they were
57
00:05:21.200 --> 00:05:26.320
grappling with and they were looking for a partner. So I think, I think what were they looking for?
58
00:05:26.320 --> 00:05:30.000
They were looking for a partner. What were they were selling? They were selling GPT-3.
59
00:05:31.360 --> 00:05:35.600
That was the product at the time. And, you know, that was seven months before,
60
00:05:36.560 --> 00:05:43.280
before CHECK GPT hit the, hit the transom. And, you know, we had our CHECK GPT moment,
61
00:05:43.280 --> 00:05:48.400
like back in March, right, four years ago. And, you know, if you think about financial services,
62
00:05:48.400 --> 00:05:54.400
we move a lot of information across a lot of different channels. And it was clear that this technology,
63
00:05:54.400 --> 00:06:01.440
although, again, clearly undefined, uninformed in many ways, would be very helpful. Actually,
64
00:06:01.520 --> 00:06:05.120
the first use case is we spent a lot of time thinking about the first use case. And our first
65
00:06:05.120 --> 00:06:10.480
use case was a simple bot for the financial advisors to help them find, you know, information in real
66
00:06:10.480 --> 00:06:14.880
time as they were talking to clients about any number of a million different topics. Two comments,
67
00:06:14.880 --> 00:06:21.520
one is it ended up being incredibly useful and impactful. And two, it was incredibly challenging
68
00:06:21.520 --> 00:06:27.360
and difficult to actually deploy that, you know, in an enterprise solution, given the nature of
69
00:06:27.440 --> 00:06:31.680
the new technology and just a lack of understanding and the governance and all that other stuff that
70
00:06:31.680 --> 00:06:39.360
was necessary to be successful. Flash forward four years, it's 2026. You just wrote a couple of weeks
71
00:06:39.360 --> 00:06:46.880
ago in article or a framework, I guess, that really breaks down enterprise AI into six layers.
72
00:06:47.760 --> 00:06:55.040
And I'm assuming that that based on the learnings that you had as part of sort of this rollout,
73
00:06:55.040 --> 00:06:58.160
I'm wondering if you don't mind just walking us through those layers for a second,
74
00:06:58.160 --> 00:07:01.040
because it really resonated with me when I read it. It all made a lot of sense.
75
00:07:01.040 --> 00:07:05.600
There were two drivers. One is how do my job is I'm really am trying to just educate people and
76
00:07:05.600 --> 00:07:13.440
raise the literacy of our society, because I think a well educated, responsible group of professionals
77
00:07:13.440 --> 00:07:19.120
in this world will be better fiduciaries of this technology than ones that are not. So my goal is to
78
00:07:19.120 --> 00:07:24.160
hopefully make people smarter to include myself in a world in which this technology is moving very
79
00:07:24.160 --> 00:07:29.600
quickly. That's pointable. And point number two is everyone is focused on two things. They're
80
00:07:29.600 --> 00:07:34.960
focused on vendor or own built solutions, applications, and they're focused on models.
81
00:07:35.600 --> 00:07:41.280
And what in my own personal experience, those are the visible things and they certainly get the hype
82
00:07:41.280 --> 00:07:47.040
that's not where the ultimate long-term value is going to come from. Particularly, if you're only
83
00:07:47.040 --> 00:07:54.000
looking to deploy six bots in your organization, sure go ahead, pick a model or pick two models
84
00:07:54.000 --> 00:07:59.120
and pick three applications and go to work. But what happens when my organization or your organization
85
00:07:59.120 --> 00:08:08.240
has 20 or 200 or 2,000 or 20,000? And I don't believe I'm exaggerating any narrative
86
00:08:08.240 --> 00:08:15.360
reform that in the next 10 years, Fortune 500 firms are going to have tens of thousands of agents
87
00:08:15.440 --> 00:08:20.240
that are going to be talking to each other. And if you believe that to be true, which I fundamentally
88
00:08:20.240 --> 00:08:27.760
believe in the timing we can argue about, you need to have an infrastructure that is scalable
89
00:08:28.640 --> 00:08:34.080
and is able to manage the growth of that infrastructure in a controlled and responsible way.
90
00:08:34.080 --> 00:08:39.440
So what does that mean? That means first and foremost, you need an accessible data layer,
91
00:08:39.440 --> 00:08:45.440
which by the way, I'd probably have done 30 CEO conversations the last two months.
92
00:08:46.640 --> 00:08:52.160
Everyone's data is a problem and it's either a data access problem or it's a data quality problem.
93
00:08:52.160 --> 00:08:56.640
And the point is AI doesn't solve that just by putting it on top. You actually have to do
94
00:08:56.640 --> 00:09:00.400
real stuff to make that work. So there's a, first of all, you got your what I'll call your
95
00:09:00.400 --> 00:09:05.280
foundational layer. Then you have what I'll call your data accessibility later, which is your models,
96
00:09:05.280 --> 00:09:09.440
your rag infrastructure, your knowledge graphs. And again, we could talk a lot about that.
97
00:09:09.440 --> 00:09:14.320
But the point is is that AI does better when the information is structured in a certain way,
98
00:09:15.040 --> 00:09:19.520
just throwing up a bunch of PDFs into a database doesn't necessarily drive value.
99
00:09:20.720 --> 00:09:24.480
The third is what I'll call the governance and control layer, which by the way no one's talking
100
00:09:24.480 --> 00:09:32.320
about. You need a mechanism to entitle the right information, the right people, to essentially
101
00:09:32.560 --> 00:09:39.600
make sure that the model is doing what you want it to do and not doing what you don't want it to do.
102
00:09:39.600 --> 00:09:44.960
And that needs to be very explicit. The fourth is the models, right? And by the way, you can get
103
00:09:44.960 --> 00:09:49.360
one model, two models. Most firms are using more than one right now. But I will tell you,
104
00:09:49.360 --> 00:09:52.640
you're not going to fail because you don't pick the right model. All the models are brilliant.
105
00:09:53.360 --> 00:09:57.840
There's so much more advanced than we as a business star. It's not even worth arguing about.
106
00:09:58.320 --> 00:10:03.440
The fifth layer is the other piece that I don't think we get enough play. And I know we talk about
107
00:10:03.440 --> 00:10:06.960
agents. But what I'd like to talk about is what I'm going to call the
108
00:10:06.960 --> 00:10:14.000
agentic orchestration layer. So if you think about, if you think about the world that we're moving
109
00:10:14.000 --> 00:10:22.400
into, you're going to need this orchestration tool to be able to not only do the individual tasks
110
00:10:22.400 --> 00:10:28.480
that you want the AI to do, but to do those tasks across the organization. And if you have more
111
00:10:28.480 --> 00:10:33.440
than one of these, what I'm calling orchestration layers, you're going to get yourself into a mess
112
00:10:33.440 --> 00:10:38.240
because the compliance tool is not going to talk to the risk tool. And the risk tool is not going
113
00:10:38.240 --> 00:10:44.000
to talk to the financial tool. And we can talk about the right vendors and there's hundreds of
114
00:10:44.000 --> 00:10:48.720
in this space right now. And I'm not even here to sort of argue about that. But what I will say is
115
00:10:49.280 --> 00:10:53.840
that organizations need to be very thoughtful about the angle of agentic layer and building that
116
00:10:53.840 --> 00:10:57.760
in a way that scales. And then finally, you have applications, right? And those applications,
117
00:10:58.320 --> 00:11:02.560
the build versus buy decision is a lot more complicated than used to be. I think people are going
118
00:11:02.560 --> 00:11:09.040
to build a lot more than they used to because the ease of the solutions. But even if you are buying,
119
00:11:09.040 --> 00:11:14.640
you're still going to want the thing that you buy. What are the criterias to fit on that agentic layer?
120
00:11:14.640 --> 00:11:17.520
So those are what I'll call the six layers. And there's actually one what I'm going to call
121
00:11:17.520 --> 00:11:22.480
horizontal layer, which is really your monitoring and evaluation observability, right? You're going to
122
00:11:22.480 --> 00:11:29.680
need a tool that essentially is able to sit across all six of these components and essentially
123
00:11:29.680 --> 00:11:36.960
watch what's going on, being able to not understand when things break, understand how you're going
124
00:11:36.960 --> 00:11:44.000
to fix those things, being able to demonstrate that you know why you got the answers that you got.
125
00:11:44.560 --> 00:11:48.960
And whether that's internal needs or regulatory, I don't really care. You still need to be able to
126
00:11:48.960 --> 00:11:54.240
and be able to defend that your system is operating in a way that is consistent with your
127
00:11:54.240 --> 00:11:59.200
expectations. Now the good news there is that you can actually use AI to look at the AI,
128
00:11:59.200 --> 00:12:03.440
which is a pretty powerful capability. But I think the point when I'm talking to my customers
129
00:12:04.480 --> 00:12:09.280
and you know leaders, I'm really emphasizing this six tiered approach and it's not like you just
130
00:12:09.280 --> 00:12:13.520
go out and buy the whole thing, you know, it's pieces and it's messy and whatever. But I think
131
00:12:13.520 --> 00:12:18.640
has a strategy or organizations really need to think about layers and that horizontal piece and
132
00:12:18.640 --> 00:12:25.760
starting to build that out in a structured way. A lot of focus is on that model layer and the press
133
00:12:25.760 --> 00:12:32.240
certainly likes to talk about the model layer. The application layer ultimately, how important is
134
00:12:32.800 --> 00:12:41.840
to pick a vendor that can scale horizontally because it'll create interoperability? What are your
135
00:12:41.840 --> 00:12:46.080
thoughts on? Okay, we're going to go with a, you know, I think it's Sarah and customer service,
136
00:12:46.080 --> 00:12:51.040
right? And they're going to do it and then they're going to go everywhere or do you think it doesn't
137
00:12:51.040 --> 00:12:57.680
matter for where we are today? Well, so first of all, if you're in the application layer,
138
00:12:57.680 --> 00:13:01.520
you're a vendor in the application layer, you're trying to move down the stack. So you're
139
00:13:01.520 --> 00:13:08.320
already built out your agentry framework. If you are one of the data providers, you're trying to
140
00:13:08.320 --> 00:13:12.800
move up the stack and creating an attempt at what? And if you're a cloud services company, you're
141
00:13:12.800 --> 00:13:16.400
trying to do all of it. And that's right. If you're a cloud services company, you're trying to do
142
00:13:16.400 --> 00:13:24.960
all of it. So which by the way, if I'm advising a firm, I'm, I'm, I'm, I am going to tell them to do
143
00:13:24.960 --> 00:13:33.440
that because I don't know how else you're going to compete unless, unless I think what will likely
144
00:13:33.440 --> 00:13:38.400
happen is that there will be n number of models. You're not going to have that many. You're going
145
00:13:38.400 --> 00:13:44.000
to have n number of agentic orchestration layers. No, this could be from Azure, they could be Amazon,
146
00:13:44.000 --> 00:13:49.200
they could be in traffic. I, again, I'm not here to pick the winners. And then what's very likely,
147
00:13:49.200 --> 00:13:55.200
similar to like we have the Apple store and the Google store, I can imagine a world where there's
148
00:13:55.200 --> 00:14:01.200
a handful of agentic players in the space. And then if you are a vendor solution that wants to
149
00:14:01.200 --> 00:14:06.640
build something in a specific vertical, you will just build it in the context of that framework.
150
00:14:06.640 --> 00:14:13.200
So when you come to me as a customer, I'll say, are you, are you certified by the MCP protocol
151
00:14:13.200 --> 00:14:19.920
of XYZ infrastructure? And you will say, yes, I am certified. I meet all the protocol standards
152
00:14:19.920 --> 00:14:24.480
and I work very well in their stack. And if you don't, I won't be able to do business with you.
153
00:14:24.480 --> 00:14:32.480
So I could imagine a world where we have a few number of agentic solutions, a few number of,
154
00:14:32.480 --> 00:14:38.400
of models. And then that there's a lot of mix and match underneath it in terms of the data set
155
00:14:38.480 --> 00:14:44.800
themselves. Maybe the governance gets tied into the agentic frameworks, probably does. But you've
156
00:14:44.800 --> 00:14:50.800
got more flexibility on the applications so long as they are built in a way that's, you know,
157
00:14:50.800 --> 00:14:56.320
structured for that protocol that you've defined. So just for those who aren't familiar, can you
158
00:14:56.320 --> 00:15:04.720
explain the MCP? MCP is model control protocol. It actually came out of Anthropic. And you know,
159
00:15:04.720 --> 00:15:09.120
the way I kind of describe it, if I wrote, actually wrote a piece on this recently as well,
160
00:15:09.120 --> 00:15:14.480
you know, the 1930s, and you can fact check them in this, but there were 27 different gauges of
161
00:15:14.480 --> 00:15:20.640
railways in the United States. So every, by the way, if you want to read about like the AI
162
00:15:20.640 --> 00:15:25.360
Bloom, read about the railways, it's the same, it's almost the same story. It's that the parallels
163
00:15:25.360 --> 00:15:33.600
are actually quite funny in some ways. And it wasn't until industry government regulators got
164
00:15:33.600 --> 00:15:39.280
together and said, this is how wide a railway gauge is going to be. And oh, by the way, we're going
165
00:15:39.280 --> 00:15:42.960
to have time zones in three parts of the country. I mean, at one point, there was like 30 different
166
00:15:42.960 --> 00:15:46.560
time zones, like you could be in Stanford and I could be New York City and it would be, you know,
167
00:15:46.560 --> 00:15:51.920
three hours difference. It wasn't until that, and then they established some safety standards
168
00:15:51.920 --> 00:15:57.600
that you were able to drive to Florida to New York on one trade, right? And that was a massively
169
00:15:57.600 --> 00:16:02.480
disruptive concept because that's when capital formed. And if you think about MCP, think about MCP
170
00:16:02.480 --> 00:16:11.120
as the gauge for AI. It's the language that allows these disparate AI solutions, not just to
171
00:16:11.120 --> 00:16:16.640
talk to each other, but to talk to applications because when you get into an autonomous or semi-autonomous
172
00:16:16.640 --> 00:16:24.640
world, the AI does the thinking, but then it needs to act with machines, databases, applications,
173
00:16:24.640 --> 00:16:34.000
websites to action the work. And think about MCP as the common language that we can all use so
174
00:16:34.000 --> 00:16:42.160
that when I talk to you, I know who you are as an application, I know what you do, I know the
175
00:16:42.160 --> 00:16:48.800
information that you can't provide me, the entitlements that I have access to, maybe even the
176
00:16:48.800 --> 00:16:54.320
financial economics of it over time. And what I'll say is, you know, this is a whole another
177
00:16:54.320 --> 00:17:01.600
conversation, that there is still a tremendous amount of work that needs to be done around these
178
00:17:01.600 --> 00:17:07.600
protocols on an application by application level to make them really effective both within organizations
179
00:17:07.600 --> 00:17:12.640
and outside of organizations. But I think over time you're going to see a closing of standards and
180
00:17:12.640 --> 00:17:17.920
what that comes from governments industries or somebody else. I'm not really sure, but I think
181
00:17:17.920 --> 00:17:22.960
that's going to be super important as we try to really start to scale and commercialize
182
00:17:22.960 --> 00:17:29.520
authentic solutions. You're at this point where you're a leader of a, there's the Morgan
183
00:17:29.520 --> 00:17:36.240
Stanley's of the world and then there's a $150,200 million business and each of those problems
184
00:17:36.240 --> 00:17:40.080
are very different. Sort of take each of those spectrums. If you're looking at this layer,
185
00:17:40.960 --> 00:17:45.760
then you assume based on what you've said, the data, you have to start somewhere foundationally
186
00:17:45.760 --> 00:17:53.200
with data. Where do you go next? Right now, there are no proven players that will work across
187
00:17:53.200 --> 00:18:00.880
these seven stack, right? Now, there's lots of startups with no customers that have built these
188
00:18:00.880 --> 00:18:07.280
solutions. So what do you do? You, you work with the hyper scalers that are in place today,
189
00:18:08.400 --> 00:18:12.640
who are bringing it together, but bringing it together slowly, the Microsoft, to the world,
190
00:18:13.360 --> 00:18:22.080
Google or AWS. If I'm a $100 million business, that's probably what I'm doing. Or I'm working very
191
00:18:22.080 --> 00:18:30.160
closely with one of my data, the snowflakes and the Databricks of the world, right? Or I'm going
192
00:18:30.160 --> 00:18:38.400
to make up that on one of the big SaaS providers, the service, now's, the sales forces,
193
00:18:39.360 --> 00:18:44.400
because by the way, they all have the identical strategy. I'm not here to tell you who's going
194
00:18:44.400 --> 00:18:53.120
to win, right? I don't know. But I think you have to be very thoughtful. And what's unfortunate
195
00:18:53.840 --> 00:18:59.920
is picking three or four options is a bad option, I think. I think you actually have to pick
196
00:19:00.960 --> 00:19:07.600
a provider that you think, even if they're not even if not the A plus player right now,
197
00:19:08.400 --> 00:19:14.640
even if maybe the B or B minus player, that you think that they've got the long term durability,
198
00:19:14.640 --> 00:19:20.880
and they're going to be around in five years. I would much rather pick a perceived legacy player
199
00:19:20.880 --> 00:19:25.200
that I know is going to exist in five years than the single greatest
200
00:19:26.240 --> 00:19:33.040
agentic solution in the marketplace, because one, I don't, I'm very hard to evaluate. I mean,
201
00:19:33.040 --> 00:19:36.480
I get calls all the time, they're like, my thing's the best thing in the world. I mean, I get that
202
00:19:36.480 --> 00:19:41.840
three times a day. It would take me a month to figure out if your thing is the best thing in the world.
203
00:19:41.840 --> 00:19:45.840
I have no idea. And I don't have the time. Well, that was a question before you jump ahead. Like,
204
00:19:45.840 --> 00:19:50.000
let's say it is the best thing in the world. How hard is it when you have an engineer that can do
205
00:19:50.000 --> 00:19:56.160
500,000 lines of code a month for somebody to copy at at this point? If you're a multi-billion dollar
206
00:19:56.160 --> 00:20:01.040
company, yes, I'm not so sure if you're a hundred million dollar company that you really want to be
207
00:20:01.120 --> 00:20:07.040
building, you know, agentic frameworks for yourself. I think where you want to build potentially.
208
00:20:07.040 --> 00:20:12.320
This is an important point. Like, I would, I would, I would recommend to people to buy an
209
00:20:12.320 --> 00:20:17.040
agentic infrastructure. And obviously, they're not going to, I mean, people, I think, I think we're
210
00:20:17.040 --> 00:20:21.200
beyond the point where people will argue that they're going to build their own models themselves. I mean,
211
00:20:21.200 --> 00:20:25.280
I think that's kind of, I mean, there's certain use cases, but putting aside a few of them, for the
212
00:20:25.280 --> 00:20:29.360
most part, I think you can use right now, assuming pricing stays the same. You can use any of the
213
00:20:29.360 --> 00:20:34.240
biggest players. I would similarly suggest that you buy an agentic framework because you're going
214
00:20:34.240 --> 00:20:39.360
to get all of these controls. You're going to get a monitoring framework. Sure, you could build that,
215
00:20:39.360 --> 00:20:44.320
but I don't think that's what you want to build, where I think if I'm a developer, I want to think
216
00:20:44.320 --> 00:20:50.880
about that application layer. And I want to think about building the best client intelligence tool,
217
00:20:50.880 --> 00:20:59.360
the best sales and trading tool, the best, you know, investment banking presentation analysis tool.
218
00:20:59.360 --> 00:21:06.720
Right. I would focus on using your proprietary knowledge and expertise and your proprietary data.
219
00:21:07.680 --> 00:21:14.320
And then leverage a generic model infrastructure and a generic agentic. So that, that sort of
220
00:21:14.320 --> 00:21:19.520
three, four and five spot, the governance controls, the models and the agentic layer. I'd personally,
221
00:21:19.520 --> 00:21:23.520
even on a large firm, I'd probably buy it because it's so commoditized.
222
00:21:23.520 --> 00:21:27.200
When you say agentic layer, just to be clear, like, what are some of those companies? Let's just
223
00:21:27.200 --> 00:21:33.120
make sure everyone understands what those are. I mean, Microsoft is building out a three tiered
224
00:21:34.720 --> 00:21:42.720
agentic layer, companies like Manus, in an agentic orchestration layer, service now used to be
225
00:21:42.720 --> 00:21:49.760
a firm's company, they're becoming an agentic sales force is now becoming. And what I guess,
226
00:21:49.760 --> 00:21:53.600
let me be arguably any big SaaS player at this point. Anyway, they're all becoming, because what,
227
00:21:53.600 --> 00:21:58.720
think about what what I would describe is, okay, we have agents, agents do not just, they don't
228
00:21:58.720 --> 00:22:08.880
just think they act, but an agentic layer is a firm wide AI orchestration capability that is able
229
00:22:08.880 --> 00:22:16.640
to not only manage work within a vertical, but is able to manage work across horizontally
230
00:22:16.640 --> 00:22:22.720
across an organization. Think about that. And the reason I talk about it as a thing, as opposed
231
00:22:22.720 --> 00:22:31.120
to just agents, is if you're the CEO of a multinational corporation, the value that your organization
232
00:22:31.120 --> 00:22:36.320
brings is the intellectual capital that's sitting in little pockets across the organization.
233
00:22:36.880 --> 00:22:42.080
And if you create an orchestration layer that really understands the work that people do,
234
00:22:42.720 --> 00:22:47.440
you will unlock value that you don't even know exists, because you will essentially be able to
235
00:22:47.440 --> 00:22:52.880
start to bring the intellectual capital and the individual decision making of people to bear
236
00:22:52.880 --> 00:22:58.480
across. I mean, you could have agents that could be working 24 hours a seven days a week solving
237
00:22:58.480 --> 00:23:05.440
problems for you with resources outside of your four walls of your business area in 50 other
238
00:23:05.440 --> 00:23:10.640
countries around the world completely independent of you knowing any single person in that group.
239
00:23:10.640 --> 00:23:14.800
Like, that's that's what I'm talking about in an agentic orchestration layer. And I'm not even
240
00:23:14.800 --> 00:23:18.240
talking just about the work. I'm talking about like the intellectual capital, but I'm the work
241
00:23:18.240 --> 00:23:26.560
matters too. So if I'm a hundred five hundred million dollar business, even I get that in practical
242
00:23:26.560 --> 00:23:33.520
terms, if I'm a multi multi billion dollar corporation, I'm sure that there's a head of customer
243
00:23:33.520 --> 00:23:37.360
success. He says, I really want this solution. And as a head of something that says, I really want
244
00:23:37.360 --> 00:23:44.400
this solution. So is that is that vision even possible in the way actual human nature is operating
245
00:23:44.400 --> 00:23:49.200
in these companies? I think now we're talking about why leadership matters so much here. There are
246
00:23:49.200 --> 00:23:56.720
some CEOs that I talk to that understand this and understand that this is an existential opportunity
247
00:23:56.800 --> 00:24:04.000
for their business. Those CEOs are very switched on. And they really understand this vision.
248
00:24:04.000 --> 00:24:10.720
And they are aggressively pushing this concept of an orchestration framework for their organization.
249
00:24:10.720 --> 00:24:15.120
And canily the one this I then I feel very strongly the ones that are able to do this and the ones
250
00:24:15.120 --> 00:24:22.960
that are able to not allow that one special interest to buy the one best CRM system that's in
251
00:24:22.960 --> 00:24:30.720
the marketplace. I mean canily give me give me the B solutions across a hundred different things
252
00:24:30.720 --> 00:24:36.160
that's on the same agenda framework than the A plus solutions that are not a hundred times out
253
00:24:36.160 --> 00:24:42.400
of a hundred because it's the connectivity. And the other point I'll make Keith, there's the
254
00:24:42.400 --> 00:24:50.080
analogy and I apologize from me myself, you know, the world has given us a Ferrari and most of us
255
00:24:50.080 --> 00:24:54.640
are driving circles in a Walmart parking lot. And we don't even have to turn the air conditioner on
256
00:24:55.440 --> 00:25:00.640
and don't know how to play the radio, right? Like everybody wants the best solution.
257
00:25:01.680 --> 00:25:08.880
The most mediocre solution with a good organization that has good frameworks in place and a decent
258
00:25:08.880 --> 00:25:15.360
data set will crush it a hundred percent of the time. You don't need the best model. You don't
259
00:25:15.360 --> 00:25:19.760
need the best agenda framework. What you need is you need smart people who know how to deliver this.
260
00:25:20.720 --> 00:25:25.840
And my point is is that it's more of an organizational management to your point to these issues
261
00:25:25.840 --> 00:25:30.480
of like, well, I want my tool. No, I want my tool. And the ability to, by the way, not only do that,
262
00:25:30.480 --> 00:25:36.400
but also to share capabilities, right? Like you don't need more than one document processing tool
263
00:25:37.200 --> 00:25:41.440
and an organization. You should pick one. Pick the best one you can find. Don't care. And then
264
00:25:41.440 --> 00:25:46.720
make everybody use it as opposed to letting all the businesses. Now, is that the nature of large
265
00:25:46.720 --> 00:25:51.280
multi-billion dollar corporations? Absolutely not. And I think that's the challenge, but that's
266
00:25:51.280 --> 00:25:56.400
where again, that's where I think the CEO matters in this equation so much because if he or she
267
00:25:57.040 --> 00:26:01.280
is really thoughtful about those decisions, I think they're going to have a massive advantage.
268
00:26:01.280 --> 00:26:07.600
I think what you're implicitly saying, if not explicitly, is the solution itself may or may
269
00:26:07.600 --> 00:26:15.120
or may not matter as much as the approach. And I find imagining, though, your CEO, you're
270
00:26:15.120 --> 00:26:20.320
on an investor call, your board's calling, they want to see some action. They want to see movement.
271
00:26:20.320 --> 00:26:25.600
Is the advice, go move, but don't move on 20 different fronts, move with a single partner?
272
00:26:25.600 --> 00:26:30.640
I think, first of all, you have to believe in what I'm saying. You have to believe that a long-term
273
00:26:31.360 --> 00:26:35.280
consolidated orchestration framework with these components matters for your business. I think,
274
00:26:35.280 --> 00:26:40.560
first of all, you have to acknowledge it. Now, you can't wait five years for the mall to be built
275
00:26:40.560 --> 00:26:45.600
and everything. I mean, you got to deliver value. And I think so long as you are doing that
276
00:26:46.320 --> 00:26:50.880
in the context of a broader framework, and if you're going to take shortcuts,
277
00:26:51.840 --> 00:26:56.960
sometimes you will, then you just do it knowing that you're taking a shortcut and there might be
278
00:26:56.960 --> 00:27:02.080
the risk of having to rebuild something or rewire. So I'm not suggesting you wait. And by the way,
279
00:27:02.080 --> 00:27:07.920
I would actually suggest the opposite. I think, you know, if I'm a CEO of a company, I tell them,
280
00:27:07.920 --> 00:27:14.560
take your five biggest, most important strategic initiatives and get in a room and ask yourself,
281
00:27:14.560 --> 00:27:22.240
one, how can we make these things better faster, cheaper? Number two, if in five years we're blown
282
00:27:22.240 --> 00:27:26.160
out, how are we going to get blown out and what are we going to do about it? And then finally,
283
00:27:26.960 --> 00:27:30.960
what could we do tomorrow with GNI that we're not doing today that would improve our whole business
284
00:27:30.960 --> 00:27:35.440
model? I would ask those questions and I'd walk away with five projects and I would just execute
285
00:27:35.440 --> 00:27:39.840
relentlessly. I take my best people out of their day jobs. I get my best technologist.
286
00:27:40.800 --> 00:27:47.120
I would be personally involved if I was a CEO and I would look to drive real material KPI.
287
00:27:47.120 --> 00:27:51.840
And I would just do that in the context of the framework that we're talking about and knowing that
288
00:27:51.840 --> 00:27:55.040
things are going to evolve, there's going to be products that are wouldn't lose. So I don't think
289
00:27:55.040 --> 00:27:59.120
it's one or the other. I think you just have to do it and do it thoughtfully. As opposed to just
290
00:27:59.120 --> 00:28:04.560
letting your organization buy a bunch of products that won't work in three years together because
291
00:28:04.560 --> 00:28:13.200
you haven't been strategic. In 2018, you launched the first AI ML product. And then in
292
00:28:13.200 --> 00:28:22.160
22, you started partnering with OpenAI. What metric did you look at after that 2018
293
00:28:22.960 --> 00:28:28.080
implementation to say, this is a success. And then as you got deeper and deeper in over the last
294
00:28:28.080 --> 00:28:31.440
four years, what were you looking at? You know, listen, everyone's got a different,
295
00:28:31.520 --> 00:28:36.080
everyone's got a different objective, right? In the wealth business, you know, and if you go back
296
00:28:36.080 --> 00:28:40.480
to 15 and 16 when there was a perception that the robot advisors were going to essentially
297
00:28:40.480 --> 00:28:46.640
eliminate all of wealth management. And by the way, we intuitively felt that wasn't the case,
298
00:28:46.640 --> 00:28:51.040
but we didn't have any data. So I had done a lot of work. And by the way, what I'm sharing is all
299
00:28:51.040 --> 00:28:58.880
public information from prior. So I can certainly share it with you. You know, what we found is that
300
00:28:59.120 --> 00:29:06.240
we looked at 300 plus attributes on a multivariate model. And the only thing that really mattered
301
00:29:06.240 --> 00:29:12.720
for success for a happy customer or a successful financial advisor was contact. I mean, I used to
302
00:29:12.720 --> 00:29:17.840
jump with the FAs that it took me three years and 10 PhDs to figure out if you called your customers,
303
00:29:17.840 --> 00:29:23.680
you'd make more money than if you didn't. It's true. And if I walk into any branch,
304
00:29:24.320 --> 00:29:29.840
by this is true for any company. And if I can, if I can see the financial advisors that have
305
00:29:29.840 --> 00:29:34.960
the most outbound phone calls per week, I guarantee six or seven and we're going to have the greatest
306
00:29:34.960 --> 00:29:42.720
and that acquired asset revenue. Because even in 2018, and by the way, in 2026, maybe, you know,
307
00:29:42.720 --> 00:29:47.360
what happened to 28 or 29, I don't know, but today, the most valuable thing that financial advisors
308
00:29:47.360 --> 00:29:52.720
can do in any business is call their clients. But the problem is that they're way down with so many
309
00:29:52.720 --> 00:29:57.600
other things, right? They're managing portfolios. They're moving money. They're doing planning,
310
00:29:57.600 --> 00:30:04.080
right? So one, the strategy is how do you free up capacity? But two, how do you give them the
311
00:30:04.080 --> 00:30:09.520
ability to scale? Because when the Dow drops, you know, 400 points, you can't pick up the phone
312
00:30:09.520 --> 00:30:16.000
and call every client. So you create a digital tool that increases their ability to velocity. And
313
00:30:16.000 --> 00:30:21.840
then what's the measurement? The measurement is one is velocity of contact increasing,
314
00:30:22.720 --> 00:30:29.680
right? Both human and digital. And two, is that driving net acquired asset? Because it's not
315
00:30:29.680 --> 00:30:34.160
about revenue and wealth. It's, I mean, revenue comes and it's not about whether you buy a bond or
316
00:30:34.160 --> 00:30:38.160
sell a bond like that's not where people make money and financial advice. They, they make money by
317
00:30:38.160 --> 00:30:43.360
you staying with them of a long period of time and putting more money in, right? As you grow. And
318
00:30:44.080 --> 00:30:48.880
the only thing that matters is contact. And if because contact matters, then we measure contact.
319
00:30:48.880 --> 00:30:53.520
And if you do that contact, then you'll basically drive more, more assets. So that's really was the
320
00:30:53.520 --> 00:30:58.320
key metric for next best action. And in terms of the other ones, I mean, some of them are efficiency
321
00:30:58.320 --> 00:31:04.560
plays. You know, are you doing a unit of code faster? And then is the question, do you use that
322
00:31:04.560 --> 00:31:13.200
unit to reinvest in more software development or not? Some of them are about how many PowerPoint
323
00:31:13.200 --> 00:31:18.800
or just kind of cash flow models are you're able to do in a day? And is that creasing? And again,
324
00:31:18.800 --> 00:31:26.880
do you use that capacity to develop more solutions for clients or to create more efficiency? A lot of
325
00:31:26.880 --> 00:31:31.600
it's about risk. I don't think that gets enough play. I mean, one thing's great about AI is it
326
00:31:31.600 --> 00:31:38.080
reduces, it reduces the standard deviation of quality and organization and allows you to monitor
327
00:31:38.080 --> 00:31:43.600
stuff at scale. And then finally, which is a part of play is just revenue. Are you producing a new
328
00:31:43.600 --> 00:31:48.480
product, a new service that you weren't producing before? And does the introduction of that service
329
00:31:48.480 --> 00:31:53.120
create or product create incremental revenue for our business? But all these things are measurable,
330
00:31:53.120 --> 00:31:57.440
but can we most organizations really struggle with this kind of stuff? It's like, oh, it got better
331
00:31:57.440 --> 00:32:01.760
and we gave co-pilot everyone and people seemed to like, I mean, that's not really a strategy.
332
00:32:01.760 --> 00:32:07.360
Well, that's interesting because as you were speaking, what I was thinking of is what time frame do
333
00:32:07.360 --> 00:32:12.160
you have to set realistically internally? Because I think a lot of the feedback I get from people
334
00:32:12.160 --> 00:32:18.960
is they did this project. The CEO is looking for accountability, which is something I know you
335
00:32:18.960 --> 00:32:23.600
believe in. And that could be on the cost side to reduce costs. It could be how is it impacting
336
00:32:23.600 --> 00:32:29.360
the business? But there's a time frame by which things have to bake. For sure. And by the way,
337
00:32:29.360 --> 00:32:34.880
I actually have a slide on this that I do. I do, I teach a class for executives and I talk about
338
00:32:34.880 --> 00:32:41.440
it. I think for most organizations, like year one is purely getting your feet underneath you,
339
00:32:41.440 --> 00:32:47.040
giving an enterprise tool to everybody, developing your guardrails and policies, teaching people how
340
00:32:47.040 --> 00:32:55.280
to prompt, and probably getting one or two projects done. That's year one. Year two is then saying
341
00:32:55.280 --> 00:32:59.120
to everybody, okay, every business now that you don't want to have a use of tool on what you do,
342
00:32:59.120 --> 00:33:04.240
a project. So maybe in year two, you get 20 or 30. And then maybe by year three, you're sort of
343
00:33:04.240 --> 00:33:09.600
up and running with material. And because the thing is, it's, it's almost impossible for me to come
344
00:33:09.600 --> 00:33:13.680
to somebody who's never used AI and said, I want you to deploy something. I want you to deploy
345
00:33:13.680 --> 00:33:18.880
it effectively. And I want to know what your KPIs are. Like most people can't even spell AI.
346
00:33:18.880 --> 00:33:24.640
So to say you want material impacting like year one, I think it's like, you know, it's like asking
347
00:33:24.640 --> 00:33:30.720
your, it's asking your, your 10 year old to go dunk a ball, right? They just don't have the skills
348
00:33:30.800 --> 00:33:35.760
yet to do that. They need to mature. So I think people really do need to take a much longer time
349
00:33:35.760 --> 00:33:40.720
frame. And I think it is sort of in the three plus years before, I mean, we were at it for four,
350
00:33:41.680 --> 00:33:45.040
right? And, and you know, I would, again, I don't, I don't work through anymore, but I would like
351
00:33:45.040 --> 00:33:49.280
to think Morgan Stanley is in a good spot. But it didn't happen overnight. And it takes,
352
00:33:49.280 --> 00:33:52.960
and by the way, the biggest challenge is not even tactical. It's just organizational,
353
00:33:52.960 --> 00:33:57.520
educational awareness and training. And how much, if you're looking at a percent of a budget,
354
00:33:57.600 --> 00:34:02.640
how much budget or time would you put behind that? Here's how I respond to that question.
355
00:34:02.640 --> 00:34:07.360
I don't think people should be spending any more technology dollars than they're spending today.
356
00:34:07.360 --> 00:34:12.560
I think what they should be doing is they should look at the technology dollars that they're spending.
357
00:34:14.480 --> 00:34:19.440
And they should be asking themselves, with the advent of generative AI, how does it change my
358
00:34:19.440 --> 00:34:24.960
investment? In many ways, GNI is actually cheaper to design and deploy. That being said,
359
00:34:24.960 --> 00:34:30.000
one thing I will say Keith, and this is important, the piece that's missing in most organizations
360
00:34:31.040 --> 00:34:38.400
is that AI, generative AI, is changing the way we design and develop software. So before,
361
00:34:39.200 --> 00:34:43.040
if you were the business guy, and I'm the tech guy, you would write requirements documents,
362
00:34:43.040 --> 00:34:47.440
and then you'd hand them to me, and I write a technical specification and a user specification
363
00:34:47.440 --> 00:34:52.800
document. And then I would hand it to 20 developers. And then six weeks later, we would build it,
364
00:34:52.800 --> 00:34:55.840
and I'd give it back to you, and you would say, Jeff, this is what I wanted, and I'd say,
365
00:34:55.840 --> 00:34:59.280
well, you didn't explain it to me, right? And then we'd circle through for a month or two,
366
00:34:59.280 --> 00:35:04.880
and then you'd run out of budget, and we would just call it a day, right? In 2026,
367
00:35:05.760 --> 00:35:11.120
you could actually, in some cases, you could completely build that solution on yourself,
368
00:35:11.120 --> 00:35:15.840
like completely build it without any of my involvement. Now, maybe you might need my help,
369
00:35:15.840 --> 00:35:20.160
maybe you won't, but let's just assume you could build it. The problem is, most people in the
370
00:35:20.160 --> 00:35:24.480
business don't have actually the skill to discipline to build that appropriately. They don't know how
371
00:35:24.480 --> 00:35:29.680
to prompt. They certainly don't know how to evaluate software solutions, and they don't have
372
00:35:29.680 --> 00:35:35.280
capacity. So what we've done is we shifted this bubble to the business, but we haven't given them
373
00:35:35.280 --> 00:35:41.360
the training and the skills and the knowledge and the oversight to do that effectively. And so when
374
00:35:41.360 --> 00:35:46.560
you talk about budget, I mean, net net is probably the same dollars, but you're probably if you're
375
00:35:46.560 --> 00:35:51.920
really smart, your cost of software development is going down. So what you probably should be doing,
376
00:35:51.920 --> 00:35:56.880
and by the way, maybe not taking a head out at all, but you're shifting the resource capacity
377
00:35:56.880 --> 00:36:01.680
that has historically been developing software in the back, and you're moving those capacities
378
00:36:01.680 --> 00:36:07.040
closer to the business so that they have the knowledge, capabilities, and they're able to build
379
00:36:07.040 --> 00:36:14.240
this stuff in a responsible way. So I totally agree with this idea that people don't know how to
380
00:36:14.240 --> 00:36:20.240
use the tools. And one thing I've discovered from having these conversations, a lot of companies
381
00:36:20.240 --> 00:36:27.920
are very strict on sort of shadow tools and paying for solutions. So how are people, if their
382
00:36:27.920 --> 00:36:32.800
companies aren't supporting them, if they're sort of banning shadow tools, supposed to develop the
383
00:36:32.800 --> 00:36:39.520
knowledge to be able to use these things when presented opportunities? So I tell everyone you have
384
00:36:39.600 --> 00:36:46.320
to give every employee an end user AI tool. And by the way, you're going to reduce your risk.
385
00:36:46.320 --> 00:36:48.800
People are like, oh, I can't give this, they're going to do something stupid. Well, it gets
386
00:36:48.800 --> 00:36:53.840
what they're already using cloud, and they're probably already uploading customer information.
387
00:36:56.240 --> 00:37:00.880
So you want to reduce that risk, and you want them to do it. The other thing is that by putting it
388
00:37:00.880 --> 00:37:07.440
in your infrastructure, you can monitor it. I mean, very simple, take one tool, and then throw
389
00:37:07.440 --> 00:37:16.000
another tool on top of it. Put your code of conduct in place, and say, you are a compliance officer
390
00:37:16.000 --> 00:37:22.080
at XYZ Corporation, and your responsibility is to monitor the prompting behavior of every
391
00:37:22.080 --> 00:37:26.080
employee that uses this system. So then take every input prompt, put every output prompt,
392
00:37:26.080 --> 00:37:29.760
and then dump it into another model and have it tell you what it thinks, right? I mean, this
393
00:37:29.760 --> 00:37:35.600
actually reduces your risk profile. So I think number one, you have to give, but I think the other
394
00:37:35.600 --> 00:37:40.800
thing is that you cannot think that giving people a tool is going to save you much money.
395
00:37:42.240 --> 00:37:49.440
Because what these generative AI tools do is they go out to tasks, and very few of us in this
396
00:37:49.440 --> 00:37:54.480
world do one task all day long. Now, maybe if you're a developer, maybe if you're in a call center,
397
00:37:54.480 --> 00:38:00.560
right? Those are use cases. But most of them actually do 20 or 30 tasks today. So if you take 50
398
00:38:00.560 --> 00:38:07.280
percent of a task that I do eight minutes a day, even if it's across 50,000 people,
399
00:38:07.280 --> 00:38:11.920
you're never going to see that. But my argument is that these end user tools are not because
400
00:38:11.920 --> 00:38:16.720
they're going to save you money, but because they educate your workforce in the effect of use
401
00:38:16.720 --> 00:38:21.280
of these technologies. And then by doing that, that's like what I'm talking about, you're one,
402
00:38:21.280 --> 00:38:24.560
get people using these tools because the other thing that happens is that they start to say,
403
00:38:25.360 --> 00:38:31.920
geez, if I can do this little thing with an agent, you know, in co-pilot or GPT and enterprise GPT,
404
00:38:32.800 --> 00:38:41.200
or a project in Claude, I could, if I only had an API, and if I could wire that API into this service,
405
00:38:42.000 --> 00:38:47.280
and if I could connect to my CRM system in this way, wow, I could do something really interesting.
406
00:38:47.280 --> 00:38:51.920
And it opens up the aperture. And the point is that unless you know how to do this kind of stuff
407
00:38:51.920 --> 00:38:56.960
on your own, it's very hard to even think that way, which is why when I teach executives,
408
00:38:56.960 --> 00:39:02.480
one of the things I do, we spend two hours building. We actually sit in a room designing and
409
00:39:02.480 --> 00:39:06.640
building solutions because that's the only way to effectively learn. And then it kind of opens up
410
00:39:06.640 --> 00:39:13.600
the opportunity. You had one of the earliest and most controversial titles, just a set of AI
411
00:39:13.600 --> 00:39:18.240
title, which people feel, I posted a lot about this on LinkedIn and people feel very strongly
412
00:39:18.320 --> 00:39:26.160
about this title. So one, I'd like to, to hear what you think of that, the importance of a role
413
00:39:26.160 --> 00:39:31.760
like that in most organizations today. And then interestingly, you had this title, which is very
414
00:39:32.560 --> 00:39:38.160
in demand, in my opinion, and you chose to walk away to go do something else. And so what did you see
415
00:39:38.160 --> 00:39:46.080
that made you want to leave? So, so first of all, I've been in Morgan Stanley since 2009. I mean,
416
00:39:46.080 --> 00:39:53.520
it is, it is home. And I worked for a gentleman named Andy Saperstein, who's the co-president there,
417
00:39:53.520 --> 00:40:02.400
and Candleley is not only was a boss, a mentor, and a friend. I mean, I love Morgan Stanley,
418
00:40:02.400 --> 00:40:07.440
and I love the people there, and have only the utmost respect. I mean, I really mean that, and there's
419
00:40:07.440 --> 00:40:13.520
no hyperbole there. But I would say two things. Number one, when I took the job, and I said this
420
00:40:13.520 --> 00:40:19.040
publicly, I said, when I took the job, if I did the job right, I would be out of a job in two years.
421
00:40:19.840 --> 00:40:26.560
And you know, part of the controversy of these roles is, is they're not permanent roles?
422
00:40:27.440 --> 00:40:33.920
They are temporary roles. They are part, they are, because you need, because people will argue,
423
00:40:33.920 --> 00:40:37.520
they'll say, some people say, well, we have to centralize everything because we don't centralize
424
00:40:37.520 --> 00:40:42.240
everything. People are going to get out of control. And then the business say, no, no, no, we need to
425
00:40:42.240 --> 00:40:47.360
do whatever we want to do. And the answer is yes, you need to do both. You need to have somebody
426
00:40:47.360 --> 00:40:52.960
that is able to put your infrastructure in place, set the standards, establish the guard rails,
427
00:40:53.760 --> 00:40:59.440
put your evaluation frameworks, figure out what it means to like, how do I, how do I submit a
428
00:40:59.440 --> 00:41:04.480
project? What are the steps? Who approves it? What's the, you know, all that kind of stuff? And then
429
00:41:04.480 --> 00:41:08.480
most importantly, you need someone to teach the organization. And then once that's done, you need to
430
00:41:09.280 --> 00:41:14.160
give the, give the organization the ability to add value. So I think number one, I saw myself,
431
00:41:15.280 --> 00:41:20.480
can they work myself at a job? And by the way, I'm, I'm, couldn't be proud of, right? I wasn't trying
432
00:41:20.480 --> 00:41:26.480
to, I wasn't trying to hang on too long. And I'm very proud of where Morgan Stanley is and
433
00:41:26.480 --> 00:41:30.080
the work that we did. And by the way, they're in great shape, great shape. And they're going to,
434
00:41:30.080 --> 00:41:34.560
I think they're going to do great things because the technology is diffused to the right people
435
00:41:34.560 --> 00:41:42.000
in the business to do that. But secondly, I saw an enormous gap in the, the level of awareness
436
00:41:42.000 --> 00:41:46.400
at the senior level. And it's not, you know, it's not about the models, it's even not about the
437
00:41:46.400 --> 00:41:51.120
agenda clear. It's about, do senior executives really understand what this technology can do their
438
00:41:51.120 --> 00:41:56.000
business and drive value? And then lastly, and I don't want to be, I don't want to be too sentimental
439
00:41:56.000 --> 00:42:01.760
here, but I think like this is a once in a generation technology. And how, what a privilege
440
00:42:02.320 --> 00:42:06.880
to be able to be part of the dialogue here, to drive, you know, I mean, I teach a Columbia,
441
00:42:07.920 --> 00:42:13.280
you know, we just had our, we just had our event last week where I had 50 students,
442
00:42:13.280 --> 00:42:17.680
Bill Day, I solutions in three weeks from scratch, not a single one of the technologists, right?
443
00:42:18.320 --> 00:42:22.160
And to be able to sort of be part of this ecosystem where we're thinking about
444
00:42:23.040 --> 00:42:28.320
policy and education, I don't know, what a, what a, what an opportunity. So, you know, for those
445
00:42:28.320 --> 00:42:32.400
reasons, one, my work was done and two, I think there's a lot of work to be done outside.
446
00:42:33.200 --> 00:42:39.360
I decided to, um, to set up my little McMillan AI shop. It's awesome. Thank you so much
447
00:42:39.360 --> 00:42:43.440
for coming on. This was incredible. It can save from having talked to you for a long time now.
448
00:42:43.440 --> 00:42:49.600
Most people should give you a call. Well, keep, you can find me on McMillan AI.com. Is this
449
00:42:49.600 --> 00:42:56.400
like a, is this like an advertisement? McMillan AI.com. And what I would say is there's lots of good
450
00:42:56.400 --> 00:43:02.400
content on there, but there's also a Jeff bot. If you're interested in talking to the AI version
451
00:43:02.400 --> 00:43:10.320
of me, which I love, I don't know, probably four years of content, um, every speech, every framework.
452
00:43:10.960 --> 00:43:14.800
By the way, Keith, you should ask all these questions to Jeff bot and see if you get a better answer.
453
00:43:14.800 --> 00:43:18.800
You probably will. I think that's awesome. I think I will. That's great.
