1
00:00:05,390 --> 00:00:08,910
Jennifer Reif: You are listening to the
Breaktime Tech Talks podcast, a bite-sized
2
00:00:08,950 --> 00:00:13,489
tech podcast for busy developers where
we'll briefly cover technical topics, news
3
00:00:13,489 --> 00:00:15,749
snippets, and more in short time blocks.
4
00:00:16,230 --> 00:00:19,940
I'm your host, Jennifer Reif, an
avid developer and problem solver
5
00:00:20,149 --> 00:00:24,100
with special interest in data,
learning, and all things technology.
6
00:00:26,327 --> 00:00:28,557
Hello, and welcome to
Breaktime Tech Talks.
7
00:00:28,557 --> 00:00:30,947
Today I have with me John Willis.
8
00:00:31,247 --> 00:00:32,107
So welcome, John.
9
00:00:32,117 --> 00:00:34,107
Thank you so much for
being on the podcast.
10
00:00:34,567 --> 00:00:38,987
Just to give everyone a bit of background,
could you give a high level or quick
11
00:00:39,007 --> 00:00:40,787
bio of who you are and what you do?
12
00:00:41,488 --> 00:00:41,858
John Willis: Yeah.
13
00:00:41,858 --> 00:00:46,678
I-- these days I sorta consider myself
an author, advisor, and investor.
14
00:00:46,678 --> 00:00:53,448
But I've done five decades of different
technology shifts, if you will.
15
00:00:53,538 --> 00:00:57,708
But today I focus on obviously
what everybody else focuses on, AI.
16
00:00:57,718 --> 00:01:00,978
But I, I focus more on the
social technical aspects.
17
00:01:00,978 --> 00:01:01,978
And then I'm an author.
18
00:01:01,978 --> 00:01:03,498
I've written, numerous books.
19
00:01:03,578 --> 00:01:07,538
Probably the most famous is The
DevOps Handbook, la-last book is
20
00:01:07,538 --> 00:01:10,768
called Rebels of Reason, which is
the history-- 100-year history of AI.
21
00:01:10,778 --> 00:01:11,198
Oh, okay.
22
00:01:11,747 --> 00:01:12,377
Jennifer Reif: Oh, very cool.
23
00:01:12,458 --> 00:01:12,718
John Willis: Yeah.
24
00:01:12,768 --> 00:01:13,048
Yeah.
25
00:01:13,078 --> 00:01:13,298
So
26
00:01:13,327 --> 00:01:14,087
Jennifer Reif: I haven't
heard of that one.
27
00:01:14,087 --> 00:01:15,217
I will add that to my reading list.
28
00:01:15,558 --> 00:01:15,788
John Willis: Cool.
29
00:01:15,788 --> 00:01:16,058
Yeah.
30
00:01:16,058 --> 00:01:16,398
Awesome.
31
00:01:17,107 --> 00:01:20,737
Jennifer Reif: So you've spent a
lot of time in IT, and obviously
32
00:01:20,737 --> 00:01:22,657
DevOps is a, is a huge focus for you.
33
00:01:23,027 --> 00:01:27,427
What are some of the lessons that you've
learned, maybe especially through DevOps,
34
00:01:27,507 --> 00:01:30,847
that we really need to keep in mind
as we're going through this new shift?
35
00:01:31,930 --> 00:01:35,680
John Willis: Yeah, I think there is like
I said, I've had five decades of shifts.
36
00:01:35,710 --> 00:01:40,450
I won't bore you with all of them, but
I think the ones that come to mind as
37
00:01:40,450 --> 00:01:43,680
we're dealing with AI right now is Linux.
38
00:01:43,710 --> 00:01:47,400
Most, most people probably weren't
around, but there was this big
39
00:01:47,420 --> 00:01:51,690
debate about the people who were
moving really fast on Linux, and
40
00:01:51,690 --> 00:01:53,950
even some regul-regulated businesses.
41
00:01:54,370 --> 00:01:58,160
And the ones that were telling
me, "John, this bank will never
42
00:01:58,160 --> 00:01:59,780
run Linux in production," right?
43
00:02:00,260 --> 00:02:03,310
And, and you-- And so there,
there are some patterns there.
44
00:02:03,407 --> 00:02:03,747
Jennifer Reif: Sure.
45
00:02:03,750 --> 00:02:05,510
John Willis: Then we had
the same thing in cloud.
46
00:02:05,800 --> 00:02:10,290
And then sort of DevOps was this subset of
cloud, but, but it had all these sort of
47
00:02:10,390 --> 00:02:16,040
things where it-- This is why I talk about
social technical systems, in that you,
48
00:02:16,070 --> 00:02:20,390
you really have to understand the human
conditions related to the technology.
49
00:02:20,880 --> 00:02:23,720
And that sounds obvious, but
there's a lot to it, and especially
50
00:02:23,720 --> 00:02:24,930
at the organizational level.
51
00:02:25,320 --> 00:02:29,060
And so I think the things that are
common are these things that you,
52
00:02:29,170 --> 00:02:33,200
you can't just slam technology on
people and say, "Go forth and do it."
53
00:02:33,220 --> 00:02:38,770
Or I, I describe this idea of these
technologies have this fluorescence.
54
00:02:39,310 --> 00:02:42,350
A lot of people talk about winters
and springs of technology shifts,
55
00:02:42,410 --> 00:02:47,100
particularly AI, but, but there's
really this energy that sort of exists.
56
00:02:47,120 --> 00:02:48,420
It, it gets fluorescent.
57
00:02:48,810 --> 00:02:49,930
The energy stays.
58
00:02:49,930 --> 00:02:51,000
It then inherits.
59
00:02:51,010 --> 00:02:53,770
It glazes or brightens.
60
00:02:54,190 --> 00:02:59,630
And the problem with that fluorescence
is the blinding part, so you
61
00:02:59,729 --> 00:02:59,769
Jennifer Reif: Yeah.
62
00:03:00,080 --> 00:03:01,040
John Willis: Everything's new.
63
00:03:01,040 --> 00:03:06,460
Oh, and you start forgetting what are
the parts that truly are new and what are
64
00:03:06,500 --> 00:03:11,060
the parts that are actually consistent
to all human technology behavior.
65
00:03:11,390 --> 00:03:14,680
And therein lies the,
the real hard problem.
66
00:03:15,027 --> 00:03:15,437
Jennifer Reif: Right.
67
00:03:15,467 --> 00:03:15,717
Right.
68
00:03:15,887 --> 00:03:19,037
And if I were to guess, I would
say we're still probably in that
69
00:03:19,037 --> 00:03:20,727
fluorescence of AI a little bit.
70
00:03:21,060 --> 00:03:21,310
John Willis: Yeah.
71
00:03:21,390 --> 00:03:21,580
Yeah.
72
00:03:21,780 --> 00:03:22,270
Oh, absolutely.
73
00:03:22,270 --> 00:03:22,470
Yeah.
74
00:03:22,470 --> 00:03:24,930
I think we're as deep into
it as you can get, right?
75
00:03:24,930 --> 00:03:26,590
We're probably right at the peak,
76
00:03:26,919 --> 00:03:27,169
Jennifer Reif: Okay.
77
00:03:27,360 --> 00:03:31,240
John Willis: Maybe getting to a little
bit more reality checks on certain things.
78
00:03:31,241 --> 00:03:31,631
Jennifer Reif: Sure.
79
00:03:31,740 --> 00:03:34,430
John Willis: But certainly, yeah, we're,
we're smack right in the middle right now.
80
00:03:34,761 --> 00:03:35,101
Jennifer Reif: Okay.
81
00:03:35,571 --> 00:03:40,791
Are there, I guess, maybe specific things
about the people or processes within
82
00:03:40,791 --> 00:03:46,111
an organization to help adapt to these
big shifts that you've seen work well?
83
00:03:46,706 --> 00:03:49,336
John Willis: The-- there's
always the early adopters, right?
84
00:03:49,336 --> 00:03:53,086
And I, I think early adopters are
great, but, y-you have to calibrate
85
00:03:53,206 --> 00:03:54,646
what your adoption looks like.
86
00:03:55,076 --> 00:03:57,636
And then, particularly as you
go up the sort of the food
87
00:03:57,636 --> 00:04:01,726
chain of regulatory controlled
businesses, you have to be slow.
88
00:04:01,826 --> 00:04:07,206
You can't just race, you can't do cargo
culting of what everybody else is doing.
89
00:04:07,406 --> 00:04:10,146
This knee-jerk reaction is
like, "We're behind." "Who are
90
00:04:10,146 --> 00:04:11,246
you behind?" "I don't know.
91
00:04:11,246 --> 00:04:12,346
We're behind," right?
92
00:04:12,665 --> 00:04:13,085
Jennifer Reif: Yeah.
93
00:04:13,316 --> 00:04:17,296
John Willis: What I'm trying to work
with the clients that will listen to me.
94
00:04:17,356 --> 00:04:22,156
So I'm an independent, and these
days, I'm semi-retired, but I do do
95
00:04:22,156 --> 00:04:25,366
work when a chief of staff or a CIO
calls me or has heard my presentation.
96
00:04:25,459 --> 00:04:25,739
Jennifer Reif: Okay.
97
00:04:26,766 --> 00:04:30,096
John Willis: And I come in and I, I don't
even take the work unless they're willing
98
00:04:30,116 --> 00:04:32,446
to listen to this idea of like, calm down.
99
00:04:33,176 --> 00:04:36,116
People ask me if their first question
is, "What would you recommend?
100
00:04:36,116 --> 00:04:39,026
Is your thirty, sixty, ninety-day
plan?" I'm like, "Don't have one."
101
00:04:39,636 --> 00:04:41,366
So you have to learn how to learn.
102
00:04:41,366 --> 00:04:44,616
And this is consistent with
every technology shift I've seen.
103
00:04:44,616 --> 00:04:45,506
I'm not smart.
104
00:04:45,556 --> 00:04:47,466
I've just been around for a long time,
105
00:04:47,765 --> 00:04:47,845
Jennifer Reif: Yeah.
106
00:04:47,926 --> 00:04:48,266
John Willis: Right?
107
00:04:48,496 --> 00:04:52,406
And, and so I think learning to learn,
and one of the things I've created
108
00:04:52,406 --> 00:04:54,456
is the idea of ideation hackathons.
109
00:04:54,669 --> 00:04:55,119
Jennifer Reif: Okay.
110
00:04:55,556 --> 00:04:58,516
John Willis: The developers are
a whole-- that's a whole sort of
111
00:04:58,526 --> 00:05:01,356
animal that I don't really focus on.
112
00:05:01,386 --> 00:05:03,996
Although I have a lot of experience,
obviously DevOps and all that stuff.
113
00:05:04,091 --> 00:05:04,541
Jennifer Reif: Sure.
114
00:05:05,296 --> 00:05:09,206
John Willis: But my concern is that
the organization wants to take all
115
00:05:09,206 --> 00:05:12,296
the executive assistants in the
finance department, in the supply
116
00:05:12,296 --> 00:05:16,416
chain, and in marketing, and they
wanna just throw them right into
117
00:05:16,416 --> 00:05:17,726
the middle of the ocean with AI.
118
00:05:17,961 --> 00:05:18,401
Jennifer Reif: Mm.
119
00:05:19,336 --> 00:05:22,736
John Willis: And, and I think there's
an organizational calibration.
120
00:05:22,926 --> 00:05:24,476
You have to learn how to learn.
121
00:05:24,486 --> 00:05:30,016
And so I think the ideation hackathon
is a great way to figure out how are
122
00:05:30,016 --> 00:05:32,136
these groups gonna work together?
123
00:05:32,236 --> 00:05:34,376
How are they gonna work
within the organization?
124
00:05:34,666 --> 00:05:36,776
People ask me, "What, what
tools you're gonna use?" None.
125
00:05:37,506 --> 00:05:38,556
"Well, what do you mean no tools?
126
00:05:38,596 --> 00:05:40,686
Can-- Why aren't you just
giving them, you know, Codex?
127
00:05:40,686 --> 00:05:41,976
Or why don't you give them Claude?
128
00:05:41,976 --> 00:05:45,556
Or why don't you give them, a Club
AI?" I'm like, "Because that's the
129
00:05:45,556 --> 00:05:47,296
worst possible thing you can do."
130
00:05:48,066 --> 00:05:53,456
Having a couple hundred executive
assistants all individually choosing the
131
00:05:53,456 --> 00:05:57,326
technical debt nightmare, and that-- I'm
not even getting into the risk problem.
132
00:05:57,541 --> 00:05:57,901
Jennifer Reif: Right.
133
00:05:58,186 --> 00:05:59,636
John Willis: So just slow down.
134
00:05:59,756 --> 00:06:02,826
Learn how to ideate in this new world.
135
00:06:03,066 --> 00:06:04,796
Learn as an organization hand off.
136
00:06:04,806 --> 00:06:05,456
And I've done these.
137
00:06:05,476 --> 00:06:06,086
They're brilliant.
138
00:06:06,506 --> 00:06:09,706
You let those teams come up with
the ideas, figure out what kind
139
00:06:09,706 --> 00:06:13,396
of things actually do make sense
with AI, what things don't.
140
00:06:13,706 --> 00:06:14,476
Because that's the other problem.
141
00:06:14,486 --> 00:06:17,016
If you just hand somebody and
said, "Go automate everything,"
142
00:06:17,316 --> 00:06:18,466
they're gonna automate everything,
143
00:06:18,687 --> 00:06:19,097
Jennifer Reif: Yeah.
144
00:06:19,837 --> 00:06:20,117
Yeah,
145
00:06:20,196 --> 00:06:20,286
John Willis: And
146
00:06:20,777 --> 00:06:23,437
Jennifer Reif: that's actually a
fair point too, because a lot of
147
00:06:23,437 --> 00:06:27,887
times outside of the developer space,
we don't think about how different
148
00:06:27,907 --> 00:06:31,097
adopting that technology looks like
for somebody who's non-technical.
149
00:06:31,457 --> 00:06:33,547
How do they know the guardrails?
150
00:06:33,557 --> 00:06:36,477
How do they know what to
trust and what not to?
151
00:06:36,477 --> 00:06:40,697
How do they know how to guide
the AI tools into the results and
152
00:06:40,697 --> 00:06:41,867
the answers they're looking for?
153
00:06:42,426 --> 00:06:45,716
John Willis: Yeah, no, a great example,
like in an ideation hackathon I did
154
00:06:45,716 --> 00:06:49,466
last year for a candy bar company,
th- this one team had a brilliant
155
00:06:49,466 --> 00:06:55,316
idea, and they didn't even understand
the consequences of data provenance.
156
00:06:55,559 --> 00:06:56,149
Jennifer Reif: Mm-hmm.
157
00:06:56,696 --> 00:06:57,106
John Willis: right?
158
00:06:57,299 --> 00:06:57,589
Jennifer Reif: Yeah
159
00:06:57,716 --> 00:07:00,946
John Willis: Again,, th- w- you want--
you don't wanna find that out later.
160
00:07:01,119 --> 00:07:01,499
Jennifer Reif: Right.
161
00:07:01,586 --> 00:07:01,736
John Willis: You
162
00:07:01,909 --> 00:07:03,329
Jennifer Reif: Yeah, that's
that's a big enough concern
163
00:07:03,329 --> 00:07:04,069
you need to know ahead of time.
164
00:07:04,236 --> 00:07:06,626
John Willis: You want to have an
advisement during the ideation phase
165
00:07:06,626 --> 00:07:08,796
to where "Here's what we're gonna
do." "Oh, well, we're gonna have
166
00:07:08,796 --> 00:07:11,406
to rethink that data right there."
167
00:07:11,591 --> 00:07:15,341
And we can spend an hour on what
can happen with, wrong data,
168
00:07:15,478 --> 00:07:15,768
Jennifer Reif: Yeah.
169
00:07:16,178 --> 00:07:16,598
Oh, yeah.
170
00:07:16,681 --> 00:07:17,111
John Willis: and inference.
171
00:07:17,111 --> 00:07:17,401
Yeah.
172
00:07:17,558 --> 00:07:18,128
Jennifer Reif: Absolutely.
173
00:07:18,838 --> 00:07:25,348
Okay, is there a big disconnect between
maybe leadership and teams or maybe just
174
00:07:25,348 --> 00:07:27,388
even within teams and departments then?
175
00:07:28,767 --> 00:07:29,067
John Willis: Yeah.
176
00:07:29,077 --> 00:07:30,797
This is across the board.
177
00:07:30,797 --> 00:07:32,637
There's so many… I'm pro-AI, right?
178
00:07:32,647 --> 00:07:35,117
So, the one thing I always, when
I'm presenting or something,
179
00:07:35,117 --> 00:07:38,297
I wanna say, "Let me get this
clear." I think this is phenomenal.
180
00:07:39,097 --> 00:07:42,297
When you think about pre-GPS
for getting anywhere.
181
00:07:42,344 --> 00:07:42,684
Jennifer Reif: Ja
182
00:07:42,787 --> 00:07:43,707
John Willis: I can't imagine.
183
00:07:43,727 --> 00:07:46,657
There'll be a point where I'm
like, I can't imagine not having
184
00:07:46,657 --> 00:07:49,747
these tools, as, as a companion.
185
00:07:49,857 --> 00:07:53,557
Again, I understand that it is
inference, it's probability, but, just
186
00:07:53,557 --> 00:07:54,977
don't eliminate critical thinking.
187
00:07:55,176 --> 00:07:55,496
Jennifer Reif: Right.
188
00:07:56,207 --> 00:07:57,577
John Willis: But, back to
the fluorescence, right?
189
00:07:57,577 --> 00:08:01,707
Getting blinded, and they're
throwing out all this sort of logic.
190
00:08:02,307 --> 00:08:04,107
I can go through a whole list, but audit.
191
00:08:04,457 --> 00:08:08,987
I'm ridiculously concerned
about what audit's gonna look.
192
00:08:09,007 --> 00:08:12,397
And worse, I go to a lot of conferences,
and I talk to people, really smart
193
00:08:12,397 --> 00:08:15,697
people giving presentations about
how well we're doing AI here.
194
00:08:15,997 --> 00:08:21,017
And I ask them simple questions about
evidence or evaluations or sort of
195
00:08:21,017 --> 00:08:25,957
notion of LLM as a judge or, or how
you's gonna manage, like There's a
196
00:08:25,977 --> 00:08:28,087
term called human on the loop, right?
197
00:08:28,097 --> 00:08:28,347
Instead of
198
00:08:28,362 --> 00:08:28,492
Jennifer Reif: Yeah.
199
00:08:28,827 --> 00:08:29,867
John Willis: In the loop, right?
200
00:08:29,947 --> 00:08:31,787
How are you gonna deal with agentics?
201
00:08:31,827 --> 00:08:36,757
Because there's a whole 'nother
level of containment, kill switches.
202
00:08:36,817 --> 00:08:40,327
It's like in a headlight
discussions, right?
203
00:08:40,327 --> 00:08:44,697
So, I think this, this push to go
fast, and everybody has to catch
204
00:08:44,747 --> 00:08:48,097
up, and what is our competitor
doing, and we can't lose out.
205
00:08:48,597 --> 00:08:51,907
Risk I've-- I didn't go to Black
Hat this year, but a friend of
206
00:08:51,907 --> 00:08:55,317
mine just got back and said, "Cyber
is completely broken," right?
207
00:08:55,647 --> 00:08:58,577
And I'm like, "Yeah, no,
risk is completely broken."
208
00:08:58,678 --> 00:08:59,138
Jennifer Reif: Yeah.
209
00:08:59,647 --> 00:09:04,657
John Willis: And everybody's trying to
overlay the old cyber and risk patterns
210
00:09:05,267 --> 00:09:11,357
on this new completely different, it's
like Von Neumann to quantum, right?
211
00:09:11,357 --> 00:09:12,827
Von Neumann architecture to quantum.
212
00:09:12,877 --> 00:09:14,967
It is, like you can't think the same.
213
00:09:15,262 --> 00:09:15,442
Jennifer Reif: Right.
214
00:09:15,777 --> 00:09:19,217
John Willis: And I see people
writing governance manifestos for AI.
215
00:09:19,217 --> 00:09:22,177
And it's like, okay, you just
added the word AI into the list
216
00:09:22,512 --> 00:09:22,842
Jennifer Reif: Yeah.
217
00:09:23,022 --> 00:09:27,142
Yeah, I think we're still trying to
figure out how to switch that mental
218
00:09:27,142 --> 00:09:31,592
model, and I think it goes back to
what you said earlier, where everybody
219
00:09:31,592 --> 00:09:35,422
feels behind, no matter how fast
or how far they might be moving.
220
00:09:35,802 --> 00:09:39,302
They just feel behind everything
else, and so therefore, I think some
221
00:09:39,302 --> 00:09:43,262
of this stuff kind of falls to the
wayside because they're just trying
222
00:09:43,262 --> 00:09:45,052
to keep up or stay ahead of the game.
223
00:09:46,057 --> 00:09:48,407
John Willis: The other thing I wanna be
writing a little bit more about where
224
00:09:48,407 --> 00:09:50,047
we are in this sort of fluorescence.
225
00:09:50,107 --> 00:09:53,907
We're at a point now where I think you
can, you can honestly go back and compare
226
00:09:53,907 --> 00:09:55,577
what happened in the early days of cloud.
227
00:09:56,117 --> 00:09:58,937
And in the early days of cloud, everybody
raced to put things on the cloud.
228
00:09:59,727 --> 00:10:01,777
And then there was this pullback, right?
229
00:10:01,787 --> 00:10:02,967
For obvious reasons.
230
00:10:02,977 --> 00:10:03,717
Oh, what do you mean?
231
00:10:03,757 --> 00:10:05,367
That probably shouldn't
have been in the cloud.
232
00:10:05,777 --> 00:10:06,927
You can't put that in the cloud.
233
00:10:06,957 --> 00:10:07,927
You could get sued.
234
00:10:07,977 --> 00:10:12,487
And I, I think we're gonna see
that sort of reactive process
235
00:10:12,487 --> 00:10:13,797
here hopefully happening.
236
00:10:13,937 --> 00:10:18,097
I think the real thing is when,
when internal auditing catches up.
237
00:10:18,717 --> 00:10:19,927
And that, that's what happened to cloud.
238
00:10:20,067 --> 00:10:24,517
Internal auditing, like, why
is that data in a public cloud?
239
00:10:24,918 --> 00:10:25,868
How did that get there?
240
00:10:25,878 --> 00:10:27,228
Or, or it got exposed.
241
00:10:27,578 --> 00:10:31,448
And so, we're gonna have to
learn how to, how to pull back.
242
00:10:31,498 --> 00:10:33,108
Cloud it was mostly about data.
243
00:10:33,138 --> 00:10:37,018
AI inference, there's a lot about
data, but there's sort of new risks.
244
00:10:37,518 --> 00:10:42,298
Cloud was could the service or data
running on the cloud be breached?
245
00:10:43,285 --> 00:10:43,535
Jennifer Reif: Mhm.
246
00:10:43,668 --> 00:10:46,358
John Willis: Or, or were you
protecting regulatory data?
247
00:10:46,668 --> 00:10:50,788
Now we got very the same, but now
we got the question of what goes
248
00:10:50,788 --> 00:10:55,608
into a public model, and, and
what's being kept, what's not kept.
249
00:10:55,608 --> 00:10:57,618
These are opaque systems that we have
250
00:10:57,668 --> 00:10:57,788
Jennifer Reif: Right.
251
00:10:57,858 --> 00:10:58,478
John Willis: no idea.
252
00:10:58,628 --> 00:11:00,447
Jennifer Reif: Yeah, it's
still a bit of a black box.
253
00:11:00,808 --> 00:11:01,118
John Willis: That's right.
254
00:11:01,128 --> 00:11:02,108
It is a black box.
255
00:11:02,247 --> 00:11:02,507
Jennifer Reif: Yeah.
256
00:11:02,568 --> 00:11:04,708
John Willis: No matter what they
tell you, it is a black box.
257
00:11:04,738 --> 00:11:05,157
The public ones
258
00:11:05,177 --> 00:11:05,507
Jennifer Reif: Yeah.
259
00:11:05,708 --> 00:11:06,058
John Willis: for sure.
260
00:11:06,267 --> 00:11:10,897
Jennifer Reif: Do you think this is
more of a people or a process type of
261
00:11:11,037 --> 00:11:12,717
function, or maybe a little bit of both?
262
00:11:13,494 --> 00:11:15,574
John Willis: Yeah, the, the
quick answer is both of course.
263
00:11:15,574 --> 00:11:18,334
But, there's another point I was
driving on that last conversation,
264
00:11:18,334 --> 00:11:22,694
which was, the one thing we learned in
the cloud, which was we had to figure
265
00:11:22,694 --> 00:11:28,234
out in, in the simplest form was,
green, yellow, red, blue applications.
266
00:11:28,363 --> 00:11:28,673
Jennifer Reif: Okay.
267
00:11:28,724 --> 00:11:30,164
John Willis: It was mostly around
data classification, but it
268
00:11:30,164 --> 00:11:31,334
was a service classification.
269
00:11:32,164 --> 00:11:38,244
So I'm suggesting for a large corporation,
don't put all your eggs in one basket.
270
00:11:38,244 --> 00:11:41,164
Don't do nothing or don't do everything,
271
00:11:41,247 --> 00:11:41,857
Jennifer Reif: Right.
272
00:11:42,017 --> 00:11:42,337
Sure.
273
00:11:42,344 --> 00:11:44,694
John Willis: Start thinking
about services and learn how to
274
00:11:44,694 --> 00:11:47,274
learn on the low-hanging fruit.
275
00:11:47,604 --> 00:11:49,204
So go as fast as possible.
276
00:11:49,214 --> 00:11:52,124
Have pilots, have groups
that are investing in trying
277
00:11:52,124 --> 00:11:53,494
out the different providers.
278
00:11:53,794 --> 00:11:55,084
Try open source ones.
279
00:11:55,344 --> 00:11:56,194
Try the open weight.
280
00:11:56,204 --> 00:11:59,974
Try the, the Gemini, the family from
Google is very powerful or OpenAI.
281
00:11:59,974 --> 00:12:00,274
Jennifer Reif: Yeah.
282
00:12:00,274 --> 00:12:01,854
John Willis: And, and spread your wings.
283
00:12:01,854 --> 00:12:03,354
You don't have to lock in right now.
284
00:12:03,429 --> 00:12:04,919
The game is changing so fast.
285
00:12:05,084 --> 00:12:05,224
Jennifer Reif: Yeah.
286
00:12:05,339 --> 00:12:07,899
John Willis: So but, but
figure out that spectrum.
287
00:12:08,709 --> 00:12:13,049
That's part of the take a deep
breath, to the coffee machine, get
288
00:12:13,049 --> 00:12:17,109
a really good cup of coffee, smell
it, taste it, then sit down and start
289
00:12:17,129 --> 00:12:18,166
thinking about what you're gonna do.
290
00:12:18,166 --> 00:12:21,969
So I actually tell when, when, a
chief of staff calls me, and they're
291
00:12:21,969 --> 00:12:25,459
like, "Oh, John, things are going
crazy." I go, I want you to calm down.
292
00:12:26,519 --> 00:12:29,259
I want you to go to the coffee
machine." I've, I've actually said this.
293
00:12:29,419 --> 00:12:32,909
"I want you to call me back in about
ten minutes or go down to the local
294
00:12:33,819 --> 00:12:35,619
barista, 20 minutes, and call me back."
295
00:12:35,726 --> 00:12:36,106
Jennifer Reif: Yeah.
296
00:12:36,179 --> 00:12:38,499
John Willis: So yeah, and,
and it really is that simple.
297
00:12:38,874 --> 00:12:39,104
Jennifer Reif: Okay.
298
00:12:39,794 --> 00:12:47,384
I, I've mentioned on the podcast how the
AI systems especially can be sensitive
299
00:12:47,384 --> 00:12:49,534
to configuration or setting changes.
300
00:12:49,614 --> 00:12:55,084
A new model comes out, something changes
on a language version or a library
301
00:12:55,084 --> 00:12:59,044
version somewhere, and that causes
kind of upset within your application.
302
00:12:59,044 --> 00:13:03,074
And of course, because these things
release so quickly, all technologies
303
00:13:03,074 --> 00:13:07,134
really now, then you feel like you're
in a constant state of breaking.
304
00:13:07,754 --> 00:13:14,764
How should people and teams and processes
maybe start thinking about that constant
305
00:13:14,764 --> 00:13:18,844
state of moving fast and breaking
things, but on a large scale now?
306
00:13:19,623 --> 00:13:21,243
John Willis: Yeah, this
is a hard problem, right?
307
00:13:21,243 --> 00:13:23,233
This is probably the hardest
problem there is right now.
308
00:13:23,263 --> 00:13:25,593
'Cause I think this-- th-that you
could tier this in a number of ways.
309
00:13:25,593 --> 00:13:27,703
One is the buy versus build conversation.
310
00:13:27,894 --> 00:13:28,274
Jennifer Reif: Okay.
311
00:13:28,333 --> 00:13:28,553
John Willis: Right?
312
00:13:28,553 --> 00:13:31,753
Which is a really interesting conversation
going on in large corporations.
313
00:13:31,763 --> 00:13:33,993
And then you can imagine
it has lots of flavors.
314
00:13:34,343 --> 00:13:35,553
Some wrong, some good.
315
00:13:35,843 --> 00:13:39,043
Some of the ones is I've got, like
I have four thousand developers.
316
00:13:39,043 --> 00:13:41,603
I don't wanna lay them off 'cause
they're integral to the business.
317
00:13:42,613 --> 00:13:43,753
What can I do with them?
318
00:13:43,963 --> 00:13:46,543
Could I replace an institutional product.
319
00:13:47,562 --> 00:13:47,822
Jennifer Reif: Mm.
320
00:13:47,961 --> 00:13:49,331
John Willis: I could set a reduction.
321
00:13:50,081 --> 00:13:53,621
I could actually get productive and
remove five million, ten million dollar.
322
00:13:53,871 --> 00:13:57,961
But then the question is,
what is the technology?
323
00:13:57,961 --> 00:13:59,991
What is the complete
system that you've got?
324
00:13:59,991 --> 00:14:01,931
So that's a whole conversation itself.
325
00:14:02,072 --> 00:14:06,362
I think there's this other idea, and this
one I, I kinda get, but again, I worry
326
00:14:06,362 --> 00:14:11,252
about the highly regulated businesses,
the banks, the healthcare, the ones
327
00:14:11,252 --> 00:14:13,512
where, high consequence environments.
328
00:14:14,392 --> 00:14:17,622
Is this idea that the
code is the new assembler.
329
00:14:17,622 --> 00:14:20,472
And I, I, I, I get it,
but that's an all-in.
330
00:14:21,343 --> 00:14:21,723
Jennifer Reif: Okay.
331
00:14:22,582 --> 00:14:24,542
John Willis: And in that world, I'm
just hearing really smart people and
332
00:14:24,582 --> 00:14:29,672
friends of mine who literally are like
top Java architects for banks say things
333
00:14:29,672 --> 00:14:33,462
like, "Code is the new assembler,"
meaning just like we didn't care.
334
00:14:33,482 --> 00:14:35,222
I started my career as
an assembler programmer.
335
00:14:35,522 --> 00:14:38,372
I don't, I don't write anything
in assembler today, right?
336
00:14:38,629 --> 00:14:39,099
Jennifer Reif: Right.
337
00:14:39,412 --> 00:14:41,782
John Willis: Maybe there's a
world where, and I'm not trying
338
00:14:41,782 --> 00:14:43,152
to get anybody riled up now.
339
00:14:43,212 --> 00:14:46,492
But maybe there's a world where you
don't ev- ever write code, and I'm
340
00:14:46,492 --> 00:14:50,032
seeing it, a lot of people right
now talk about no, no code review
341
00:14:50,032 --> 00:14:51,862
in six months, haven't written code.
342
00:14:52,272 --> 00:14:55,782
And in that world, you're still
gonna have to deal with your
343
00:14:55,782 --> 00:15:00,392
question, but maybe it's less because
it's dynamic and it's changing.
344
00:15:01,112 --> 00:15:06,662
And then I think the third tier
answer to this question is, back
345
00:15:06,662 --> 00:15:11,152
to understanding the service that
you're dealing with and how sensitive
346
00:15:11,152 --> 00:15:16,932
it is it to model changes and
particularly inference and probability.
347
00:15:17,229 --> 00:15:17,529
Jennifer Reif: Okay.
348
00:15:17,552 --> 00:15:18,732
John Willis: Because that's
the biggest problem, right?
349
00:15:18,732 --> 00:15:23,312
Is that I could have an application
working on Opus four out of whatever
350
00:15:23,812 --> 00:15:27,142
right now, perfect, and then I
upgrade the model, and it just
351
00:15:27,162 --> 00:15:28,932
completely gets different answers.
352
00:15:28,983 --> 00:15:29,443
Jennifer Reif: Right.
353
00:15:29,512 --> 00:15:31,802
John Willis: to a degree, maybe
it's three percent different,
354
00:15:31,812 --> 00:15:33,142
maybe it's four percent different.
355
00:15:33,442 --> 00:15:36,812
But again, depending what
the actual delivery mechanism
356
00:15:36,812 --> 00:15:38,772
is, could kill people, right?
357
00:15:38,945 --> 00:15:39,245
Jennifer Reif: True.
358
00:15:39,482 --> 00:15:42,492
John Willis: Here's another thing I, I
like about my career is we've learned
359
00:15:42,562 --> 00:15:45,422
to borrow ideas from other spaces.
360
00:15:45,422 --> 00:15:48,312
So in the DevOps world, we
looked at resilience and people
361
00:15:48,312 --> 00:15:49,752
look at critical safety and,
362
00:15:49,909 --> 00:15:50,309
Jennifer Reif: Okay.
363
00:15:50,482 --> 00:15:53,932
John Willis: And stuff like airplane
crashes and patients dying in a hospital
364
00:15:53,932 --> 00:15:56,832
due to some form of procedural problem.
365
00:15:57,069 --> 00:15:57,489
Jennifer Reif: Right
366
00:15:57,542 --> 00:15:59,955
John Willis: There's great wealth
of information there to learn from
367
00:16:00,005 --> 00:16:00,415
Jennifer Reif: Mhm.
368
00:16:00,622 --> 00:16:02,992
John Willis: But one is something
called ETO principle, which is the
369
00:16:02,992 --> 00:16:04,602
efficiency thoroughness trade-off.
370
00:16:05,029 --> 00:16:05,429
Jennifer Reif: Okay.
371
00:16:05,860 --> 00:16:09,390
John Willis: And so then thinking
that way about maybe I should
372
00:16:09,390 --> 00:16:11,640
use open weights and open source.
373
00:16:12,830 --> 00:16:18,590
There's gonna be a cost of doing that
the big provider that gives me all
374
00:16:18,590 --> 00:16:20,640
this stuff and is better at inference.
375
00:16:20,873 --> 00:16:21,233
Jennifer Reif: Okay.
376
00:16:21,810 --> 00:16:24,830
John Willis: And I'm gonna have to
maintain the models, but at least I can
377
00:16:24,830 --> 00:16:26,970
control those model and maintenance.
378
00:16:27,330 --> 00:16:29,960
There's this sort of balance
that you have to figure out.
379
00:16:29,960 --> 00:16:33,930
And this, this is probably one of
the harder problems right now that
380
00:16:33,930 --> 00:16:37,190
I think organizations, particularly
regulated industries, are facing.
381
00:16:37,543 --> 00:16:37,953
Jennifer Reif: Okay.
382
00:16:38,573 --> 00:16:45,050
So perhaps especially for very high
criticality use cases and industries
383
00:16:45,050 --> 00:16:50,390
and businesses looking at things
that they have more, I, I don't
384
00:16:50,390 --> 00:16:53,760
wanna say necessarily control, but at
least more fine-tuning capabilities.
385
00:16:55,115 --> 00:16:57,085
John Willis: Yeah, think about
the answer to the question.
386
00:16:57,105 --> 00:17:01,385
Or in agentic process that
literally you've given away
387
00:17:01,395 --> 00:17:02,895
some level of human control.
388
00:17:03,835 --> 00:17:08,965
What are the consequences
of degree of a wrong answer?
389
00:17:09,894 --> 00:17:09,904
Jennifer Reif: Mm.
390
00:17:10,505 --> 00:17:13,695
John Willis: That could be a cost, that
could be a human experience, right?
391
00:17:14,035 --> 00:17:18,035
That's where you have to decide the
cost of maintaining and building
392
00:17:18,695 --> 00:17:22,835
on top of open or open weight
models and supporting them yourself
393
00:17:23,180 --> 00:17:23,420
Jennifer Reif: Yeah.
394
00:17:23,915 --> 00:17:27,005
John Willis: And getting probably
lower efficacy, but more th-
395
00:17:27,295 --> 00:17:29,105
throughput or thoroughness, right?
396
00:17:30,476 --> 00:17:30,496
Jennifer Reif: Yeah.
397
00:17:31,695 --> 00:17:35,415
John Willis: Versus what is the risk
of using a very powerful public model.
398
00:17:36,295 --> 00:17:40,295
I think the good news too is I just read
the other day, and again, I just saw this.
399
00:17:40,295 --> 00:17:41,655
It's in my queue to follow up on it.
400
00:17:41,655 --> 00:17:48,595
But, Google is probably three in a
five-horse race on models, but they
401
00:17:48,595 --> 00:17:52,715
do have this hybridity where you can
run their public and private Gemini.
402
00:17:53,228 --> 00:17:53,828
Jennifer Reif: Oh, that's nice.
403
00:17:54,255 --> 00:17:58,125
John Willis: But, up until recently,
I'm almost certain OpenAI does not have
404
00:17:58,125 --> 00:18:01,105
that hybridity, and Claude did not.
405
00:18:01,115 --> 00:18:03,705
But I just saw something the other
day that implied that they might
406
00:18:03,775 --> 00:18:08,175
be allowing some customers to run
their own sort of private version.
407
00:18:08,175 --> 00:18:09,785
And again, that's all I know about it.
408
00:18:09,925 --> 00:18:11,795
But that is very encouraging.
409
00:18:11,822 --> 00:18:13,612
Because now you can get
the best of all worlds.
410
00:18:13,923 --> 00:18:14,923
Jennifer Reif: More things to test.
411
00:18:15,382 --> 00:18:18,012
John Willis: Well, you can
pin a very powerful model.
412
00:18:18,062 --> 00:18:19,622
You, you shouldn't have
got me on a podcast.
413
00:18:19,782 --> 00:18:20,792
I always go over.
414
00:18:21,083 --> 00:18:21,533
Jennifer Reif: No worries.
415
00:18:21,542 --> 00:18:24,972
John Willis: The cloud thing, the
metaphor we would use is, should you
416
00:18:25,822 --> 00:18:29,812
your own electricity creating grid, or
417
00:18:30,048 --> 00:18:31,308
use a public service?
418
00:18:31,524 --> 00:18:34,234
And that would be the argument for
why you might want to use cloud.
419
00:18:34,234 --> 00:18:36,914
Should you build your own infrastructure,
maintain your infrastructure when
420
00:18:36,914 --> 00:18:38,684
you could get all this as a utility?
421
00:18:38,743 --> 00:18:39,193
Jennifer Reif: Right.
422
00:18:39,294 --> 00:18:42,104
John Willis: A-and we're in that
same question now about inference.
423
00:18:42,293 --> 00:18:42,643
Jennifer Reif: Right.
424
00:18:43,184 --> 00:18:47,364
John Willis: What is the cost of managing
and building and maintaining your own
425
00:18:47,434 --> 00:18:50,024
open weight models that can keep up?
426
00:18:50,184 --> 00:18:51,874
And the open weight models
are pretty powerful.
427
00:18:52,774 --> 00:18:55,754
And I've said it multiple
times, it depends on what the
428
00:18:55,754 --> 00:18:56,924
service you're trying to do.
429
00:18:57,159 --> 00:18:57,419
Jennifer Reif: Sure.
430
00:18:57,644 --> 00:18:57,904
John Willis: So
431
00:18:58,809 --> 00:19:02,599
Jennifer Reif: Yep, the age-old, you
know, development answer is, it depends.
432
00:19:03,008 --> 00:19:04,448
John Willis: It depends, absolutely.
433
00:19:04,688 --> 00:19:05,558
More now than ever.
434
00:19:05,689 --> 00:19:06,989
Jennifer Reif: Yeah, yep, very true.
435
00:19:07,229 --> 00:19:11,999
Well, John, thank you so much for coming
onto the podcast and talking about your
436
00:19:11,999 --> 00:19:15,109
experience and the things that you've
learned through DevOps, and how that can
437
00:19:15,109 --> 00:19:18,159
be applied now throughout the AI era.
438
00:19:18,229 --> 00:19:19,729
John Willis: Yeah, no,
thank you for inviting me.
439
00:19:19,749 --> 00:19:21,619
Jennifer Reif: My other
question is for resources.
440
00:19:21,629 --> 00:19:24,349
It, I know you mentioned
your book as well.
441
00:19:24,349 --> 00:19:28,139
I'll link to that, and if there's anything
else you wanna share or things coming up?
442
00:19:28,620 --> 00:19:30,050
John Willis: Yeah, I got author portal.
443
00:19:30,120 --> 00:19:31,800
My author portal has all my books.
444
00:19:31,989 --> 00:19:32,359
Jennifer Reif: Great.
445
00:19:32,430 --> 00:19:35,220
John Willis: So… my LinkedIn is
the best way to get a hold of me.
446
00:19:35,270 --> 00:19:38,640
I do strategy advisement
for very select customers.
447
00:19:38,680 --> 00:19:42,550
And I always start with the innovation
hackathons is I think the best way
448
00:19:42,550 --> 00:19:44,210
to get started for an organization.
449
00:19:44,260 --> 00:19:44,740
Jennifer Reif: Perfect.
450
00:19:44,963 --> 00:19:45,813
John Willis: I'll get you
all those links, yeah.
451
00:19:45,920 --> 00:19:46,510
Jennifer Reif: Excellent.
452
00:19:46,683 --> 00:19:47,060
John Willis: Thanks so
much for inviting me.
453
00:19:47,060 --> 00:19:47,960
Jennifer Reif: Yeah, absolutely.
454
00:19:48,120 --> 00:19:48,960
John Willis: All right, take care.
455
00:19:49,010 --> 00:19:49,290
Jennifer Reif: Bye.
00:00:05,390 --> 00:00:08,910
Jennifer Reif: You are listening to the
Breaktime Tech Talks podcast, a bite-sized
2
00:00:08,950 --> 00:00:13,489
tech podcast for busy developers where
we'll briefly cover technical topics, news
3
00:00:13,489 --> 00:00:15,749
snippets, and more in short time blocks.
4
00:00:16,230 --> 00:00:19,940
I'm your host, Jennifer Reif, an
avid developer and problem solver
5
00:00:20,149 --> 00:00:24,100
with special interest in data,
learning, and all things technology.
6
00:00:26,327 --> 00:00:28,557
Hello, and welcome to
Breaktime Tech Talks.
7
00:00:28,557 --> 00:00:30,947
Today I have with me John Willis.
8
00:00:31,247 --> 00:00:32,107
So welcome, John.
9
00:00:32,117 --> 00:00:34,107
Thank you so much for
being on the podcast.
10
00:00:34,567 --> 00:00:38,987
Just to give everyone a bit of background,
could you give a high level or quick
11
00:00:39,007 --> 00:00:40,787
bio of who you are and what you do?
12
00:00:41,488 --> 00:00:41,858
John Willis: Yeah.
13
00:00:41,858 --> 00:00:46,678
I-- these days I sorta consider myself
an author, advisor, and investor.
14
00:00:46,678 --> 00:00:53,448
But I've done five decades of different
technology shifts, if you will.
15
00:00:53,538 --> 00:00:57,708
But today I focus on obviously
what everybody else focuses on, AI.
16
00:00:57,718 --> 00:01:00,978
But I, I focus more on the
social technical aspects.
17
00:01:00,978 --> 00:01:01,978
And then I'm an author.
18
00:01:01,978 --> 00:01:03,498
I've written, numerous books.
19
00:01:03,578 --> 00:01:07,538
Probably the most famous is The
DevOps Handbook, la-last book is
20
00:01:07,538 --> 00:01:10,768
called Rebels of Reason, which is
the history-- 100-year history of AI.
21
00:01:10,778 --> 00:01:11,198
Oh, okay.
22
00:01:11,747 --> 00:01:12,377
Jennifer Reif: Oh, very cool.
23
00:01:12,458 --> 00:01:12,718
John Willis: Yeah.
24
00:01:12,768 --> 00:01:13,048
Yeah.
25
00:01:13,078 --> 00:01:13,298
So
26
00:01:13,327 --> 00:01:14,087
Jennifer Reif: I haven't
heard of that one.
27
00:01:14,087 --> 00:01:15,217
I will add that to my reading list.
28
00:01:15,558 --> 00:01:15,788
John Willis: Cool.
29
00:01:15,788 --> 00:01:16,058
Yeah.
30
00:01:16,058 --> 00:01:16,398
Awesome.
31
00:01:17,107 --> 00:01:20,737
Jennifer Reif: So you've spent a
lot of time in IT, and obviously
32
00:01:20,737 --> 00:01:22,657
DevOps is a, is a huge focus for you.
33
00:01:23,027 --> 00:01:27,427
What are some of the lessons that you've
learned, maybe especially through DevOps,
34
00:01:27,507 --> 00:01:30,847
that we really need to keep in mind
as we're going through this new shift?
35
00:01:31,930 --> 00:01:35,680
John Willis: Yeah, I think there is like
I said, I've had five decades of shifts.
36
00:01:35,710 --> 00:01:40,450
I won't bore you with all of them, but
I think the ones that come to mind as
37
00:01:40,450 --> 00:01:43,680
we're dealing with AI right now is Linux.
38
00:01:43,710 --> 00:01:47,400
Most, most people probably weren't
around, but there was this big
39
00:01:47,420 --> 00:01:51,690
debate about the people who were
moving really fast on Linux, and
40
00:01:51,690 --> 00:01:53,950
even some regul-regulated businesses.
41
00:01:54,370 --> 00:01:58,160
And the ones that were telling
me, "John, this bank will never
42
00:01:58,160 --> 00:01:59,780
run Linux in production," right?
43
00:02:00,260 --> 00:02:03,310
And, and you-- And so there,
there are some patterns there.
44
00:02:03,407 --> 00:02:03,747
Jennifer Reif: Sure.
45
00:02:03,750 --> 00:02:05,510
John Willis: Then we had
the same thing in cloud.
46
00:02:05,800 --> 00:02:10,290
And then sort of DevOps was this subset of
cloud, but, but it had all these sort of
47
00:02:10,390 --> 00:02:16,040
things where it-- This is why I talk about
social technical systems, in that you,
48
00:02:16,070 --> 00:02:20,390
you really have to understand the human
conditions related to the technology.
49
00:02:20,880 --> 00:02:23,720
And that sounds obvious, but
there's a lot to it, and especially
50
00:02:23,720 --> 00:02:24,930
at the organizational level.
51
00:02:25,320 --> 00:02:29,060
And so I think the things that are
common are these things that you,
52
00:02:29,170 --> 00:02:33,200
you can't just slam technology on
people and say, "Go forth and do it."
53
00:02:33,220 --> 00:02:38,770
Or I, I describe this idea of these
technologies have this fluorescence.
54
00:02:39,310 --> 00:02:42,350
A lot of people talk about winters
and springs of technology shifts,
55
00:02:42,410 --> 00:02:47,100
particularly AI, but, but there's
really this energy that sort of exists.
56
00:02:47,120 --> 00:02:48,420
It, it gets fluorescent.
57
00:02:48,810 --> 00:02:49,930
The energy stays.
58
00:02:49,930 --> 00:02:51,000
It then inherits.
59
00:02:51,010 --> 00:02:53,770
It glazes or brightens.
60
00:02:54,190 --> 00:02:59,630
And the problem with that fluorescence
is the blinding part, so you
61
00:02:59,729 --> 00:02:59,769
Jennifer Reif: Yeah.
62
00:03:00,080 --> 00:03:01,040
John Willis: Everything's new.
63
00:03:01,040 --> 00:03:06,460
Oh, and you start forgetting what are
the parts that truly are new and what are
64
00:03:06,500 --> 00:03:11,060
the parts that are actually consistent
to all human technology behavior.
65
00:03:11,390 --> 00:03:14,680
And therein lies the,
the real hard problem.
66
00:03:15,027 --> 00:03:15,437
Jennifer Reif: Right.
67
00:03:15,467 --> 00:03:15,717
Right.
68
00:03:15,887 --> 00:03:19,037
And if I were to guess, I would
say we're still probably in that
69
00:03:19,037 --> 00:03:20,727
fluorescence of AI a little bit.
70
00:03:21,060 --> 00:03:21,310
John Willis: Yeah.
71
00:03:21,390 --> 00:03:21,580
Yeah.
72
00:03:21,780 --> 00:03:22,270
Oh, absolutely.
73
00:03:22,270 --> 00:03:22,470
Yeah.
74
00:03:22,470 --> 00:03:24,930
I think we're as deep into
it as you can get, right?
75
00:03:24,930 --> 00:03:26,590
We're probably right at the peak,
76
00:03:26,919 --> 00:03:27,169
Jennifer Reif: Okay.
77
00:03:27,360 --> 00:03:31,240
John Willis: Maybe getting to a little
bit more reality checks on certain things.
78
00:03:31,241 --> 00:03:31,631
Jennifer Reif: Sure.
79
00:03:31,740 --> 00:03:34,430
John Willis: But certainly, yeah, we're,
we're smack right in the middle right now.
80
00:03:34,761 --> 00:03:35,101
Jennifer Reif: Okay.
81
00:03:35,571 --> 00:03:40,791
Are there, I guess, maybe specific things
about the people or processes within
82
00:03:40,791 --> 00:03:46,111
an organization to help adapt to these
big shifts that you've seen work well?
83
00:03:46,706 --> 00:03:49,336
John Willis: The-- there's
always the early adopters, right?
84
00:03:49,336 --> 00:03:53,086
And I, I think early adopters are
great, but, y-you have to calibrate
85
00:03:53,206 --> 00:03:54,646
what your adoption looks like.
86
00:03:55,076 --> 00:03:57,636
And then, particularly as you
go up the sort of the food
87
00:03:57,636 --> 00:04:01,726
chain of regulatory controlled
businesses, you have to be slow.
88
00:04:01,826 --> 00:04:07,206
You can't just race, you can't do cargo
culting of what everybody else is doing.
89
00:04:07,406 --> 00:04:10,146
This knee-jerk reaction is
like, "We're behind." "Who are
90
00:04:10,146 --> 00:04:11,246
you behind?" "I don't know.
91
00:04:11,246 --> 00:04:12,346
We're behind," right?
92
00:04:12,665 --> 00:04:13,085
Jennifer Reif: Yeah.
93
00:04:13,316 --> 00:04:17,296
John Willis: What I'm trying to work
with the clients that will listen to me.
94
00:04:17,356 --> 00:04:22,156
So I'm an independent, and these
days, I'm semi-retired, but I do do
95
00:04:22,156 --> 00:04:25,366
work when a chief of staff or a CIO
calls me or has heard my presentation.
96
00:04:25,459 --> 00:04:25,739
Jennifer Reif: Okay.
97
00:04:26,766 --> 00:04:30,096
John Willis: And I come in and I, I don't
even take the work unless they're willing
98
00:04:30,116 --> 00:04:32,446
to listen to this idea of like, calm down.
99
00:04:33,176 --> 00:04:36,116
People ask me if their first question
is, "What would you recommend?
100
00:04:36,116 --> 00:04:39,026
Is your thirty, sixty, ninety-day
plan?" I'm like, "Don't have one."
101
00:04:39,636 --> 00:04:41,366
So you have to learn how to learn.
102
00:04:41,366 --> 00:04:44,616
And this is consistent with
every technology shift I've seen.
103
00:04:44,616 --> 00:04:45,506
I'm not smart.
104
00:04:45,556 --> 00:04:47,466
I've just been around for a long time,
105
00:04:47,765 --> 00:04:47,845
Jennifer Reif: Yeah.
106
00:04:47,926 --> 00:04:48,266
John Willis: Right?
107
00:04:48,496 --> 00:04:52,406
And, and so I think learning to learn,
and one of the things I've created
108
00:04:52,406 --> 00:04:54,456
is the idea of ideation hackathons.
109
00:04:54,669 --> 00:04:55,119
Jennifer Reif: Okay.
110
00:04:55,556 --> 00:04:58,516
John Willis: The developers are
a whole-- that's a whole sort of
111
00:04:58,526 --> 00:05:01,356
animal that I don't really focus on.
112
00:05:01,386 --> 00:05:03,996
Although I have a lot of experience,
obviously DevOps and all that stuff.
113
00:05:04,091 --> 00:05:04,541
Jennifer Reif: Sure.
114
00:05:05,296 --> 00:05:09,206
John Willis: But my concern is that
the organization wants to take all
115
00:05:09,206 --> 00:05:12,296
the executive assistants in the
finance department, in the supply
116
00:05:12,296 --> 00:05:16,416
chain, and in marketing, and they
wanna just throw them right into
117
00:05:16,416 --> 00:05:17,726
the middle of the ocean with AI.
118
00:05:17,961 --> 00:05:18,401
Jennifer Reif: Mm.
119
00:05:19,336 --> 00:05:22,736
John Willis: And, and I think there's
an organizational calibration.
120
00:05:22,926 --> 00:05:24,476
You have to learn how to learn.
121
00:05:24,486 --> 00:05:30,016
And so I think the ideation hackathon
is a great way to figure out how are
122
00:05:30,016 --> 00:05:32,136
these groups gonna work together?
123
00:05:32,236 --> 00:05:34,376
How are they gonna work
within the organization?
124
00:05:34,666 --> 00:05:36,776
People ask me, "What, what
tools you're gonna use?" None.
125
00:05:37,506 --> 00:05:38,556
"Well, what do you mean no tools?
126
00:05:38,596 --> 00:05:40,686
Can-- Why aren't you just
giving them, you know, Codex?
127
00:05:40,686 --> 00:05:41,976
Or why don't you give them Claude?
128
00:05:41,976 --> 00:05:45,556
Or why don't you give them, a Club
AI?" I'm like, "Because that's the
129
00:05:45,556 --> 00:05:47,296
worst possible thing you can do."
130
00:05:48,066 --> 00:05:53,456
Having a couple hundred executive
assistants all individually choosing the
131
00:05:53,456 --> 00:05:57,326
technical debt nightmare, and that-- I'm
not even getting into the risk problem.
132
00:05:57,541 --> 00:05:57,901
Jennifer Reif: Right.
133
00:05:58,186 --> 00:05:59,636
John Willis: So just slow down.
134
00:05:59,756 --> 00:06:02,826
Learn how to ideate in this new world.
135
00:06:03,066 --> 00:06:04,796
Learn as an organization hand off.
136
00:06:04,806 --> 00:06:05,456
And I've done these.
137
00:06:05,476 --> 00:06:06,086
They're brilliant.
138
00:06:06,506 --> 00:06:09,706
You let those teams come up with
the ideas, figure out what kind
139
00:06:09,706 --> 00:06:13,396
of things actually do make sense
with AI, what things don't.
140
00:06:13,706 --> 00:06:14,476
Because that's the other problem.
141
00:06:14,486 --> 00:06:17,016
If you just hand somebody and
said, "Go automate everything,"
142
00:06:17,316 --> 00:06:18,466
they're gonna automate everything,
143
00:06:18,687 --> 00:06:19,097
Jennifer Reif: Yeah.
144
00:06:19,837 --> 00:06:20,117
Yeah,
145
00:06:20,196 --> 00:06:20,286
John Willis: And
146
00:06:20,777 --> 00:06:23,437
Jennifer Reif: that's actually a
fair point too, because a lot of
147
00:06:23,437 --> 00:06:27,887
times outside of the developer space,
we don't think about how different
148
00:06:27,907 --> 00:06:31,097
adopting that technology looks like
for somebody who's non-technical.
149
00:06:31,457 --> 00:06:33,547
How do they know the guardrails?
150
00:06:33,557 --> 00:06:36,477
How do they know what to
trust and what not to?
151
00:06:36,477 --> 00:06:40,697
How do they know how to guide
the AI tools into the results and
152
00:06:40,697 --> 00:06:41,867
the answers they're looking for?
153
00:06:42,426 --> 00:06:45,716
John Willis: Yeah, no, a great example,
like in an ideation hackathon I did
154
00:06:45,716 --> 00:06:49,466
last year for a candy bar company,
th- this one team had a brilliant
155
00:06:49,466 --> 00:06:55,316
idea, and they didn't even understand
the consequences of data provenance.
156
00:06:55,559 --> 00:06:56,149
Jennifer Reif: Mm-hmm.
157
00:06:56,696 --> 00:06:57,106
John Willis: right?
158
00:06:57,299 --> 00:06:57,589
Jennifer Reif: Yeah
159
00:06:57,716 --> 00:07:00,946
John Willis: Again,, th- w- you want--
you don't wanna find that out later.
160
00:07:01,119 --> 00:07:01,499
Jennifer Reif: Right.
161
00:07:01,586 --> 00:07:01,736
John Willis: You
162
00:07:01,909 --> 00:07:03,329
Jennifer Reif: Yeah, that's
that's a big enough concern
163
00:07:03,329 --> 00:07:04,069
you need to know ahead of time.
164
00:07:04,236 --> 00:07:06,626
John Willis: You want to have an
advisement during the ideation phase
165
00:07:06,626 --> 00:07:08,796
to where "Here's what we're gonna
do." "Oh, well, we're gonna have
166
00:07:08,796 --> 00:07:11,406
to rethink that data right there."
167
00:07:11,591 --> 00:07:15,341
And we can spend an hour on what
can happen with, wrong data,
168
00:07:15,478 --> 00:07:15,768
Jennifer Reif: Yeah.
169
00:07:16,178 --> 00:07:16,598
Oh, yeah.
170
00:07:16,681 --> 00:07:17,111
John Willis: and inference.
171
00:07:17,111 --> 00:07:17,401
Yeah.
172
00:07:17,558 --> 00:07:18,128
Jennifer Reif: Absolutely.
173
00:07:18,838 --> 00:07:25,348
Okay, is there a big disconnect between
maybe leadership and teams or maybe just
174
00:07:25,348 --> 00:07:27,388
even within teams and departments then?
175
00:07:28,767 --> 00:07:29,067
John Willis: Yeah.
176
00:07:29,077 --> 00:07:30,797
This is across the board.
177
00:07:30,797 --> 00:07:32,637
There's so many… I'm pro-AI, right?
178
00:07:32,647 --> 00:07:35,117
So, the one thing I always, when
I'm presenting or something,
179
00:07:35,117 --> 00:07:38,297
I wanna say, "Let me get this
clear." I think this is phenomenal.
180
00:07:39,097 --> 00:07:42,297
When you think about pre-GPS
for getting anywhere.
181
00:07:42,344 --> 00:07:42,684
Jennifer Reif: Ja
182
00:07:42,787 --> 00:07:43,707
John Willis: I can't imagine.
183
00:07:43,727 --> 00:07:46,657
There'll be a point where I'm
like, I can't imagine not having
184
00:07:46,657 --> 00:07:49,747
these tools, as, as a companion.
185
00:07:49,857 --> 00:07:53,557
Again, I understand that it is
inference, it's probability, but, just
186
00:07:53,557 --> 00:07:54,977
don't eliminate critical thinking.
187
00:07:55,176 --> 00:07:55,496
Jennifer Reif: Right.
188
00:07:56,207 --> 00:07:57,577
John Willis: But, back to
the fluorescence, right?
189
00:07:57,577 --> 00:08:01,707
Getting blinded, and they're
throwing out all this sort of logic.
190
00:08:02,307 --> 00:08:04,107
I can go through a whole list, but audit.
191
00:08:04,457 --> 00:08:08,987
I'm ridiculously concerned
about what audit's gonna look.
192
00:08:09,007 --> 00:08:12,397
And worse, I go to a lot of conferences,
and I talk to people, really smart
193
00:08:12,397 --> 00:08:15,697
people giving presentations about
how well we're doing AI here.
194
00:08:15,997 --> 00:08:21,017
And I ask them simple questions about
evidence or evaluations or sort of
195
00:08:21,017 --> 00:08:25,957
notion of LLM as a judge or, or how
you's gonna manage, like There's a
196
00:08:25,977 --> 00:08:28,087
term called human on the loop, right?
197
00:08:28,097 --> 00:08:28,347
Instead of
198
00:08:28,362 --> 00:08:28,492
Jennifer Reif: Yeah.
199
00:08:28,827 --> 00:08:29,867
John Willis: In the loop, right?
200
00:08:29,947 --> 00:08:31,787
How are you gonna deal with agentics?
201
00:08:31,827 --> 00:08:36,757
Because there's a whole 'nother
level of containment, kill switches.
202
00:08:36,817 --> 00:08:40,327
It's like in a headlight
discussions, right?
203
00:08:40,327 --> 00:08:44,697
So, I think this, this push to go
fast, and everybody has to catch
204
00:08:44,747 --> 00:08:48,097
up, and what is our competitor
doing, and we can't lose out.
205
00:08:48,597 --> 00:08:51,907
Risk I've-- I didn't go to Black
Hat this year, but a friend of
206
00:08:51,907 --> 00:08:55,317
mine just got back and said, "Cyber
is completely broken," right?
207
00:08:55,647 --> 00:08:58,577
And I'm like, "Yeah, no,
risk is completely broken."
208
00:08:58,678 --> 00:08:59,138
Jennifer Reif: Yeah.
209
00:08:59,647 --> 00:09:04,657
John Willis: And everybody's trying to
overlay the old cyber and risk patterns
210
00:09:05,267 --> 00:09:11,357
on this new completely different, it's
like Von Neumann to quantum, right?
211
00:09:11,357 --> 00:09:12,827
Von Neumann architecture to quantum.
212
00:09:12,877 --> 00:09:14,967
It is, like you can't think the same.
213
00:09:15,262 --> 00:09:15,442
Jennifer Reif: Right.
214
00:09:15,777 --> 00:09:19,217
John Willis: And I see people
writing governance manifestos for AI.
215
00:09:19,217 --> 00:09:22,177
And it's like, okay, you just
added the word AI into the list
216
00:09:22,512 --> 00:09:22,842
Jennifer Reif: Yeah.
217
00:09:23,022 --> 00:09:27,142
Yeah, I think we're still trying to
figure out how to switch that mental
218
00:09:27,142 --> 00:09:31,592
model, and I think it goes back to
what you said earlier, where everybody
219
00:09:31,592 --> 00:09:35,422
feels behind, no matter how fast
or how far they might be moving.
220
00:09:35,802 --> 00:09:39,302
They just feel behind everything
else, and so therefore, I think some
221
00:09:39,302 --> 00:09:43,262
of this stuff kind of falls to the
wayside because they're just trying
222
00:09:43,262 --> 00:09:45,052
to keep up or stay ahead of the game.
223
00:09:46,057 --> 00:09:48,407
John Willis: The other thing I wanna be
writing a little bit more about where
224
00:09:48,407 --> 00:09:50,047
we are in this sort of fluorescence.
225
00:09:50,107 --> 00:09:53,907
We're at a point now where I think you
can, you can honestly go back and compare
226
00:09:53,907 --> 00:09:55,577
what happened in the early days of cloud.
227
00:09:56,117 --> 00:09:58,937
And in the early days of cloud, everybody
raced to put things on the cloud.
228
00:09:59,727 --> 00:10:01,777
And then there was this pullback, right?
229
00:10:01,787 --> 00:10:02,967
For obvious reasons.
230
00:10:02,977 --> 00:10:03,717
Oh, what do you mean?
231
00:10:03,757 --> 00:10:05,367
That probably shouldn't
have been in the cloud.
232
00:10:05,777 --> 00:10:06,927
You can't put that in the cloud.
233
00:10:06,957 --> 00:10:07,927
You could get sued.
234
00:10:07,977 --> 00:10:12,487
And I, I think we're gonna see
that sort of reactive process
235
00:10:12,487 --> 00:10:13,797
here hopefully happening.
236
00:10:13,937 --> 00:10:18,097
I think the real thing is when,
when internal auditing catches up.
237
00:10:18,717 --> 00:10:19,927
And that, that's what happened to cloud.
238
00:10:20,067 --> 00:10:24,517
Internal auditing, like, why
is that data in a public cloud?
239
00:10:24,918 --> 00:10:25,868
How did that get there?
240
00:10:25,878 --> 00:10:27,228
Or, or it got exposed.
241
00:10:27,578 --> 00:10:31,448
And so, we're gonna have to
learn how to, how to pull back.
242
00:10:31,498 --> 00:10:33,108
Cloud it was mostly about data.
243
00:10:33,138 --> 00:10:37,018
AI inference, there's a lot about
data, but there's sort of new risks.
244
00:10:37,518 --> 00:10:42,298
Cloud was could the service or data
running on the cloud be breached?
245
00:10:43,285 --> 00:10:43,535
Jennifer Reif: Mhm.
246
00:10:43,668 --> 00:10:46,358
John Willis: Or, or were you
protecting regulatory data?
247
00:10:46,668 --> 00:10:50,788
Now we got very the same, but now
we got the question of what goes
248
00:10:50,788 --> 00:10:55,608
into a public model, and, and
what's being kept, what's not kept.
249
00:10:55,608 --> 00:10:57,618
These are opaque systems that we have
250
00:10:57,668 --> 00:10:57,788
Jennifer Reif: Right.
251
00:10:57,858 --> 00:10:58,478
John Willis: no idea.
252
00:10:58,628 --> 00:11:00,447
Jennifer Reif: Yeah, it's
still a bit of a black box.
253
00:11:00,808 --> 00:11:01,118
John Willis: That's right.
254
00:11:01,128 --> 00:11:02,108
It is a black box.
255
00:11:02,247 --> 00:11:02,507
Jennifer Reif: Yeah.
256
00:11:02,568 --> 00:11:04,708
John Willis: No matter what they
tell you, it is a black box.
257
00:11:04,738 --> 00:11:05,157
The public ones
258
00:11:05,177 --> 00:11:05,507
Jennifer Reif: Yeah.
259
00:11:05,708 --> 00:11:06,058
John Willis: for sure.
260
00:11:06,267 --> 00:11:10,897
Jennifer Reif: Do you think this is
more of a people or a process type of
261
00:11:11,037 --> 00:11:12,717
function, or maybe a little bit of both?
262
00:11:13,494 --> 00:11:15,574
John Willis: Yeah, the, the
quick answer is both of course.
263
00:11:15,574 --> 00:11:18,334
But, there's another point I was
driving on that last conversation,
264
00:11:18,334 --> 00:11:22,694
which was, the one thing we learned in
the cloud, which was we had to figure
265
00:11:22,694 --> 00:11:28,234
out in, in the simplest form was,
green, yellow, red, blue applications.
266
00:11:28,363 --> 00:11:28,673
Jennifer Reif: Okay.
267
00:11:28,724 --> 00:11:30,164
John Willis: It was mostly around
data classification, but it
268
00:11:30,164 --> 00:11:31,334
was a service classification.
269
00:11:32,164 --> 00:11:38,244
So I'm suggesting for a large corporation,
don't put all your eggs in one basket.
270
00:11:38,244 --> 00:11:41,164
Don't do nothing or don't do everything,
271
00:11:41,247 --> 00:11:41,857
Jennifer Reif: Right.
272
00:11:42,017 --> 00:11:42,337
Sure.
273
00:11:42,344 --> 00:11:44,694
John Willis: Start thinking
about services and learn how to
274
00:11:44,694 --> 00:11:47,274
learn on the low-hanging fruit.
275
00:11:47,604 --> 00:11:49,204
So go as fast as possible.
276
00:11:49,214 --> 00:11:52,124
Have pilots, have groups
that are investing in trying
277
00:11:52,124 --> 00:11:53,494
out the different providers.
278
00:11:53,794 --> 00:11:55,084
Try open source ones.
279
00:11:55,344 --> 00:11:56,194
Try the open weight.
280
00:11:56,204 --> 00:11:59,974
Try the, the Gemini, the family from
Google is very powerful or OpenAI.
281
00:11:59,974 --> 00:12:00,274
Jennifer Reif: Yeah.
282
00:12:00,274 --> 00:12:01,854
John Willis: And, and spread your wings.
283
00:12:01,854 --> 00:12:03,354
You don't have to lock in right now.
284
00:12:03,429 --> 00:12:04,919
The game is changing so fast.
285
00:12:05,084 --> 00:12:05,224
Jennifer Reif: Yeah.
286
00:12:05,339 --> 00:12:07,899
John Willis: So but, but
figure out that spectrum.
287
00:12:08,709 --> 00:12:13,049
That's part of the take a deep
breath, to the coffee machine, get
288
00:12:13,049 --> 00:12:17,109
a really good cup of coffee, smell
it, taste it, then sit down and start
289
00:12:17,129 --> 00:12:18,166
thinking about what you're gonna do.
290
00:12:18,166 --> 00:12:21,969
So I actually tell when, when, a
chief of staff calls me, and they're
291
00:12:21,969 --> 00:12:25,459
like, "Oh, John, things are going
crazy." I go, I want you to calm down.
292
00:12:26,519 --> 00:12:29,259
I want you to go to the coffee
machine." I've, I've actually said this.
293
00:12:29,419 --> 00:12:32,909
"I want you to call me back in about
ten minutes or go down to the local
294
00:12:33,819 --> 00:12:35,619
barista, 20 minutes, and call me back."
295
00:12:35,726 --> 00:12:36,106
Jennifer Reif: Yeah.
296
00:12:36,179 --> 00:12:38,499
John Willis: So yeah, and,
and it really is that simple.
297
00:12:38,874 --> 00:12:39,104
Jennifer Reif: Okay.
298
00:12:39,794 --> 00:12:47,384
I, I've mentioned on the podcast how the
AI systems especially can be sensitive
299
00:12:47,384 --> 00:12:49,534
to configuration or setting changes.
300
00:12:49,614 --> 00:12:55,084
A new model comes out, something changes
on a language version or a library
301
00:12:55,084 --> 00:12:59,044
version somewhere, and that causes
kind of upset within your application.
302
00:12:59,044 --> 00:13:03,074
And of course, because these things
release so quickly, all technologies
303
00:13:03,074 --> 00:13:07,134
really now, then you feel like you're
in a constant state of breaking.
304
00:13:07,754 --> 00:13:14,764
How should people and teams and processes
maybe start thinking about that constant
305
00:13:14,764 --> 00:13:18,844
state of moving fast and breaking
things, but on a large scale now?
306
00:13:19,623 --> 00:13:21,243
John Willis: Yeah, this
is a hard problem, right?
307
00:13:21,243 --> 00:13:23,233
This is probably the hardest
problem there is right now.
308
00:13:23,263 --> 00:13:25,593
'Cause I think this-- th-that you
could tier this in a number of ways.
309
00:13:25,593 --> 00:13:27,703
One is the buy versus build conversation.
310
00:13:27,894 --> 00:13:28,274
Jennifer Reif: Okay.
311
00:13:28,333 --> 00:13:28,553
John Willis: Right?
312
00:13:28,553 --> 00:13:31,753
Which is a really interesting conversation
going on in large corporations.
313
00:13:31,763 --> 00:13:33,993
And then you can imagine
it has lots of flavors.
314
00:13:34,343 --> 00:13:35,553
Some wrong, some good.
315
00:13:35,843 --> 00:13:39,043
Some of the ones is I've got, like
I have four thousand developers.
316
00:13:39,043 --> 00:13:41,603
I don't wanna lay them off 'cause
they're integral to the business.
317
00:13:42,613 --> 00:13:43,753
What can I do with them?
318
00:13:43,963 --> 00:13:46,543
Could I replace an institutional product.
319
00:13:47,562 --> 00:13:47,822
Jennifer Reif: Mm.
320
00:13:47,961 --> 00:13:49,331
John Willis: I could set a reduction.
321
00:13:50,081 --> 00:13:53,621
I could actually get productive and
remove five million, ten million dollar.
322
00:13:53,871 --> 00:13:57,961
But then the question is,
what is the technology?
323
00:13:57,961 --> 00:13:59,991
What is the complete
system that you've got?
324
00:13:59,991 --> 00:14:01,931
So that's a whole conversation itself.
325
00:14:02,072 --> 00:14:06,362
I think there's this other idea, and this
one I, I kinda get, but again, I worry
326
00:14:06,362 --> 00:14:11,252
about the highly regulated businesses,
the banks, the healthcare, the ones
327
00:14:11,252 --> 00:14:13,512
where, high consequence environments.
328
00:14:14,392 --> 00:14:17,622
Is this idea that the
code is the new assembler.
329
00:14:17,622 --> 00:14:20,472
And I, I, I, I get it,
but that's an all-in.
330
00:14:21,343 --> 00:14:21,723
Jennifer Reif: Okay.
331
00:14:22,582 --> 00:14:24,542
John Willis: And in that world, I'm
just hearing really smart people and
332
00:14:24,582 --> 00:14:29,672
friends of mine who literally are like
top Java architects for banks say things
333
00:14:29,672 --> 00:14:33,462
like, "Code is the new assembler,"
meaning just like we didn't care.
334
00:14:33,482 --> 00:14:35,222
I started my career as
an assembler programmer.
335
00:14:35,522 --> 00:14:38,372
I don't, I don't write anything
in assembler today, right?
336
00:14:38,629 --> 00:14:39,099
Jennifer Reif: Right.
337
00:14:39,412 --> 00:14:41,782
John Willis: Maybe there's a
world where, and I'm not trying
338
00:14:41,782 --> 00:14:43,152
to get anybody riled up now.
339
00:14:43,212 --> 00:14:46,492
But maybe there's a world where you
don't ev- ever write code, and I'm
340
00:14:46,492 --> 00:14:50,032
seeing it, a lot of people right
now talk about no, no code review
341
00:14:50,032 --> 00:14:51,862
in six months, haven't written code.
342
00:14:52,272 --> 00:14:55,782
And in that world, you're still
gonna have to deal with your
343
00:14:55,782 --> 00:15:00,392
question, but maybe it's less because
it's dynamic and it's changing.
344
00:15:01,112 --> 00:15:06,662
And then I think the third tier
answer to this question is, back
345
00:15:06,662 --> 00:15:11,152
to understanding the service that
you're dealing with and how sensitive
346
00:15:11,152 --> 00:15:16,932
it is it to model changes and
particularly inference and probability.
347
00:15:17,229 --> 00:15:17,529
Jennifer Reif: Okay.
348
00:15:17,552 --> 00:15:18,732
John Willis: Because that's
the biggest problem, right?
349
00:15:18,732 --> 00:15:23,312
Is that I could have an application
working on Opus four out of whatever
350
00:15:23,812 --> 00:15:27,142
right now, perfect, and then I
upgrade the model, and it just
351
00:15:27,162 --> 00:15:28,932
completely gets different answers.
352
00:15:28,983 --> 00:15:29,443
Jennifer Reif: Right.
353
00:15:29,512 --> 00:15:31,802
John Willis: to a degree, maybe
it's three percent different,
354
00:15:31,812 --> 00:15:33,142
maybe it's four percent different.
355
00:15:33,442 --> 00:15:36,812
But again, depending what
the actual delivery mechanism
356
00:15:36,812 --> 00:15:38,772
is, could kill people, right?
357
00:15:38,945 --> 00:15:39,245
Jennifer Reif: True.
358
00:15:39,482 --> 00:15:42,492
John Willis: Here's another thing I, I
like about my career is we've learned
359
00:15:42,562 --> 00:15:45,422
to borrow ideas from other spaces.
360
00:15:45,422 --> 00:15:48,312
So in the DevOps world, we
looked at resilience and people
361
00:15:48,312 --> 00:15:49,752
look at critical safety and,
362
00:15:49,909 --> 00:15:50,309
Jennifer Reif: Okay.
363
00:15:50,482 --> 00:15:53,932
John Willis: And stuff like airplane
crashes and patients dying in a hospital
364
00:15:53,932 --> 00:15:56,832
due to some form of procedural problem.
365
00:15:57,069 --> 00:15:57,489
Jennifer Reif: Right
366
00:15:57,542 --> 00:15:59,955
John Willis: There's great wealth
of information there to learn from
367
00:16:00,005 --> 00:16:00,415
Jennifer Reif: Mhm.
368
00:16:00,622 --> 00:16:02,992
John Willis: But one is something
called ETO principle, which is the
369
00:16:02,992 --> 00:16:04,602
efficiency thoroughness trade-off.
370
00:16:05,029 --> 00:16:05,429
Jennifer Reif: Okay.
371
00:16:05,860 --> 00:16:09,390
John Willis: And so then thinking
that way about maybe I should
372
00:16:09,390 --> 00:16:11,640
use open weights and open source.
373
00:16:12,830 --> 00:16:18,590
There's gonna be a cost of doing that
the big provider that gives me all
374
00:16:18,590 --> 00:16:20,640
this stuff and is better at inference.
375
00:16:20,873 --> 00:16:21,233
Jennifer Reif: Okay.
376
00:16:21,810 --> 00:16:24,830
John Willis: And I'm gonna have to
maintain the models, but at least I can
377
00:16:24,830 --> 00:16:26,970
control those model and maintenance.
378
00:16:27,330 --> 00:16:29,960
There's this sort of balance
that you have to figure out.
379
00:16:29,960 --> 00:16:33,930
And this, this is probably one of
the harder problems right now that
380
00:16:33,930 --> 00:16:37,190
I think organizations, particularly
regulated industries, are facing.
381
00:16:37,543 --> 00:16:37,953
Jennifer Reif: Okay.
382
00:16:38,573 --> 00:16:45,050
So perhaps especially for very high
criticality use cases and industries
383
00:16:45,050 --> 00:16:50,390
and businesses looking at things
that they have more, I, I don't
384
00:16:50,390 --> 00:16:53,760
wanna say necessarily control, but at
least more fine-tuning capabilities.
385
00:16:55,115 --> 00:16:57,085
John Willis: Yeah, think about
the answer to the question.
386
00:16:57,105 --> 00:17:01,385
Or in agentic process that
literally you've given away
387
00:17:01,395 --> 00:17:02,895
some level of human control.
388
00:17:03,835 --> 00:17:08,965
What are the consequences
of degree of a wrong answer?
389
00:17:09,894 --> 00:17:09,904
Jennifer Reif: Mm.
390
00:17:10,505 --> 00:17:13,695
John Willis: That could be a cost, that
could be a human experience, right?
391
00:17:14,035 --> 00:17:18,035
That's where you have to decide the
cost of maintaining and building
392
00:17:18,695 --> 00:17:22,835
on top of open or open weight
models and supporting them yourself
393
00:17:23,180 --> 00:17:23,420
Jennifer Reif: Yeah.
394
00:17:23,915 --> 00:17:27,005
John Willis: And getting probably
lower efficacy, but more th-
395
00:17:27,295 --> 00:17:29,105
throughput or thoroughness, right?
396
00:17:30,476 --> 00:17:30,496
Jennifer Reif: Yeah.
397
00:17:31,695 --> 00:17:35,415
John Willis: Versus what is the risk
of using a very powerful public model.
398
00:17:36,295 --> 00:17:40,295
I think the good news too is I just read
the other day, and again, I just saw this.
399
00:17:40,295 --> 00:17:41,655
It's in my queue to follow up on it.
400
00:17:41,655 --> 00:17:48,595
But, Google is probably three in a
five-horse race on models, but they
401
00:17:48,595 --> 00:17:52,715
do have this hybridity where you can
run their public and private Gemini.
402
00:17:53,228 --> 00:17:53,828
Jennifer Reif: Oh, that's nice.
403
00:17:54,255 --> 00:17:58,125
John Willis: But, up until recently,
I'm almost certain OpenAI does not have
404
00:17:58,125 --> 00:18:01,105
that hybridity, and Claude did not.
405
00:18:01,115 --> 00:18:03,705
But I just saw something the other
day that implied that they might
406
00:18:03,775 --> 00:18:08,175
be allowing some customers to run
their own sort of private version.
407
00:18:08,175 --> 00:18:09,785
And again, that's all I know about it.
408
00:18:09,925 --> 00:18:11,795
But that is very encouraging.
409
00:18:11,822 --> 00:18:13,612
Because now you can get
the best of all worlds.
410
00:18:13,923 --> 00:18:14,923
Jennifer Reif: More things to test.
411
00:18:15,382 --> 00:18:18,012
John Willis: Well, you can
pin a very powerful model.
412
00:18:18,062 --> 00:18:19,622
You, you shouldn't have
got me on a podcast.
413
00:18:19,782 --> 00:18:20,792
I always go over.
414
00:18:21,083 --> 00:18:21,533
Jennifer Reif: No worries.
415
00:18:21,542 --> 00:18:24,972
John Willis: The cloud thing, the
metaphor we would use is, should you
416
00:18:25,822 --> 00:18:29,812
your own electricity creating grid, or
417
00:18:30,048 --> 00:18:31,308
use a public service?
418
00:18:31,524 --> 00:18:34,234
And that would be the argument for
why you might want to use cloud.
419
00:18:34,234 --> 00:18:36,914
Should you build your own infrastructure,
maintain your infrastructure when
420
00:18:36,914 --> 00:18:38,684
you could get all this as a utility?
421
00:18:38,743 --> 00:18:39,193
Jennifer Reif: Right.
422
00:18:39,294 --> 00:18:42,104
John Willis: A-and we're in that
same question now about inference.
423
00:18:42,293 --> 00:18:42,643
Jennifer Reif: Right.
424
00:18:43,184 --> 00:18:47,364
John Willis: What is the cost of managing
and building and maintaining your own
425
00:18:47,434 --> 00:18:50,024
open weight models that can keep up?
426
00:18:50,184 --> 00:18:51,874
And the open weight models
are pretty powerful.
427
00:18:52,774 --> 00:18:55,754
And I've said it multiple
times, it depends on what the
428
00:18:55,754 --> 00:18:56,924
service you're trying to do.
429
00:18:57,159 --> 00:18:57,419
Jennifer Reif: Sure.
430
00:18:57,644 --> 00:18:57,904
John Willis: So
431
00:18:58,809 --> 00:19:02,599
Jennifer Reif: Yep, the age-old, you
know, development answer is, it depends.
432
00:19:03,008 --> 00:19:04,448
John Willis: It depends, absolutely.
433
00:19:04,688 --> 00:19:05,558
More now than ever.
434
00:19:05,689 --> 00:19:06,989
Jennifer Reif: Yeah, yep, very true.
435
00:19:07,229 --> 00:19:11,999
Well, John, thank you so much for coming
onto the podcast and talking about your
436
00:19:11,999 --> 00:19:15,109
experience and the things that you've
learned through DevOps, and how that can
437
00:19:15,109 --> 00:19:18,159
be applied now throughout the AI era.
438
00:19:18,229 --> 00:19:19,729
John Willis: Yeah, no,
thank you for inviting me.
439
00:19:19,749 --> 00:19:21,619
Jennifer Reif: My other
question is for resources.
440
00:19:21,629 --> 00:19:24,349
It, I know you mentioned
your book as well.
441
00:19:24,349 --> 00:19:28,139
I'll link to that, and if there's anything
else you wanna share or things coming up?
442
00:19:28,620 --> 00:19:30,050
John Willis: Yeah, I got author portal.
443
00:19:30,120 --> 00:19:31,800
My author portal has all my books.
444
00:19:31,989 --> 00:19:32,359
Jennifer Reif: Great.
445
00:19:32,430 --> 00:19:35,220
John Willis: So… my LinkedIn is
the best way to get a hold of me.
446
00:19:35,270 --> 00:19:38,640
I do strategy advisement
for very select customers.
447
00:19:38,680 --> 00:19:42,550
And I always start with the innovation
hackathons is I think the best way
448
00:19:42,550 --> 00:19:44,210
to get started for an organization.
449
00:19:44,260 --> 00:19:44,740
Jennifer Reif: Perfect.
450
00:19:44,963 --> 00:19:45,813
John Willis: I'll get you
all those links, yeah.
451
00:19:45,920 --> 00:19:46,510
Jennifer Reif: Excellent.
452
00:19:46,683 --> 00:19:47,060
John Willis: Thanks so
much for inviting me.
453
00:19:47,060 --> 00:19:47,960
Jennifer Reif: Yeah, absolutely.
454
00:19:48,120 --> 00:19:48,960
John Willis: All right, take care.
455
00:19:49,010 --> 00:19:49,290
Jennifer Reif: Bye.
