TRANSCRIPT 🔗
WEBVTT
00:00:01.199 --> 00:00:03.040
Hey, I'm Reza Markovadi.
00:00:03.200 --> 00:00:04.639
Welcome back to QuantoBits.
00:00:05.040 --> 00:00:08.400
We watch Minions for the first time for Family Movie Night.
00:00:08.560 --> 00:00:15.359
If you haven't seen it, it's hundreds of little yellow helpers, all super eager, all building things at once.
00:00:15.599 --> 00:00:18.879
Mostly chaos with the odd thing that actually works.
00:00:19.120 --> 00:00:23.359
And it looked a lot like AI experimentation in some companies right now.
00:00:23.600 --> 00:00:25.679
I'm quite AI pilled.
00:00:25.839 --> 00:00:36.320
I've been pushing companies to experiment with AI as early as possible, and I still am, but I'm also starting to see the practical limits of how far that should go.
00:00:36.560 --> 00:00:43.039
So this week is about how to keep experimenting and make sure the experiments leave something behind.
00:00:43.280 --> 00:00:47.039
Here's what happens when experimentation becomes the strategy.
00:00:47.280 --> 00:00:55.200
People build a lot of things that demo well dashboards, command centers, tools that pull data together before a customer meeting.
00:00:55.359 --> 00:01:04.879
They look great, but you get only a few real automations, the kind that actually force a process to change and make a real dent in productivity.
00:01:05.120 --> 00:01:09.920
And very few of all these projects change and number the CFO tracks.
00:01:10.239 --> 00:01:12.879
And it's not just smaller companies.
00:01:13.200 --> 00:01:24.159
Accenture, the big consulting firm, surveyed 300 technology leaders at companies with more than a billion dollars in revenue and published the results on November 3rd.
00:01:24.400 --> 00:01:30.079
70% said they use AI mainly to make individual people more productive.
00:01:30.319 --> 00:01:39.840
Only 37% use AI agents, meaning AI that takes actions on its own, to run a piece of work from start to finish.
00:01:40.159 --> 00:01:43.200
Accenture's own summary was pretty blunt.
00:01:43.519 --> 00:01:48.319
Enterprise AI deployment is broad, value conversion is not.
00:01:48.560 --> 00:01:55.439
In other words, a lot of companies are using AI, and far fewer are turning it into business results.
00:01:55.680 --> 00:02:01.920
A mid-market company can't afford to spread AI that thin without knowing which bets pay off.
00:02:02.079 --> 00:02:05.200
There's the kind of beauty contest going on.
00:02:05.439 --> 00:02:10.560
The projects that win at PLOS are the ones that demo well and are easy to count.
00:02:10.800 --> 00:02:15.360
A new dashboard, hours saved, how many people use the tool this month?
00:02:15.520 --> 00:02:17.840
None of those is a result yet.
00:02:18.159 --> 00:02:20.240
Hours saved is a starting point.
00:02:20.400 --> 00:02:24.400
The result is what changes in the business because of those hours.
00:02:24.879 --> 00:02:29.520
Now, most of these experiments will never turn into lasting workflows.
00:02:29.680 --> 00:02:33.599
That's fine as long as each one stays cheap and contained.
00:02:33.840 --> 00:02:37.360
So here's a test I will use instead of counting experiments.
00:02:37.599 --> 00:02:42.240
Is your tenth experiment cheaper and safer to run than your first?
00:02:42.639 --> 00:02:47.520
If every team starts from scratch on its own, the answer is usually no.
00:02:47.840 --> 00:02:48.960
Here's why.
00:02:49.199 --> 00:02:59.439
Every disconnected experiment builds its own plumbing, its own way to signing, its own copy of the data, its own connection to the CRM.
00:02:59.599 --> 00:03:01.439
And some of that plumbing leaks.
00:03:01.680 --> 00:03:09.520
I spoke with someone at a software company recently, and they had found that about a tenth of their AI build was based.
00:03:09.680 --> 00:03:12.319
Some of it was forgotten API keys.
00:03:12.560 --> 00:03:16.800
Those are the credentials software uses to call an AI service.
00:03:17.039 --> 00:03:21.199
And these were still running up chargers for work nobody needed.
00:03:21.439 --> 00:03:25.120
Some was automated jobs nobody had switched off.
00:03:25.520 --> 00:03:27.360
Same company, different problem.
00:03:27.759 --> 00:03:33.520
Their AI search tool was answering revenue questions with numbers it found in team chat messages.
00:03:33.759 --> 00:03:39.199
So now every revenue requestion goes to one verified source and nothing else.
00:03:39.520 --> 00:03:45.199
Goldman Sachs Investment Bank went through its own version of reining in a free-for-all.
00:03:45.520 --> 00:03:54.319
For months, it let traders and bankers use AI coding tools, La Cloud Code, a coding assistant from the AI company Anthropic.
00:03:54.719 --> 00:04:04.400
Then in October, it updated its internal guidance to limit those tools mostly to its roughly 12,000 engineers, Bloomberg reported.
00:04:04.639 --> 00:04:12.879
A spokesperson said the change followed a period of experimentation that weighed productivity and risk management.
00:04:13.759 --> 00:04:16.480
So what should experiments leave behind?
00:04:16.800 --> 00:04:19.439
At its simplest, four things.
00:04:19.759 --> 00:04:22.480
First, who and what can act.
00:04:22.720 --> 00:04:30.800
That's sign in and permissions for people and for their AI agents, plus which AI models are approved for which tasks.
00:04:31.040 --> 00:04:38.800
So routine work doesn't default to the most expensive model and sensitive work doesn't go to one nobody's checked.
00:04:39.040 --> 00:04:41.360
Second, what the AI can trust.
00:04:41.600 --> 00:04:44.959
One approved source for each kind of company data.
00:04:45.199 --> 00:04:47.360
Third, what it can touch.
00:04:47.600 --> 00:04:52.879
Approved connections into company systems with limits on what each one can change.
00:04:53.120 --> 00:04:56.000
And fourth, what it did and what it cost.
00:04:56.319 --> 00:05:01.920
A record of every action checks on the quality of what it produced and cost tracking.
00:05:02.160 -->
Load full transcript
00:05:09.439
A failed experiment can still earn back part of its cost if the next team gets to use what it left behind.
00:05:09.600 --> 00:05:15.920
And that fourth piece, the record of actions and cost, is how you finally answer the CFO's question.
00:05:16.160 --> 00:05:20.240
What are we actually getting back in revenue or productivity revolut?
00:05:20.480 --> 00:05:24.720
The British fintech shows what this looks like at a bigger scale.
00:05:24.959 --> 00:05:40.319
Its head of operations and infrastructure, David Abbitball, told a conference in New York in October that engineers had been spending$1,000 to$10,000 a month each on AI coding tools, with little to show for it.
00:05:40.560 --> 00:05:44.000
So Revolute put strict budget limits in place.
00:05:44.240 --> 00:05:49.360
Then it moved to an open weight model, meaning one anyone can download and run.
00:05:49.519 --> 00:05:51.680
On computers it runs itself.
00:05:51.920 --> 00:05:57.839
By its own figures, that costs about a tenth as much per use as the commercial coding tools.
00:05:58.079 --> 00:06:02.000
Once AI was that cheap, engineers stopped rationing it.
00:06:02.240 --> 00:06:05.040
He says it improved coding time by 60%.
00:06:05.680 --> 00:06:13.279
That's Revolute's own number, and the register, the tech site that reported his talk, doesn't say how it was measured.
00:06:13.519 --> 00:06:18.399
Most mid-market companies won't run their own models, but the lesson travels.
00:06:18.560 --> 00:06:23.040
Once the infrastructure changed the economics, they could open up news.
00:06:23.519 --> 00:06:25.839
You don't have to build all four at once.
00:06:26.000 --> 00:06:29.360
Start with the one your last AI problem came from.
00:06:29.600 --> 00:06:35.519
Check what your sign-in provider and your existing tools already do before you build anything new.
00:06:35.759 --> 00:06:41.519
For trusted data, it can be as simple as writing down which source counts for which number.
00:06:42.079 --> 00:06:44.639
Now there's a fair objection to all this.
00:06:44.879 --> 00:06:48.879
Building infrastructure first can become its own beauty contest.
00:06:49.120 --> 00:06:55.839
A platform team builds shared pieces nobody uses, or the reviews get so heavy that nobody starts.
00:06:56.079 --> 00:06:59.120
And moving fast and breaking things has real value.
00:06:59.360 --> 00:07:05.120
It's often how you find the old hard-end processes that slow AI projects down.
00:07:05.519 --> 00:07:07.759
So don't build the four pieces up front.
00:07:07.920 --> 00:07:10.399
Build them as your experiments show you need them.
00:07:10.560 --> 00:07:15.519
When several teams hit the same wall, build that piece once for everybody.
00:07:15.920 --> 00:07:17.199
So where do I land?
00:07:17.439 --> 00:07:18.800
Keep experimenting.
00:07:19.040 --> 00:07:28.319
But be more intentional about which experiments you back because they take time and money, and you want people rowing in roughly the same direction.
00:07:28.639 --> 00:07:37.759
Back the ones that change a KPI you care about or that make the next experiment cheaper or safer by adding one of those four pieces.
00:07:38.000 --> 00:07:40.480
The rest only if you can afford them.
00:07:40.720 --> 00:07:45.680
People get excited about playing with a new technology, and that excitement fades.
00:07:45.920 --> 00:07:52.000
By the time it fades, you want AI already running, work that matters to the business.
00:07:52.319 --> 00:07:57.120
One quick hit, AI use is turning into a ration budget line.
00:07:57.279 --> 00:08:06.879
Even at Apple, Bloomberg reports that in October, Apple cut some of its own engineers' daily AI spending caps from$300 to$200.
00:08:07.519 --> 00:08:12.160
Staff outside engineering could get a cap as low as$25 a day.
00:08:12.399 --> 00:08:22.480
Apple didn't comment, if an engineer used that$200 cap in full, that's about$50,000 a year at 250 working days.
00:08:22.639 --> 00:08:26.720
So expect your AI bill to need the same management as your cloud bill.
00:08:26.959 --> 00:08:29.759
For after hours might pick from outside work.
00:08:30.000 --> 00:08:32.240
A book, or really an audiobook.
00:08:32.559 --> 00:08:37.440
Dark Matter by Blake Crouch, a science thriller from 2016.
00:08:37.759 --> 00:08:44.159
Jason Dessen is a Chicago physics professor who gave up research career to raise a family.
00:08:44.399 --> 00:08:51.440
He's abducted on the walk home and wakes up in a lab in a world where he never married his wife or had his son.
00:08:51.679 --> 00:09:00.159
Instead, that version of him built the box that opens into other versions of reality, like books about time travel and the many words idea.
00:09:00.480 --> 00:09:03.519
That every choice splits off a new reality.
00:09:03.759 --> 00:09:12.879
The execution was weaker than the idea though, the writing is average, the romance was too much for a sci-fi book, and the showdown at the end was overdone.
00:09:13.039 --> 00:09:15.039
But the takeaway landed with me.
00:09:15.200 --> 00:09:22.159
The grass always looks greener in other versions of ourselves, and the other sign may hold something you like less.
00:09:22.399 --> 00:09:23.600
Four stars.
00:09:23.919 --> 00:09:25.200
That's it for this week.
00:09:25.279 --> 00:09:33.759
The full issue with all the links and sources is in your inbox or at quanta bits newsletter.behive.com.
00:09:33.840 --> 00:09:36.320
I'm Reza Maracabadi, thanks for listening.
00:09:36.480 --> 00:09:37.440
See you next week.