WEBVTT
00:00:02.240 --> 00:00:06.320
Can you actually prove AI is working at your company?
00:00:07.040 --> 00:00:15.439
Sure, everything feels faster, and hopefully outputs have improved, and maybe more work is getting done.
00:00:16.239 --> 00:00:17.920
But is there ROI?
00:00:18.079 --> 00:00:22.000
And don't just give me vibes, give me receipts.
00:00:22.160 --> 00:00:26.879
What's the actual return on your investment for AI at your company?
00:00:27.359 --> 00:00:29.359
Chances are you can't answer that.
00:00:29.519 --> 00:00:32.240
And the reasons are actually very simple.
00:00:32.399 --> 00:00:39.759
It's because businesses are using the same digital transformation playbook they've always used when it comes to AI.
00:00:39.920 --> 00:00:44.799
And well, that playbook is useless in 2026.
00:00:45.039 --> 00:00:58.799
So there's a 99% chance you're not actually measuring ROI, but I'm going to simplify it for you on today's show and tell you why the ROI on AI debate is absolute nonsense.
00:00:58.880 --> 00:01:04.560
And I'm going to give you seven simple steps on how your company can fix it today.
00:01:04.879 --> 00:01:07.040
All right, let's get measuring.
00:01:07.359 --> 00:01:10.640
So uh here's what you'll learn on today's show.
00:01:10.719 --> 00:01:17.680
You're going to learn the basics of measuring Gen AI ROI and why most companies never actually do it.
00:01:17.840 --> 00:01:26.959
You're going to learn why a piece of MIT marketing, disguised as a study, misled markets in what five larger studies actually show.
00:01:27.280 --> 00:01:38.640
You're going to learn how to use a four-metric scorecard covering quality, cost, reliability, and risk, and how to run a seven-step evaluation sprint that actually solves your ROI.
00:01:39.760 --> 00:01:42.719
So, what the heck is this?
00:01:42.879 --> 00:01:47.599
Well, this is everyday AI number one, but this is our start here series.
00:01:47.840 --> 00:01:53.840
I didn't have a good answer when people first subscribe to the podcast and they're like, Jordan, there's 700 episodes.
00:01:53.920 --> 00:01:54.719
Where do I start?
00:01:55.200 --> 00:01:56.319
I'm like, I don't know.
00:01:56.480 --> 00:02:00.079
Well, now I do know you start here with the start here series.
00:02:00.159 --> 00:02:05.920
This is the essential podcast series to both learn the AI basics and to double down on your AI knowledge.
00:02:06.079 --> 00:02:09.439
So make sure to go to starthearseries.com.
00:02:09.520 --> 00:02:13.199
Uh, that's going to give you free access to our inner circle community.
00:02:13.360 --> 00:02:21.599
And then you can go in the Start Here Series space and listen to now all 11 episodes of this series.
00:02:21.759 --> 00:02:33.360
Uh, so last episode, which was a banger FYI, um, we talked about from AI chatbots to autonomous workers and how consumer AI has changed and what's next.
00:02:33.520 --> 00:02:35.199
So make sure you go uh check that out.
00:02:35.280 --> 00:02:40.159
It's episode 723, but uh volume 10 of the Start Here series.
00:02:40.319 --> 00:02:47.280
And today we're gonna be measuring ROI and tell you why your company is doing it wrong and these seven steps to fix it.
00:02:48.319 --> 00:02:49.520
Let's start here.
00:02:50.080 --> 00:02:53.599
Talked about this once or twice, but let's talk about this benchmark from OpenAI.
00:02:53.680 --> 00:02:55.199
It's called GDP Val.
00:02:55.919 --> 00:03:04.800
So this benchmark tests AI on real-world deliverables across 44 jobs in nine different GDP sectors.
00:03:05.039 --> 00:03:15.120
Essentially, this is a unbiased benchmark that pits AI models against expert humans, experts in their fields, right?
00:03:15.280 --> 00:03:20.639
They each have the same task, which is generally uh to produce something of value, right?
00:03:20.719 --> 00:03:24.080
That's why it's called GDP Val, to produce something of economic value.
00:03:24.240 --> 00:03:30.800
And then there's expert judges, expert human judges, and they don't know which output came from the AI.
00:03:30.879 --> 00:03:33.439
They don't know which output came from the expert human.
00:03:33.599 --> 00:03:44.159
And right now, top AI models either tie or win 70% of the time in these blind evaluations against professionals that have an average of 14 years of experience.
00:03:44.400 --> 00:03:47.039
I mean, my gosh, we should just end the episode now, right?
00:03:47.199 --> 00:03:50.960
There's your ROI on AI, right?
00:03:51.439 --> 00:03:52.719
Done in under four minutes.
00:03:52.800 --> 00:03:53.840
There's a lot of issues here.
00:03:53.919 --> 00:03:57.120
Number one, even what I just said, right?
00:03:57.360 --> 00:04:11.520
Do you know how to use today's models to, you know, one shot personalize, synthesize information, create, you know, spreadsheets, uh, PowerPoint decks, and use your own company's information without you having to do anything and verify outputs?
00:04:11.599 --> 00:04:18.240
Well, probably not, because companies aren't educating their people, and it's hard to keep up with uh today's capabilities of AI models.
00:04:18.480 --> 00:04:26.160
But what this did show us is that the whole debate on ROI is absolute rubbish.
00:04:26.319 --> 00:04:27.439
Yes, there we go.
00:04:27.600 --> 00:04:31.680
Uh, for uh listeners across the pond, look at me.
00:04:31.759 --> 00:04:33.920
I'm uh multilingual here.
00:04:34.000 --> 00:04:35.360
Uh it's garbage.
00:04:35.439 --> 00:04:46.160
This whole discussion on does artificial intelligence give you a return on your investment, it's it's honestly a very dumb question, if I'm being honest, right?
00:04:46.319 --> 00:04:48.959
Uh go go read the GDP BAO study.
00:04:49.120 --> 00:04:51.360
Um, I keep saying, oh, I'm gonna do a show soon.
00:04:51.439 --> 00:04:54.399
I I will actually do a show soon uh because it's worth it.
00:04:54.560 --> 00:05:04.240
So not only did it show that, well, today's AI models are exponentially better than expert humans when judged by other expert humans, right?
00:05:04.399 --> 00:05:08.399
But it said that in many tasks they do it a hundred times faster.
00:05:08.639 --> 00:05:20.480
So again, I am not a mathematician, but if the AI model is the same or better 70% of the time and it's a hundred times faster, that's the math.
00:05:20.639 --> 00:05:27.279
So why are we still even debating this concept of is there return on investment?
00:05:27.439 --> 00:05:32.000
Well, that's because of an infamous piece of marketing from MIT.
00:05:32.720 --> 00:05:39.040
So uh, and yes, I'm calling it a piece of marketing or study in very heavy quotes, right?
00:05:39.120 --> 00:05:44.639
Like the uh uh what is that, the the Austin Powers uh, you know, a billion dollar.
00:05:44.959 --> 00:05:47.199
Yeah, huge quotes on the word study.
00:05:47.360 --> 00:05:47.839
Uh, right.
00:05:47.920 --> 00:05:57.279
So MIT claimed uh in their wildly viral study that 95% of enterprise AI pilots delivered zero ROI.
00:05:57.360 --> 00:06:01.279
And this was in August of 2025, right?
00:06:01.519 --> 00:06:03.600
And I mean the markets moved, right?
00:06:03.759 --> 00:06:09.040
Nvidia stock tanked three and a half points, Palantir went down uh nearly nine percent.
00:06:09.199 --> 00:06:23.360
There's actually billions of dollars of losses across the uh not just the tech sector, just the business sector, because right, AI has essentially propped up an otherwise struggling uh US and at times global economy.
00:06:23.600 --> 00:06:28.480
Um, and then you know, this study came out and said, Oh, AI's, you know, there's no ROI.
00:06:28.639 --> 00:06:31.199
Well, uh most people, right?
00:06:31.439 --> 00:06:34.000
As a former journalist, I understand this, right?
00:06:34.079 --> 00:06:41.040
There is one study that came out, and then essentially every other uh journalist hit the copy and paste, uh, you know, more or less.
00:06:41.199 --> 00:06:47.519
Hey, that headline sells, uh, you know, 95% of AI pilots don't uh produce any ROI, right?
00:06:47.600 --> 00:06:48.000
That's great.
00:06:48.079 --> 00:06:48.959
Let's go ahead and run that.
00:06:49.040 --> 00:06:53.199
But I don't think any journalists actually read the story, or very few actually did, uh, right.
00:06:53.279 --> 00:06:58.399
That's because this quote unquote study was based on 52 qualitative interviews.
00:06:58.560 --> 00:07:01.360
There's no quantitative piece to that 95%.
00:07:01.759 --> 00:07:04.160
And it was uh directionally, right?
00:07:04.240 --> 00:07:05.759
Uh, this was a directional study.
00:07:05.920 --> 00:07:07.519
So it's more or less this was a vibe.
00:07:07.600 --> 00:07:08.560
This was a vibe study.
00:07:08.720 --> 00:07:12.480
I'm gonna talk to 52 people, you know, nothing quantitative.
00:07:12.560 --> 00:07:14.000
I'm just gonna give my vibe on this.
00:07:14.079 --> 00:07:21.759
But the stat came from 52 qualitative interviews that measured PL impact in under six months, which there's no such thing, right?
00:07:21.920 --> 00:07:26.800
When you talk about digital transformation, go back and look at the internet, right?
00:07:27.199 --> 00:07:33.040
You could have shown the PL impact in under six months on using the internet, right?
00:07:33.439 --> 00:07:37.680
Because if you could have, you could apply that exact same methodology to AI.
00:07:37.839 --> 00:07:42.240
Anyways, well, why this is a piece of marketing and not actually just uh a bad study.
00:07:42.399 --> 00:07:44.160
I would call it a bad study, but it wasn't.
00:07:44.240 --> 00:07:46.959
It was marketing because in the end they were selling their own product.
00:07:47.040 --> 00:07:52.480
They said, Hey, uh, right, all these AI pilots fail because you're not using our agentic AI product.
00:07:52.560 --> 00:07:55.040
And they were, you know, selling access to it.
00:07:55.279 --> 00:07:58.000
Um, yeah, I'll stop there.
00:07:58.160 --> 00:08:10.720
If you really want the down low on that one, make sure to go listen to episode 597, where I, like a human, uh read it, like a human with a brain, uh, read the study multiple times, and it's uh, you know, laughable.
00:08:10.879 --> 00:08:13.839
Anyways, if you look at the real data, right?
00:08:14.000 --> 00:08:20.639
Companies that talked to thousands and got quantitative data, obviously the ROI for AI is there.
00:08:20.720 --> 00:08:21.199
Uh, right.
00:08:21.279 --> 00:08:29.040
The IDC, the International Data Corporation, found a$3.70 return for every dollar invested in AI.
00:08:29.279 --> 00:08:37.200
Wharton, a three-year study, found that 74% of enterprises report positive ROI on gen AI.
00:08:37.440 --> 00:08:48.240
Google Cloud's 2025 annual AI study found 74% of executives reported achieving ROI just within the first year of generative AI deployment.
00:08:48.559 --> 00:08:57.600
Also, Deloitte's 2025 AI survey found 84% of those investing in AI said that they were gaining ROI.
00:08:58.000 --> 00:09:07.120
Um, this I think the this this debate is still going on because humans are innately lazy.
00:09:08.399 --> 00:09:08.879
Right?
00:09:09.039 --> 00:09:16.720
That's the reason why we don't want to actually sit and measure, right?
00:09:17.039 --> 00:09:26.639
Again, my my background, I have a little bit of marketing and advertising background, so I understand the importance of measuring something before and after to see if there's an impact, right?
00:09:26.879 --> 00:09:30.240
Running millions of dollars of ads for clients over the years.
00:09:30.320 --> 00:09:32.799
There's this thing called return on ad spend, right?
00:09:32.879 --> 00:09:37.120
You see how much uh you know how much money was spent on ads.
00:09:37.360 --> 00:09:43.200
You have to be able to attribute and track the revenue that it brought in, and then you do some simple math, right?
00:09:43.360 --> 00:09:44.480
Return on ad spent.
00:09:44.559 --> 00:09:50.799
Well, there is also a simple equation that I'm gonna give you guys here in a little bit, the same thing, return on AI.
00:09:51.360 --> 00:10:09.120
So here's the dirty little secret because you're probably thinking, okay, Jordan, well, if every company's using AI, well, why isn't everyone just seeing exponential growth?
00:10:09.360 --> 00:10:16.320
Well, uh, I think you would see that a little bit more if only 5% of uh businesses were using AI, right?
00:10:16.399 --> 00:10:20.639
But most studies say more than 95 or 97% of enterprises.
00:10:20.799 --> 00:10:24.879
So uh the playing field has just gone up, right?
00:10:25.200 --> 00:10:30.000
Uh the minimum entry is no longer like, oh, you're using AI.
00:10:30.159 --> 00:10:30.639
That's great.
00:10:30.720 --> 00:10:32.000
That's such a differentiator.
00:10:32.159 --> 00:10:34.399
No, uh, no, it's you're using AI.
00:10:34.559 --> 00:10:34.799
Okay.
00:10:34.879 --> 00:10:36.960
I've been saying since 2023.
00:10:37.120 --> 00:10:39.759
It's the same thing as, oh, our company's using the internet.
00:10:40.159 --> 00:10:41.759
So, right.
00:10:42.000 --> 00:10:44.159
Uh, but here's what's happening.
00:10:45.759 --> 00:10:47.679
Workers are pocketing it.
00:10:47.840 --> 00:10:57.279
That's where the ROI is going, uh, right, because true productivity and true ROI, well, it requires results-driven metrics, not time-based management.
00:10:57.519 --> 00:11:01.039
Most companies haven't, you know, gotten this figured out yet.
00:11:01.200 --> 00:11:18.080
Uh, and I think a big portion of this is that remote and hybrid workers are using AI, whether it's approved AI or uh, you know, whether it's uh, you know, AI sprawl, uh, you know, dark AI, whatever you want to call it, and they're just pocketing it.
00:11:18.159 --> 00:11:20.480
It's this invisible productivity, right?
00:11:20.559 --> 00:11:22.960
I've literally talked to countless people.
00:11:23.120 --> 00:11:34.240
I won't say hundreds because I don't know if it's actual 200 or more, but I've easily talked to more than a hundred people over the past few years in very successful, right?
00:11:34.320 --> 00:11:40.159
Even one of my um, you know, one of my good friends uh told me this like uh maybe one or two years ago.
00:11:40.960 --> 00:11:51.919
Pretty, pretty high up uh at a company that was uh public company when he worked there, said he automated about 95% of his job with AI, right?
00:11:52.080 --> 00:11:53.519
No one knew because he's at home.
00:11:53.600 --> 00:11:55.200
I'm like, what are you doing with all this time?
00:11:55.279 --> 00:11:58.480
He's like, ah, you know, chelling golfing a lot, uh, right?
00:11:58.559 --> 00:11:59.919
That's the reality.
00:12:00.639 --> 00:12:07.600
You know, uh so many companies, so many employees, I think, are just pocketing this time savings, right?
00:12:07.679 --> 00:12:11.519
As as we've gone to this uh, you know, remote hybrid workforce.
00:12:11.600 --> 00:12:18.159
And I think that's one of the reasons why last year uh so many companies had this, you know, big return to work push.
00:12:18.320 --> 00:12:26.799
Uh and right now, according to workday, 89% of organizations haven't even updated their roles to reflect AI, right?
00:12:27.039 --> 00:12:35.759
So they're still using uh you know 2026 tools uh inside of job structures made 10 years ago.
00:12:36.000 --> 00:12:36.720
Um, right.
00:12:37.039 --> 00:12:38.559
So why does that matter?
00:12:38.799 --> 00:12:48.000
Well, again, hiring and roles in today's enterprise are still pre-AI.
00:12:48.480 --> 00:12:58.399
So what that generally means, well, expectations or output is measured in a pre-AI way, which isn't necessarily the right thing, right?
00:12:58.559 --> 00:13:08.480
And in the same way that I've encouraged um you all for years to not upskill or reskill, uh, that's that's a waste of time when it comes to AI, right?
00:13:08.639 --> 00:13:11.840
You need to unlearn and rebuild, been saying that for a long time.
00:13:12.159 --> 00:13:19.120
Um departments and companies need to do the exact same thing, right?
00:13:19.440 --> 00:13:29.039
You can't just, oh, let's add a little bit of AI to this job role, or uh, you know, hey, let's let's try to think a little more uh AI natively here at this company.
00:13:29.200 --> 00:13:34.320
No, you you gotta tear it apart in the same way that job roles haven't changed, right?
00:13:34.480 --> 00:13:36.720
That means outputs aren't going to change either.
00:13:36.960 --> 00:13:44.159
But if you're using AI, if you've trained your people on AI, if they have the right AI, they're just banking that productivity.
00:13:44.320 --> 00:13:47.759
So here's how finding ROI is done.
00:13:47.919 --> 00:13:49.200
It's simple.
00:13:49.519 --> 00:13:51.840
Uh ROI can mean a lot of different things.
00:13:51.919 --> 00:13:55.440
It can a lot of people just think, oh, that means you're bringing in more revenue.
00:13:55.600 --> 00:13:58.960
No, I think mainly it's time saved, right?
00:13:59.039 --> 00:14:04.480
But I think ROI also means just cost reduced, revenue increased, or risk avoided.
00:14:04.559 --> 00:14:06.159
It's not just prompts sent, right?
00:14:06.240 --> 00:14:08.240
I think that's what a lot of people they're looking at.
00:14:08.320 --> 00:14:15.279
Uh the number of hours, uh, you know, your utilization rate if you're using, you know, an enterprise tool like ChatGPT, right?
00:14:15.360 --> 00:14:20.559
Everyone's like, oh, you know, we need to increase our utilization from you know eight percent to 20%, right?
00:14:20.799 --> 00:14:26.159
And then companies, you know, hire us to, you know, train hundreds or thousands of employees.
00:14:26.240 --> 00:14:30.960
And it's like, okay, utilization is not uh the metric that pushes it.
00:14:31.039 --> 00:14:32.960
It's are you saving time?
00:14:33.200 --> 00:14:36.639
Are you um increasing revenue and avoiding risk?
00:14:36.799 --> 00:14:39.120
Uh the formula, well, simple math.
00:14:39.200 --> 00:14:42.960
It's the time saved or uh increased revenue.
00:14:43.039 --> 00:14:44.320
Uh, but let's look at time saved.
00:14:44.399 --> 00:14:51.919
So it's just time saved multiplied by fully loaded hourly rates minus any any monthly AI subscription costs.
00:14:52.639 --> 00:14:54.240
Simple formula, right?
00:14:54.559 --> 00:15:04.080
Um, and right now boards and CFOs just care about cost per task, uh cost per task throughput and error rates, right?
00:15:04.320 --> 00:15:07.679
Not usage dashboards, not utilization.
00:15:09.279 --> 00:15:16.000
But we're seeing the same reasons for failure, right?
00:15:16.240 --> 00:15:18.320
Still years later, right?
00:15:18.559 --> 00:15:34.960
Um, through my own conversations and conversations with others, I can say when it comes to measuring or not measuring ROI, I still think that there's five patterns or five main reasons why either individuals or departments aren't able to do this.
00:15:35.120 --> 00:15:37.840
So, number one, not having a pre-AI baseline.
00:15:38.000 --> 00:15:41.759
Number two, you know, doing these slow year-long pilots.
00:15:41.840 --> 00:15:43.279
That's a recipe for disaster.
00:15:43.440 --> 00:15:49.039
Number three, celebrating one lucky run and making that your AI strategy.
00:15:49.200 --> 00:15:52.000
Number four, when vanity metrics become the focus.
00:15:52.159 --> 00:16:01.120
And number five, shiny object syndrome, pulling focus uh away before real company-wide implementation can actually start, right?
00:16:01.279 --> 00:16:05.519
Oh, you get a little, you get a little ground, you show ROI, and then you're like, oh, what about this model?
00:16:05.600 --> 00:16:05.919
What about this?
00:16:06.000 --> 00:16:06.480
What about this?
00:16:06.559 --> 00:16:07.279
What about this, right?
00:16:07.360 --> 00:16:10.399
Before you actually, you know, uh implement it company wide.
00:16:10.559 --> 00:16:12.960
And these aren't technology problems, right?
00:16:13.279 --> 00:16:17.440
This isn't standard digital transformation.
00:16:19.759 --> 00:16:28.879
These are we as working human beings, as knowledge workers, uh, we don't have a playbook to follow, right?
00:16:28.960 --> 00:16:33.759
So all we've been doing is following the same playbook that we've always done, right?
00:16:34.240 --> 00:16:36.080
Those slow pilots.
00:16:36.320 --> 00:16:39.440
You know, you get one thing right and you're like, oh, this is what we do now.
00:16:39.519 --> 00:16:41.279
It's the wrong way to do this.
00:16:41.919 --> 00:16:44.639
The other big reason, well, training.
00:16:45.279 --> 00:16:45.759
Right.
00:16:45.919 --> 00:16:49.519
Uh the training actually leads to those five other failures.
00:16:49.600 --> 00:16:53.679
So it's more of a of an underlying uh foundational issue, right?
00:16:53.759 --> 00:17:02.159
But right now, 49% of enterprise leaders cite recurring, uh, sorry, cite recruiting AI talent as their single biggest challenge.
00:17:02.879 --> 00:17:03.440
All right.
00:17:03.840 --> 00:17:07.920
AI, you need documented workflow steps and company data.
00:17:08.079 --> 00:17:14.319
And that process documentation is part of the ROI investment, right?
00:17:14.720 --> 00:17:20.720
Education, training, ongoing learning, it is required.
00:17:21.200 --> 00:17:21.759
Uh, right.
00:17:21.920 --> 00:17:42.480
So many, I think when um when AI, uh when an AI strategy or actual AI implementation looks to looks like you know, pushing top down, you know, either C-suite or board pushing, you know, a certain tool top down and not training or educating, that's when it's so easy to fail, uh, right.
00:17:43.920 --> 00:17:47.200
I use AI all day, every day.
00:17:47.440 --> 00:17:49.359
And I'll I'll I'll tell you this.
00:17:49.599 --> 00:17:53.279
If I take two weeks off, I'm gonna fail.
00:17:53.920 --> 00:17:56.400
That's the that's the pace, right?
00:17:56.640 --> 00:18:01.839
People are always like, okay, well, Jordan, you clearly don't know what you're saying here.
00:18:02.000 --> 00:18:06.079
If AI is a hundred times faster, well, shouldn't we just cut all these jobs?
00:18:06.160 --> 00:18:08.079
No, I think jobs are gonna look very different.
00:18:08.240 --> 00:18:10.640
I've been on record as saying this since the very first episode.
00:18:10.799 --> 00:18:17.759
AI is going to uh, I think, quote unquote, take away more jobs than it will create, but it will create many roles that we don't even understand.
00:18:17.839 --> 00:18:30.000
I've always said most enterprise organizations, they need a lot of people like me, that all they do all day is they keep up with AI and they apply it to that business, right?
00:18:30.240 --> 00:18:33.440
Very few businesses have that because they're like, okay, that's a crazy role.
00:18:33.599 --> 00:18:45.440
We're not gonna let you know, Bill over in IT or you know, Deborah over in marketing just spend all day uh, you know, tinkering with AI and seeing how to apply it.
00:18:45.519 --> 00:18:46.160
No, right?
00:18:46.240 --> 00:18:48.799
No, but yes, that's exactly what you should be doing.
00:18:48.960 --> 00:18:58.079
Uh, and here's all right, uh, this is usually something I think we only uh give away to companies that uh pay us a couple dollars.
00:18:58.240 --> 00:19:00.400
But let me just go ahead and give you some of the secrets here.
00:19:00.480 --> 00:19:00.799
All right.
00:19:00.880 --> 00:19:02.640
Here's an acronym, BASE.
00:19:02.960 --> 00:19:04.240
You need a base.
00:19:04.559 --> 00:19:05.839
This is the baseline rule.
00:19:06.000 --> 00:19:12.480
So this is BASE stands for baseline assessment of standard execution pre-AI.
00:19:12.720 --> 00:19:17.920
All right, so before AI touches any workflow, you got to get that base, right?
00:19:18.000 --> 00:19:24.880
That's your time multiple employees completing that exact task without it.
00:19:25.440 --> 00:19:26.480
You need to time it.
00:19:26.559 --> 00:19:33.039
You need to record the average time, the error rate, the rework cycles, and cost per completed task as the baseline.
00:19:33.599 --> 00:19:37.359
You can't go back and collect this retroactively, right?
00:19:37.440 --> 00:19:45.759
Because usually what happens, well, if you've already, you know, sprinkled AI in the process and you've been doing it for uh a year, right?
00:19:45.920 --> 00:19:55.279
Let's say it's a 10-step process and hey, number two and uh step two and step six, we you know uh we molded those a little bit around AI, right?
00:19:55.359 --> 00:19:57.279
You it's it's it's too late, right?
00:19:57.599 --> 00:20:02.160
You need to do it before you implement AI.
00:20:02.319 --> 00:20:09.839
You need to measure how long it takes humans to go through and do these certain projects or do these certain tasks, right?
00:20:10.000 --> 00:20:14.480
Think of it like an internal GDP valve, right?
00:20:14.640 --> 00:20:31.359
You are gonna go through and have a human with no AI go through and do this task, document every single step of the process painstakingly, because you need to see okay, uh what other humans need to be involved in this?
00:20:31.680 --> 00:20:33.039
Are there other meetings?
00:20:33.279 --> 00:20:35.279
What about the person checking the work?
00:20:35.440 --> 00:20:37.680
What about the communication going back and forth?
00:20:37.759 --> 00:20:41.519
You need to measure it all and document it all every single step.
00:20:42.480 --> 00:20:43.920
And then what's the completion rate?
00:20:44.079 --> 00:20:45.039
What's the error rate?
00:20:45.119 --> 00:20:46.400
What does quality look like?
00:20:46.799 --> 00:20:53.039
You need to establish those baselines and then redo the entire process, right?
00:20:53.200 --> 00:20:55.200
Blow it up, measure it first.
00:20:55.920 --> 00:20:57.279
That's your base, right?
00:20:57.759 --> 00:20:59.759
Baseline assessment of standard execution.
00:21:00.720 --> 00:21:06.160
That's your base, blow it up, do the same thing with AI, right?
00:21:07.039 --> 00:21:11.039
Not your first iteration, iterate on it, and then you measure.
00:21:11.920 --> 00:21:21.680
And that number, right there, you need that because that is the base, quite literally, for how you're going to ultimately measure your ROI.
00:21:22.079 --> 00:21:25.359
So now let's give you that seven-step guide.
00:21:25.599 --> 00:21:26.880
All right, I'm delivering.
00:21:27.039 --> 00:21:27.839
Here we go.
00:21:30.880 --> 00:21:41.440
Well, let me just uh bring my notes up here because you know, in typical live stream, unedited, unscripted fashion, uh, I didn't put my slide up here with my uh seven steps.
00:21:41.519 --> 00:21:43.359
But don't worry, I got my notes.
00:21:45.200 --> 00:21:46.160
All right, there we go.
00:21:46.319 --> 00:21:47.359
Notes on the screen.
00:21:47.440 --> 00:21:51.359
So here's the seven step blueprint, right?
00:21:51.519 --> 00:21:53.359
Step one, we already talked about it.
00:21:53.519 --> 00:21:57.279
That's the uh well, actually comes a little bit before the base.
00:21:57.519 --> 00:21:59.759
So step one, you have to define, you have to define.
00:22:00.319 --> 00:22:02.400
What the heck is it you're doing, uh, right?
00:22:02.559 --> 00:22:04.000
And you have to be very rigid.
00:22:04.079 --> 00:22:08.400
You can't be flexible because you need to get um an accurate before and after.
00:22:08.480 --> 00:22:16.319
So that's defining the rubric uh rubric, uh, defining the success criteria and KPIs before you even begin testing.
00:22:16.400 --> 00:22:19.119
And then step two, like I talked about, that's the base.
00:22:19.279 --> 00:22:21.599
You need to measure the human baseline.
00:22:21.680 --> 00:22:26.799
And then you're gonna uh, you know, time multiple employees doing that and then record the averages.
00:22:26.960 --> 00:22:29.680
Uh step three, get messy, right?
00:22:29.839 --> 00:22:37.599
Uh you have to build 20 to 40 real messy work examples, including drift cases, right?
00:22:37.680 --> 00:22:41.359
These aren't easy things for humans or for AI.
00:22:41.599 --> 00:22:42.000
All right.
00:22:42.079 --> 00:22:47.279
And then step four, you need to configure the exact production workspace.
00:22:47.359 --> 00:22:50.960
It's the same plan, the same model, and the same uh permissions.
00:22:51.119 --> 00:22:55.920
So you're not, you know, flip-flopping uh, you know, between different people, different accounts.
00:22:56.160 --> 00:22:56.960
No, right?
00:22:57.039 --> 00:22:59.920
It has to be controlled like any experiments.
00:23:00.000 --> 00:23:06.799
Uh, and it has to be repeatable and scalable, the exact same criteria when you're taking this to a production run.
00:23:06.960 --> 00:23:11.440
Uh, step five, you're gonna run every three times.
00:23:11.759 --> 00:23:12.079
Okay.
00:23:12.640 --> 00:23:20.559
So, sorry, run each test three times with memory off and also require proof artifacts.
00:23:20.720 --> 00:23:21.920
So, here's what I mean by that.
00:23:22.079 --> 00:23:23.359
We're doing this on the front end.
00:23:23.519 --> 00:23:23.839
Okay.
00:23:24.160 --> 00:23:34.799
Um, if you listen, if you've listened to this podcast at all, you know I'm a big believer in bringing as many of your processes over uh to front-end large language models.
00:23:34.880 --> 00:23:40.720
We actually did a dedicated episode uh on that in the start here series, talking about an AI operating system.
00:23:40.880 --> 00:23:48.960
So, yeah, this is all going to be doing things on as an example, chat gpt.com, claw.ai, gemini.google.com, right?
00:23:49.039 --> 00:23:59.839
But doing it with memory turned off and doing it usually in a temporary chat if your uh you know AI operating system of choice uh lends itself to that.
00:24:00.240 --> 00:24:00.799
All right.
00:24:01.119 --> 00:24:10.880
So essentially there, you have steps three through six is that's your internal GDP valve, right?
00:24:11.200 --> 00:24:15.119
Uh you have to get the the correct use uh use cases.
00:24:15.359 --> 00:24:19.440
Um, and then from there uh you grade blind, right?
00:24:19.599 --> 00:24:20.720
So same thing.
00:24:21.039 --> 00:24:30.400
You have the the series of people do it uh three times, those 20 to 40 use cases using the same AI model.
00:24:30.640 --> 00:24:36.559
Uh, and then you have humans do those same exact things as well, multiple humans in the same way.
00:24:36.640 --> 00:24:44.079
Uh I would suggest, you know, three different times um running the same case and then three different sets of humans.
00:24:44.640 --> 00:24:53.279
And then you grade blind, you standardize the output format, you have to agree on, you know, what's a pass, what's a fail, what's the grading scale, et cetera.
00:24:53.440 --> 00:24:58.079
But then at that point, well, you have the input and the output, right?
00:24:58.240 --> 00:25:12.960
At that point, after step six, when you can have your grading criteria, uh, you know, you run through and you do the test three times, um, you know, those 20 to 40 use cases, three times uh in a large language model, memory off, uh, temporary chat.
00:25:13.119 --> 00:25:17.119
You have your your grading rubric, you have your humans do it, uh, right?
00:25:17.279 --> 00:25:20.480
The same 20 to 40 tasks, three different humans.
00:25:21.599 --> 00:25:22.720
You have it right there.
00:25:22.880 --> 00:25:24.319
You have the time, right?
00:25:24.480 --> 00:25:30.799
You multiply uh the time that it takes the humans to do it on the AI side, all right.
00:25:31.039 --> 00:25:42.799
Minus out, uh, you know, so take, let's say it's a hundred human hours, take their hourly rate, times it, minus the uh cost of whatever AI tools that you're using.
00:25:42.880 --> 00:25:49.440
Uh, all right, there's your uh augmented cost, and then you compare it to your human-only cost, right?
00:25:49.519 --> 00:25:52.400
And obviously the human-only cost is gonna be much higher.
00:25:52.559 --> 00:25:53.039
Uh, right.
00:25:53.200 --> 00:25:58.799
The same thing if you are using any, you know, paid non-AI tools uh for the human-only cost, right?
00:25:58.880 --> 00:26:04.480
Maybe there's uh some, you know, I don't know if you're using the Bloomberg terminal or what right, whatever.
00:26:04.559 --> 00:26:09.839
Uh you know, a certain SaaS, and maybe you don't need to use that SaaS application if you're doing it AI native.
00:26:09.920 --> 00:26:13.440
The same thing, you need to look at the total human and software costs.
00:26:13.599 --> 00:26:14.640
And there you go.
00:26:14.799 --> 00:26:16.720
That's your return on investment.
00:26:16.880 --> 00:26:23.839
Uh, and then step seven, you need to retest this monthly after every model update.
00:26:23.920 --> 00:26:27.039
All right, and then track a three-month rolling average.
00:26:27.279 --> 00:26:29.119
Costs are going to go up and down, right?
00:26:29.279 --> 00:26:31.519
You think that, oh, well, they're just always gonna go down.
00:26:31.680 --> 00:26:37.359
Well, no, sometimes uh, you know, the Frontier AI labs will actually roll out an update that's under the radar.
00:26:37.519 --> 00:26:41.200
It's not like going from a GPT-5.2 to a 5.3, right?
00:26:41.279 --> 00:26:47.680
A lot of times there might be five, 10 different versions of a GPT-5.2 until there is a GPT-5.3.
00:26:48.079 --> 00:26:57.039
And, you know, whether it's OpenAI, Anthropic, Google, Microsoft, etc., sometimes, you know, one of those under-the-hood updates actually might make things worse.
00:26:57.279 --> 00:26:57.680
All right.
00:26:57.759 --> 00:27:07.279
Uh, so that's why you need to retest monthly or after any major model update, and then keep a three-month rolling average.
00:27:07.599 --> 00:27:08.480
There you go.
00:27:08.640 --> 00:27:11.440
All right, I'm gonna go through it quickly now with no commentary.
00:27:11.599 --> 00:27:14.240
Step one, define the rubric and success criteria.
00:27:14.400 --> 00:27:16.240
Step two, measure the human baseline.
00:27:16.319 --> 00:27:20.880
Step three, build the 20 to 40 uh real messy work cases.
00:27:21.039 --> 00:27:23.839
Step four, configure the exact production workspace.
00:27:24.000 --> 00:27:30.160
Step five, run uh with three times uh AI models in three sets of humans.
00:27:30.240 --> 00:27:35.839
Uh step six is grade blindly and standardize the output format and criteria.
00:27:36.000 --> 00:27:37.680
Uh, calculate your ROI.
00:27:37.759 --> 00:27:41.359
And then step tab step seven is retest monthly.
00:27:43.200 --> 00:27:45.359
So there you go.
00:27:46.240 --> 00:27:50.319
That's how you measure return on investment in AI.
00:27:50.799 --> 00:27:57.839
But let me just tell you this, and I'm gonna be uh very direct when I say this.
00:28:00.160 --> 00:28:01.759
You don't need to do this.
00:28:02.880 --> 00:28:05.759
You absolutely should, right?
00:28:06.319 --> 00:28:07.519
You absolutely should.
00:28:07.599 --> 00:28:08.880
You don't need to.
00:28:09.519 --> 00:28:12.079
This is this is gravity, right?
00:28:12.559 --> 00:28:15.440
AI is gravity at this point.
00:28:15.680 --> 00:28:21.839
Um, it is the all-encompassing force, and there's no denying it, right?
00:28:25.119 --> 00:28:43.039
Real studies with quantitative data that asked thousands of business leaders all overwhelmingly show that the ROI in AI is real.
00:28:43.200 --> 00:28:46.319
Three dollars and seventy cents for every dollar invested.
00:28:46.799 --> 00:28:54.400
The GDP valve, right AI models without training, right?
00:28:54.720 --> 00:29:10.720
Without iteration, in a single, in a simple shot, blind test, the AI model is the same or better than the expert seventy percent of the time and a hundred times faster.
00:29:11.680 --> 00:29:14.880
So I'm gonna leave you with this.
00:29:17.680 --> 00:29:24.480
Yes, you need to configure and figure your ROI on AI.
00:29:25.200 --> 00:29:30.799
But the new ROI question is not did AI work?
00:29:31.920 --> 00:29:38.799
It's how much did we lose by not educating and measuring sooner?
00:29:39.839 --> 00:29:41.119
That's the reality.
00:29:41.680 --> 00:29:46.240
And that is the challenge to you today, dear listener.
00:29:47.599 --> 00:29:58.160
Stop looking at ROI on AI like it's something tricky, like it's something we can't all achieve.
00:29:58.240 --> 00:30:06.400
It's as simple as being meticulous in your measurement, and that's it.
00:30:06.559 --> 00:30:20.160
And when you are meticulous in your measurement, you will undoubtedly see in insanely high return on investment of your AI.
00:30:20.400 --> 00:30:25.759
So no longer a question mark, no longer does AI work.
00:30:26.160 --> 00:30:36.559
The question is, and the way you need to change, is how much are we losing by not measuring sooner and educating our teams better?
00:30:36.720 --> 00:30:38.160
All right, that's a wrap, y'all.
00:30:38.319 --> 00:30:41.680
That is volume 11 of the Start Here series.
00:30:41.839 --> 00:30:43.039
I hope this was helpful.
00:30:43.200 --> 00:30:48.400
And if it was helpful, well, number one, please uh like and follow the podcast.
00:30:48.480 --> 00:30:49.200
I'd appreciate that.
00:30:49.359 --> 00:30:52.000
But then when you're done, go to starthirseries.com.
00:30:52.160 --> 00:30:56.079
That's going to give you free access to our inner circle community.
00:30:56.240 --> 00:31:06.640
And then you can go listen to the entire Start Here series all right there and connect with others who are trying to grow their company and their careers with generative AI.
00:31:06.720 --> 00:31:12.559
And hey, while I have you, make sure if you haven't already, go listen to the 2026 AI prediction and roadmap series.
00:31:12.640 --> 00:31:15.839
That's episodes 712 and 713.
00:31:16.000 --> 00:31:17.200
So thank you for tuning in.
00:31:17.359 --> 00:31:20.480
Hope to see you back tomorrow and every day for more everyday AI.
00:31:20.559 --> 00:31:21.519
Thanks, y'all.
00:00:02.240 --> 00:00:06.320
Can you actually prove AI is working at your company?
00:00:07.040 --> 00:00:15.439
Sure, everything feels faster, and hopefully outputs have improved, and maybe more work is getting done.
00:00:16.239 --> 00:00:17.920
But is there ROI?
00:00:18.079 --> 00:00:22.000
And don't just give me vibes, give me receipts.
00:00:22.160 --> 00:00:26.879
What's the actual return on your investment for AI at your company?
00:00:27.359 --> 00:00:29.359
Chances are you can't answer that.
00:00:29.519 --> 00:00:32.240
And the reasons are actually very simple.
00:00:32.399 --> 00:00:39.759
It's because businesses are using the same digital transformation playbook they've always used when it comes to AI.
00:00:39.920 --> 00:00:44.799
And well, that playbook is useless in 2026.
00:00:45.039 --> 00:00:58.799
So there's a 99% chance you're not actually measuring ROI, but I'm going to simplify it for you on today's show and tell you why the ROI on AI debate is absolute nonsense.
00:00:58.880 --> 00:01:04.560
And I'm going to give you seven simple steps on how your company can fix it today.
00:01:04.879 --> 00:01:07.040
All right, let's get measuring.
00:01:07.359 --> 00:01:10.640
So uh here's what you'll learn on today's show.
00:01:10.719 --> 00:01:17.680
You're going to learn the basics of measuring Gen AI ROI and why most companies never actually do it.
00:01:17.840 --> 00:01:26.959
You're going to learn why a piece of MIT marketing, disguised as a study, misled markets in what five larger studies actually show.
00:01:27.280 --> 00:01:38.640
You're going to learn how to use a four-metric scorecard covering quality, cost, reliability, and risk, and how to run a seven-step evaluation sprint that actually solves your ROI.
00:01:39.760 --> 00:01:42.719
So, what the heck is this?
00:01:42.879 --> 00:01:47.599
Well, this is everyday AI number one, but this is our start here series.
00:01:47.840 --> 00:01:53.840
I didn't have a good answer when people first subscribe to the podcast and they're like, Jordan, there's 700 episodes.
00:01:53.920 --> 00:01:54.719
Where do I start?
00:01:55.200 --> 00:01:56.319
I'm like, I don't know.
00:01:56.480 --> 00:02:00.079
Well, now I do know you start here with the start here series.
00:02:00.159 --> 00:02:05.920
This is the essential podcast series to both learn the AI basics and to double down on your AI knowledge.
00:02:06.079 --> 00:02:09.439
So make sure to go to starthearseries.com.
00:02:09.520 --> 00:02:13.199
Uh, that's going to give you free access to our inner circle community.
00:02:13.360 --> 00:02:21.599
And then you can go in the Start Here Series space and listen to now all 11 episodes of this series.
00:02:21.759 --> 00:02:33.360
Uh, so last episode, which was a banger FYI, um, we talked about from AI chatbots to autonomous workers and how consumer AI has changed and what's next.
00:02:33.520 --> 00:02:35.199
So make sure you go uh check that out.
00:02:35.280 --> 00:02:40.159
It's episode 723, but uh volume 10 of the Start Here series.
00:02:40.319 --> 00:02:47.280
And today we're gonna be measuring ROI and tell you why your company is doing it wrong and these seven steps to fix it.
00:02:48.319 --> 00:02:49.520
Let's start here.
00:02:50.080 --> 00:02:53.599
Talked about this once or twice, but let's talk about this benchmark from OpenAI.
00:02:53.680 --> 00:02:55.199
It's called GDP Val.
00:02:55.919 --> 00:03:04.800
So this benchmark tests AI on real-world deliverables across 44 jobs in nine different GDP sectors.
00:03:05.039 --> 00:03:15.120
Essentially, this is a unbiased benchmark that pits AI models against expert humans, experts in their fields, right?
00:03:15.280 --> 00:03:20.639
They each have the same task, which is generally uh to produce something of value, right?
00:03:20.719 --> 00:03:24.080
That's why it's called GDP Val, to produce something of economic value.
00:03:24.240 --> 00:03:30.800
And then there's expert judges, expert human judges, and they don't know which output came from the AI.
00:03:30.879 --> 00:03:33.439
They don't know which output came from the expert human.
00:03:33.599 --> 00:03:44.159
And right now, top AI models either tie or win 70% of the time in these blind evaluations against professionals that have an average of 14 years of experience.
00:03:44.400 --> 00:03:47.039
I mean, my gosh, we should just end the episode now, right?
00:03:47.199 --> 00:03:50.960
There's your ROI on AI, right?
00:03:51.439 --> 00:03:52.719
Done in under four minutes.
00:03:52.800 --> 00:03:53.840
There's a lot of issues here.
00:03:53.919 --> 00:03:57.120
Number one, even what I just said, right?
00:03:57.360 --> 00:04:11.520
Do you know how to use today's models to, you know, one shot personalize, synthesize information, create, you know, spreadsheets, uh, PowerPoint decks, and use your own company's information without you having to do anything and verify outputs?
00:04:11.599 --> 00:04:18.240
Well, probably not, because companies aren't educating their people, and it's hard to keep up with uh today's capabilities of AI models.
00:04:18.480 --> 00:04:26.160
But what this did show us is that the whole debate on ROI is absolute rubbish.
00:04:26.319 --> 00:04:27.439
Yes, there we go.
00:04:27.600 --> 00:04:31.680
Uh, for uh listeners across the pond, look at me.
00:04:31.759 --> 00:04:33.920
I'm uh multilingual here.
00:04:34.000 --> 00:04:35.360
Uh it's garbage.
00:04:35.439 --> 00:04:46.160
This whole discussion on does artificial intelligence give you a return on your investment, it's it's honestly a very dumb question, if I'm being honest, right?
00:04:46.319 --> 00:04:48.959
Uh go go read the GDP BAO study.
00:04:49.120 --> 00:04:51.360
Um, I keep saying, oh, I'm gonna do a show soon.
00:04:51.439 --> 00:04:54.399
I I will actually do a show soon uh because it's worth it.
00:04:54.560 --> 00:05:04.240
So not only did it show that, well, today's AI models are exponentially better than expert humans when judged by other expert humans, right?
00:05:04.399 --> 00:05:08.399
But it said that in many tasks they do it a hundred times faster.
00:05:08.639 --> 00:05:20.480
So again, I am not a mathematician, but if the AI model is the same or better 70% of the time and it's a hundred times faster, that's the math.
00:05:20.639 --> 00:05:27.279
So why are we still even debating this concept of is there return on investment?
00:05:27.439 --> 00:05:32.000
Well, that's because of an infamous piece of marketing from MIT.
00:05:32.720 --> 00:05:39.040
So uh, and yes, I'm calling it a piece of marketing or study in very heavy quotes, right?
00:05:39.120 --> 00:05:44.639
Like the uh uh what is that, the the Austin Powers uh, you know, a billion dollar.
00:05:44.959 --> 00:05:47.199
Yeah, huge quotes on the word study.
00:05:47.360 --> 00:05:47.839
Uh, right.
00:05:47.920 --> 00:05:57.279
So MIT claimed uh in their wildly viral study that 95% of enterprise AI pilots delivered zero ROI.
00:05:57.360 --> 00:06:01.279
And this was in August of 2025, right?
00:06:01.519 --> 00:06:03.600
And I mean the markets moved, right?
00:06:03.759 --> 00:06:09.040
Nvidia stock tanked three and a half points, Palantir went down uh nearly nine percent.
00:06:09.199 --> 00:06:23.360
There's actually billions of dollars of losses across the uh not just the tech sector, just the business sector, because right, AI has essentially propped up an otherwise struggling uh US and at times global economy.
00:06:23.600 --> 00:06:28.480
Um, and then you know, this study came out and said, Oh, AI's, you know, there's no ROI.
00:06:28.639 --> 00:06:31.199
Well, uh most people, right?
00:06:31.439 --> 00:06:34.000
As a former journalist, I understand this, right?
00:06:34.079 --> 00:06:41.040
There is one study that came out, and then essentially every other uh journalist hit the copy and paste, uh, you know, more or less.
00:06:41.199 --> 00:06:47.519
Hey, that headline sells, uh, you know, 95% of AI pilots don't uh produce any ROI, right?
00:06:47.600 --> 00:06:48.000
That's great.
00:06:48.079 --> 00:06:48.959
Let's go ahead and run that.
00:06:49.040 --> 00:06:53.199
But I don't think any journalists actually read the story, or very few actually did, uh, right.
00:06:53.279 --> 00:06:58.399
That's because this quote unquote study was based on 52 qualitative interviews.
00:06:58.560 --> 00:07:01.360
There's no quantitative piece to that 95%.
00:07:01.759 --> 00:07:04.160
And it was uh directionally, right?
00:07:04.240 --> 00:07:05.759
Uh, this was a directional study.
00:07:05.920 --> 00:07:07.519
So it's more or less this was a vibe.
00:07:07.600 --> 00:07:08.560
This was a vibe study.
00:07:08.720 --> 00:07:12.480
I'm gonna talk to 52 people, you know, nothing quantitative.
00:07:12.560 --> 00:07:14.000
I'm just gonna give my vibe on this.
00:07:14.079 --> 00:07:21.759
But the stat came from 52 qualitative interviews that measured PL impact in under six months, which there's no such thing, right?
00:07:21.920 --> 00:07:26.800
When you talk about digital transformation, go back and look at the internet, right?
00:07:27.199 --> 00:07:33.040
You could have shown the PL impact in under six months on using the internet, right?
00:07:33.439 --> 00:07:37.680
Because if you could have, you could apply that exact same methodology to AI.
00:07:37.839 --> 00:07:42.240
Anyways, well, why this is a piece of marketing and not actually just uh a bad study.
00:07:42.399 --> 00:07:44.160
I would call it a bad study, but it wasn't.
00:07:44.240 --> 00:07:46.959
It was marketing because in the end they were selling their own product.
00:07:47.040 --> 00:07:52.480
They said, Hey, uh, right, all these AI pilots fail because you're not using our agentic AI product.
00:07:52.560 --> 00:07:55.040
And they were, you know, selling access to it.
00:07:55.279 --> 00:07:58.000
Um, yeah, I'll stop there.
00:07:58.160 --> 00:08:10.720
If you really want the down low on that one, make sure to go listen to episode 597, where I, like a human, uh read it, like a human with a brain, uh, read the study multiple times, and it's uh, you know, laughable.
00:08:10.879 --> 00:08:13.839
Anyways, if you look at the real data, right?
00:08:14.000 --> 00:08:20.639
Companies that talked to thousands and got quantitative data, obviously the ROI for AI is there.
00:08:20.720 --> 00:08:21.199
Uh, right.
00:08:21.279 --> 00:08:29.040
The IDC, the International Data Corporation, found a$3.70 return for every dollar invested in AI.
00:08:29.279 --> 00:08:37.200
Wharton, a three-year study, found that 74% of enterprises report positive ROI on gen AI.
00:08:37.440 --> 00:08:48.240
Google Cloud's 2025 annual AI study found 74% of executives reported achieving ROI just within the first year of generative AI deployment.
00:08:48.559 --> 00:08:57.600
Also, Deloitte's 2025 AI survey found 84% of those investing in AI said that they were gaining ROI.
00:08:58.000 --> 00:09:07.120
Um, this I think the this this debate is still going on because humans are innately lazy.
00:09:08.399 --> 00:09:08.879
Right?
00:09:09.039 --> 00:09:16.720
That's the reason why we don't want to actually sit and measure, right?
00:09:17.039 --> 00:09:26.639
Again, my my background, I have a little bit of marketing and advertising background, so I understand the importance of measuring something before and after to see if there's an impact, right?
00:09:26.879 --> 00:09:30.240
Running millions of dollars of ads for clients over the years.
00:09:30.320 --> 00:09:32.799
There's this thing called return on ad spend, right?
00:09:32.879 --> 00:09:37.120
You see how much uh you know how much money was spent on ads.
00:09:37.360 --> 00:09:43.200
You have to be able to attribute and track the revenue that it brought in, and then you do some simple math, right?
00:09:43.360 --> 00:09:44.480
Return on ad spent.
00:09:44.559 --> 00:09:50.799
Well, there is also a simple equation that I'm gonna give you guys here in a little bit, the same thing, return on AI.
00:09:51.360 --> 00:10:09.120
So here's the dirty little secret because you're probably thinking, okay, Jordan, well, if every company's using AI, well, why isn't everyone just seeing exponential growth?
00:10:09.360 --> 00:10:16.320
Well, uh, I think you would see that a little bit more if only 5% of uh businesses were using AI, right?
00:10:16.399 --> 00:10:20.639
But most studies say more than 95 or 97% of enterprises.
00:10:20.799 --> 00:10:24.879
So uh the playing field has just gone up, right?
00:10:25.200 --> 00:10:30.000
Uh the minimum entry is no longer like, oh, you're using AI.
00:10:30.159 --> 00:10:30.639
That's great.
00:10:30.720 --> 00:10:32.000
That's such a differentiator.
00:10:32.159 --> 00:10:34.399
No, uh, no, it's you're using AI.
00:10:34.559 --> 00:10:34.799
Okay.
00:10:34.879 --> 00:10:36.960
I've been saying since 2023.
00:10:37.120 --> 00:10:39.759
It's the same thing as, oh, our company's using the internet.
00:10:40.159 --> 00:10:41.759
So, right.
00:10:42.000 --> 00:10:44.159
Uh, but here's what's happening.
00:10:45.759 --> 00:10:47.679
Workers are pocketing it.
00:10:47.840 --> 00:10:57.279
That's where the ROI is going, uh, right, because true productivity and true ROI, well, it requires results-driven metrics, not time-based management.
00:10:57.519 --> 00:11:01.039
Most companies haven't, you know, gotten this figured out yet.
00:11:01.200 --> 00:11:18.080
Uh, and I think a big portion of this is that remote and hybrid workers are using AI, whether it's approved AI or uh, you know, whether it's uh, you know, AI sprawl, uh, you know, dark AI, whatever you want to call it, and they're just pocketing it.
00:11:18.159 --> 00:11:20.480
It's this invisible productivity, right?
00:11:20.559 --> 00:11:22.960
I've literally talked to countless people.
00:11:23.120 --> 00:11:34.240
I won't say hundreds because I don't know if it's actual 200 or more, but I've easily talked to more than a hundred people over the past few years in very successful, right?
00:11:34.320 --> 00:11:40.159
Even one of my um, you know, one of my good friends uh told me this like uh maybe one or two years ago.
00:11:40.960 --> 00:11:51.919
Pretty, pretty high up uh at a company that was uh public company when he worked there, said he automated about 95% of his job with AI, right?
00:11:52.080 --> 00:11:53.519
No one knew because he's at home.
00:11:53.600 --> 00:11:55.200
I'm like, what are you doing with all this time?
00:11:55.279 --> 00:11:58.480
He's like, ah, you know, chelling golfing a lot, uh, right?
00:11:58.559 --> 00:11:59.919
That's the reality.
00:12:00.639 --> 00:12:07.600
You know, uh so many companies, so many employees, I think, are just pocketing this time savings, right?
00:12:07.679 --> 00:12:11.519
As as we've gone to this uh, you know, remote hybrid workforce.
00:12:11.600 --> 00:12:18.159
And I think that's one of the reasons why last year uh so many companies had this, you know, big return to work push.
00:12:18.320 --> 00:12:26.799
Uh and right now, according to workday, 89% of organizations haven't even updated their roles to reflect AI, right?
00:12:27.039 --> 00:12:35.759
So they're still using uh you know 2026 tools uh inside of job structures made 10 years ago.
00:12:36.000 --> 00:12:36.720
Um, right.
00:12:37.039 --> 00:12:38.559
So why does that matter?
00:12:38.799 --> 00:12:48.000
Well, again, hiring and roles in today's enterprise are still pre-AI.
00:12:48.480 --> 00:12:58.399
So what that generally means, well, expectations or output is measured in a pre-AI way, which isn't necessarily the right thing, right?
00:12:58.559 --> 00:13:08.480
And in the same way that I've encouraged um you all for years to not upskill or reskill, uh, that's that's a waste of time when it comes to AI, right?
00:13:08.639 --> 00:13:11.840
You need to unlearn and rebuild, been saying that for a long time.
00:13:12.159 --> 00:13:19.120
Um departments and companies need to do the exact same thing, right?
00:13:19.440 --> 00:13:29.039
You can't just, oh, let's add a little bit of AI to this job role, or uh, you know, hey, let's let's try to think a little more uh AI natively here at this company.
00:13:29.200 --> 00:13:34.320
No, you you gotta tear it apart in the same way that job roles haven't changed, right?
00:13:34.480 --> 00:13:36.720
That means outputs aren't going to change either.
00:13:36.960 --> 00:13:44.159
But if you're using AI, if you've trained your people on AI, if they have the right AI, they're just banking that productivity.
00:13:44.320 --> 00:13:47.759
So here's how finding ROI is done.
00:13:47.919 --> 00:13:49.200
It's simple.
00:13:49.519 --> 00:13:51.840
Uh ROI can mean a lot of different things.
00:13:51.919 --> 00:13:55.440
It can a lot of people just think, oh, that means you're bringing in more revenue.
00:13:55.600 --> 00:13:58.960
No, I think mainly it's time saved, right?
00:13:59.039 --> 00:14:04.480
But I think ROI also means just cost reduced, revenue increased, or risk avoided.
00:14:04.559 --> 00:14:06.159
It's not just prompts sent, right?
00:14:06.240 --> 00:14:08.240
I think that's what a lot of people they're looking at.
00:14:08.320 --> 00:14:15.279
Uh the number of hours, uh, you know, your utilization rate if you're using, you know, an enterprise tool like ChatGPT, right?
00:14:15.360 --> 00:14:20.559
Everyone's like, oh, you know, we need to increase our utilization from you know eight percent to 20%, right?
00:14:20.799 --> 00:14:26.159
And then companies, you know, hire us to, you know, train hundreds or thousands of employees.
00:14:26.240 --> 00:14:30.960
And it's like, okay, utilization is not uh the metric that pushes it.
00:14:31.039 --> 00:14:32.960
It's are you saving time?
00:14:33.200 --> 00:14:36.639
Are you um increasing revenue and avoiding risk?
00:14:36.799 --> 00:14:39.120
Uh the formula, well, simple math.
00:14:39.200 --> 00:14:42.960
It's the time saved or uh increased revenue.
00:14:43.039 --> 00:14:44.320
Uh, but let's look at time saved.
00:14:44.399 --> 00:14:51.919
So it's just time saved multiplied by fully loaded hourly rates minus any any monthly AI subscription costs.
00:14:52.639 --> 00:14:54.240
Simple formula, right?
00:14:54.559 --> 00:15:04.080
Um, and right now boards and CFOs just care about cost per task, uh cost per task throughput and error rates, right?
00:15:04.320 --> 00:15:07.679
Not usage dashboards, not utilization.
00:15:09.279 --> 00:15:16.000
But we're seeing the same reasons for failure, right?
00:15:16.240 --> 00:15:18.320
Still years later, right?
00:15:18.559 --> 00:15:34.960
Um, through my own conversations and conversations with others, I can say when it comes to measuring or not measuring ROI, I still think that there's five patterns or five main reasons why either individuals or departments aren't able to do this.
00:15:35.120 --> 00:15:37.840
So, number one, not having a pre-AI baseline.
00:15:38.000 --> 00:15:41.759
Number two, you know, doing these slow year-long pilots.
00:15:41.840 --> 00:15:43.279
That's a recipe for disaster.
00:15:43.440 --> 00:15:49.039
Number three, celebrating one lucky run and making that your AI strategy.
00:15:49.200 --> 00:15:52.000
Number four, when vanity metrics become the focus.
00:15:52.159 --> 00:16:01.120
And number five, shiny object syndrome, pulling focus uh away before real company-wide implementation can actually start, right?
00:16:01.279 --> 00:16:05.519
Oh, you get a little, you get a little ground, you show ROI, and then you're like, oh, what about this model?
00:16:05.600 --> 00:16:05.919
What about this?
00:16:06.000 --> 00:16:06.480
What about this?
00:16:06.559 --> 00:16:07.279
What about this, right?
00:16:07.360 --> 00:16:10.399
Before you actually, you know, uh implement it company wide.
00:16:10.559 --> 00:16:12.960
And these aren't technology problems, right?
00:16:13.279 --> 00:16:17.440
This isn't standard digital transformation.
00:16:19.759 --> 00:16:28.879
These are we as working human beings, as knowledge workers, uh, we don't have a playbook to follow, right?
00:16:28.960 --> 00:16:33.759
So all we've been doing is following the same playbook that we've always done, right?
00:16:34.240 --> 00:16:36.080
Those slow pilots.
00:16:36.320 --> 00:16:39.440
You know, you get one thing right and you're like, oh, this is what we do now.
00:16:39.519 --> 00:16:41.279
It's the wrong way to do this.
00:16:41.919 --> 00:16:44.639
The other big reason, well, training.
00:16:45.279 --> 00:16:45.759
Right.
00:16:45.919 --> 00:16:49.519
Uh the training actually leads to those five other failures.
00:16:49.600 --> 00:16:53.679
So it's more of a of an underlying uh foundational issue, right?
00:16:53.759 --> 00:17:02.159
But right now, 49% of enterprise leaders cite recurring, uh, sorry, cite recruiting AI talent as their single biggest challenge.
00:17:02.879 --> 00:17:03.440
All right.
00:17:03.840 --> 00:17:07.920
AI, you need documented workflow steps and company data.
00:17:08.079 --> 00:17:14.319
And that process documentation is part of the ROI investment, right?
00:17:14.720 --> 00:17:20.720
Education, training, ongoing learning, it is required.
00:17:21.200 --> 00:17:21.759
Uh, right.
00:17:21.920 --> 00:17:42.480
So many, I think when um when AI, uh when an AI strategy or actual AI implementation looks to looks like you know, pushing top down, you know, either C-suite or board pushing, you know, a certain tool top down and not training or educating, that's when it's so easy to fail, uh, right.
00:17:43.920 --> 00:17:47.200
I use AI all day, every day.
00:17:47.440 --> 00:17:49.359
And I'll I'll I'll tell you this.
00:17:49.599 --> 00:17:53.279
If I take two weeks off, I'm gonna fail.
00:17:53.920 --> 00:17:56.400
That's the that's the pace, right?
00:17:56.640 --> 00:18:01.839
People are always like, okay, well, Jordan, you clearly don't know what you're saying here.
00:18:02.000 --> 00:18:06.079
If AI is a hundred times faster, well, shouldn't we just cut all these jobs?
00:18:06.160 --> 00:18:08.079
No, I think jobs are gonna look very different.
00:18:08.240 --> 00:18:10.640
I've been on record as saying this since the very first episode.
00:18:10.799 --> 00:18:17.759
AI is going to uh, I think, quote unquote, take away more jobs than it will create, but it will create many roles that we don't even understand.
00:18:17.839 --> 00:18:30.000
I've always said most enterprise organizations, they need a lot of people like me, that all they do all day is they keep up with AI and they apply it to that business, right?
00:18:30.240 --> 00:18:33.440
Very few businesses have that because they're like, okay, that's a crazy role.
00:18:33.599 --> 00:18:45.440
We're not gonna let you know, Bill over in IT or you know, Deborah over in marketing just spend all day uh, you know, tinkering with AI and seeing how to apply it.
00:18:45.519 --> 00:18:46.160
No, right?
00:18:46.240 --> 00:18:48.799
No, but yes, that's exactly what you should be doing.
00:18:48.960 --> 00:18:58.079
Uh, and here's all right, uh, this is usually something I think we only uh give away to companies that uh pay us a couple dollars.
00:18:58.240 --> 00:19:00.400
But let me just go ahead and give you some of the secrets here.
00:19:00.480 --> 00:19:00.799
All right.
00:19:00.880 --> 00:19:02.640
Here's an acronym, BASE.
00:19:02.960 --> 00:19:04.240
You need a base.
00:19:04.559 --> 00:19:05.839
This is the baseline rule.
00:19:06.000 --> 00:19:12.480
So this is BASE stands for baseline assessment of standard execution pre-AI.
00:19:12.720 --> 00:19:17.920
All right, so before AI touches any workflow, you got to get that base, right?
00:19:18.000 --> 00:19:24.880
That's your time multiple employees completing that exact task without it.
00:19:25.440 --> 00:19:26.480
You need to time it.
00:19:26.559 --> 00:19:33.039
You need to record the average time, the error rate, the rework cycles, and cost per completed task as the baseline.
00:19:33.599 --> 00:19:37.359
You can't go back and collect this retroactively, right?
00:19:37.440 --> 00:19:45.759
Because usually what happens, well, if you've already, you know, sprinkled AI in the process and you've been doing it for uh a year, right?
00:19:45.920 --> 00:19:55.279
Let's say it's a 10-step process and hey, number two and uh step two and step six, we you know uh we molded those a little bit around AI, right?
00:19:55.359 --> 00:19:57.279
You it's it's it's too late, right?
00:19:57.599 --> 00:20:02.160
You need to do it before you implement AI.
00:20:02.319 --> 00:20:09.839
You need to measure how long it takes humans to go through and do these certain projects or do these certain tasks, right?
00:20:10.000 --> 00:20:14.480
Think of it like an internal GDP valve, right?
00:20:14.640 --> 00:20:31.359
You are gonna go through and have a human with no AI go through and do this task, document every single step of the process painstakingly, because you need to see okay, uh what other humans need to be involved in this?
00:20:31.680 --> 00:20:33.039
Are there other meetings?
00:20:33.279 --> 00:20:35.279
What about the person checking the work?
00:20:35.440 --> 00:20:37.680
What about the communication going back and forth?
00:20:37.759 --> 00:20:41.519
You need to measure it all and document it all every single step.
00:20:42.480 --> 00:20:43.920
And then what's the completion rate?
00:20:44.079 --> 00:20:45.039
What's the error rate?
00:20:45.119 --> 00:20:46.400
What does quality look like?
00:20:46.799 --> 00:20:53.039
You need to establish those baselines and then redo the entire process, right?
00:20:53.200 --> 00:20:55.200
Blow it up, measure it first.
00:20:55.920 --> 00:20:57.279
That's your base, right?
00:20:57.759 --> 00:20:59.759
Baseline assessment of standard execution.
00:21:00.720 --> 00:21:06.160
That's your base, blow it up, do the same thing with AI, right?
00:21:07.039 --> 00:21:11.039
Not your first iteration, iterate on it, and then you measure.
00:21:11.920 --> 00:21:21.680
And that number, right there, you need that because that is the base, quite literally, for how you're going to ultimately measure your ROI.
00:21:22.079 --> 00:21:25.359
So now let's give you that seven-step guide.
00:21:25.599 --> 00:21:26.880
All right, I'm delivering.
00:21:27.039 --> 00:21:27.839
Here we go.
00:21:30.880 --> 00:21:41.440
Well, let me just uh bring my notes up here because you know, in typical live stream, unedited, unscripted fashion, uh, I didn't put my slide up here with my uh seven steps.
00:21:41.519 --> 00:21:43.359
But don't worry, I got my notes.
00:21:45.200 --> 00:21:46.160
All right, there we go.
00:21:46.319 --> 00:21:47.359
Notes on the screen.
00:21:47.440 --> 00:21:51.359
So here's the seven step blueprint, right?
00:21:51.519 --> 00:21:53.359
Step one, we already talked about it.
00:21:53.519 --> 00:21:57.279
That's the uh well, actually comes a little bit before the base.
00:21:57.519 --> 00:21:59.759
So step one, you have to define, you have to define.
00:22:00.319 --> 00:22:02.400
What the heck is it you're doing, uh, right?
00:22:02.559 --> 00:22:04.000
And you have to be very rigid.
00:22:04.079 --> 00:22:08.400
You can't be flexible because you need to get um an accurate before and after.
00:22:08.480 --> 00:22:16.319
So that's defining the rubric uh rubric, uh, defining the success criteria and KPIs before you even begin testing.
00:22:16.400 --> 00:22:19.119
And then step two, like I talked about, that's the base.
00:22:19.279 --> 00:22:21.599
You need to measure the human baseline.
00:22:21.680 --> 00:22:26.799
And then you're gonna uh, you know, time multiple employees doing that and then record the averages.
00:22:26.960 --> 00:22:29.680
Uh step three, get messy, right?
00:22:29.839 --> 00:22:37.599
Uh you have to build 20 to 40 real messy work examples, including drift cases, right?
00:22:37.680 --> 00:22:41.359
These aren't easy things for humans or for AI.
00:22:41.599 --> 00:22:42.000
All right.
00:22:42.079 --> 00:22:47.279
And then step four, you need to configure the exact production workspace.
00:22:47.359 --> 00:22:50.960
It's the same plan, the same model, and the same uh permissions.
00:22:51.119 --> 00:22:55.920
So you're not, you know, flip-flopping uh, you know, between different people, different accounts.
00:22:56.160 --> 00:22:56.960
No, right?
00:22:57.039 --> 00:22:59.920
It has to be controlled like any experiments.
00:23:00.000 --> 00:23:06.799
Uh, and it has to be repeatable and scalable, the exact same criteria when you're taking this to a production run.
00:23:06.960 --> 00:23:11.440
Uh, step five, you're gonna run every three times.
00:23:11.759 --> 00:23:12.079
Okay.
00:23:12.640 --> 00:23:20.559
So, sorry, run each test three times with memory off and also require proof artifacts.
00:23:20.720 --> 00:23:21.920
So, here's what I mean by that.
00:23:22.079 --> 00:23:23.359
We're doing this on the front end.
00:23:23.519 --> 00:23:23.839
Okay.
00:23:24.160 --> 00:23:34.799
Um, if you listen, if you've listened to this podcast at all, you know I'm a big believer in bringing as many of your processes over uh to front-end large language models.
00:23:34.880 --> 00:23:40.720
We actually did a dedicated episode uh on that in the start here series, talking about an AI operating system.
00:23:40.880 --> 00:23:48.960
So, yeah, this is all going to be doing things on as an example, chat gpt.com, claw.ai, gemini.google.com, right?
00:23:49.039 --> 00:23:59.839
But doing it with memory turned off and doing it usually in a temporary chat if your uh you know AI operating system of choice uh lends itself to that.
00:24:00.240 --> 00:24:00.799
All right.
00:24:01.119 --> 00:24:10.880
So essentially there, you have steps three through six is that's your internal GDP valve, right?
00:24:11.200 --> 00:24:15.119
Uh you have to get the the correct use uh use cases.
00:24:15.359 --> 00:24:19.440
Um, and then from there uh you grade blind, right?
00:24:19.599 --> 00:24:20.720
So same thing.
00:24:21.039 --> 00:24:30.400
You have the the series of people do it uh three times, those 20 to 40 use cases using the same AI model.
00:24:30.640 --> 00:24:36.559
Uh, and then you have humans do those same exact things as well, multiple humans in the same way.
00:24:36.640 --> 00:24:44.079
Uh I would suggest, you know, three different times um running the same case and then three different sets of humans.
00:24:44.640 --> 00:24:53.279
And then you grade blind, you standardize the output format, you have to agree on, you know, what's a pass, what's a fail, what's the grading scale, et cetera.
00:24:53.440 --> 00:24:58.079
But then at that point, well, you have the input and the output, right?
00:24:58.240 --> 00:25:12.960
At that point, after step six, when you can have your grading criteria, uh, you know, you run through and you do the test three times, um, you know, those 20 to 40 use cases, three times uh in a large language model, memory off, uh, temporary chat.
00:25:13.119 --> 00:25:17.119
You have your your grading rubric, you have your humans do it, uh, right?
00:25:17.279 --> 00:25:20.480
The same 20 to 40 tasks, three different humans.
00:25:21.599 --> 00:25:22.720
You have it right there.
00:25:22.880 --> 00:25:24.319
You have the time, right?
00:25:24.480 --> 00:25:30.799
You multiply uh the time that it takes the humans to do it on the AI side, all right.
00:25:31.039 --> 00:25:42.799
Minus out, uh, you know, so take, let's say it's a hundred human hours, take their hourly rate, times it, minus the uh cost of whatever AI tools that you're using.
00:25:42.880 --> 00:25:49.440
Uh, all right, there's your uh augmented cost, and then you compare it to your human-only cost, right?
00:25:49.519 --> 00:25:52.400
And obviously the human-only cost is gonna be much higher.
00:25:52.559 --> 00:25:53.039
Uh, right.
00:25:53.200 --> 00:25:58.799
The same thing if you are using any, you know, paid non-AI tools uh for the human-only cost, right?
00:25:58.880 --> 00:26:04.480
Maybe there's uh some, you know, I don't know if you're using the Bloomberg terminal or what right, whatever.
00:26:04.559 --> 00:26:09.839
Uh you know, a certain SaaS, and maybe you don't need to use that SaaS application if you're doing it AI native.
00:26:09.920 --> 00:26:13.440
The same thing, you need to look at the total human and software costs.
00:26:13.599 --> 00:26:14.640
And there you go.
00:26:14.799 --> 00:26:16.720
That's your return on investment.
00:26:16.880 --> 00:26:23.839
Uh, and then step seven, you need to retest this monthly after every model update.
00:26:23.920 --> 00:26:27.039
All right, and then track a three-month rolling average.
00:26:27.279 --> 00:26:29.119
Costs are going to go up and down, right?
00:26:29.279 --> 00:26:31.519
You think that, oh, well, they're just always gonna go down.
00:26:31.680 --> 00:26:37.359
Well, no, sometimes uh, you know, the Frontier AI labs will actually roll out an update that's under the radar.
00:26:37.519 --> 00:26:41.200
It's not like going from a GPT-5.2 to a 5.3, right?
00:26:41.279 --> 00:26:47.680
A lot of times there might be five, 10 different versions of a GPT-5.2 until there is a GPT-5.3.
00:26:48.079 --> 00:26:57.039
And, you know, whether it's OpenAI, Anthropic, Google, Microsoft, etc., sometimes, you know, one of those under-the-hood updates actually might make things worse.
00:26:57.279 --> 00:26:57.680
All right.
00:26:57.759 --> 00:27:07.279
Uh, so that's why you need to retest monthly or after any major model update, and then keep a three-month rolling average.
00:27:07.599 --> 00:27:08.480
There you go.
00:27:08.640 --> 00:27:11.440
All right, I'm gonna go through it quickly now with no commentary.
00:27:11.599 --> 00:27:14.240
Step one, define the rubric and success criteria.
00:27:14.400 --> 00:27:16.240
Step two, measure the human baseline.
00:27:16.319 --> 00:27:20.880
Step three, build the 20 to 40 uh real messy work cases.
00:27:21.039 --> 00:27:23.839
Step four, configure the exact production workspace.
00:27:24.000 --> 00:27:30.160
Step five, run uh with three times uh AI models in three sets of humans.
00:27:30.240 --> 00:27:35.839
Uh step six is grade blindly and standardize the output format and criteria.
00:27:36.000 --> 00:27:37.680
Uh, calculate your ROI.
00:27:37.759 --> 00:27:41.359
And then step tab step seven is retest monthly.
00:27:43.200 --> 00:27:45.359
So there you go.
00:27:46.240 --> 00:27:50.319
That's how you measure return on investment in AI.
00:27:50.799 --> 00:27:57.839
But let me just tell you this, and I'm gonna be uh very direct when I say this.
00:28:00.160 --> 00:28:01.759
You don't need to do this.
00:28:02.880 --> 00:28:05.759
You absolutely should, right?
00:28:06.319 --> 00:28:07.519
You absolutely should.
00:28:07.599 --> 00:28:08.880
You don't need to.
00:28:09.519 --> 00:28:12.079
This is this is gravity, right?
00:28:12.559 --> 00:28:15.440
AI is gravity at this point.
00:28:15.680 --> 00:28:21.839
Um, it is the all-encompassing force, and there's no denying it, right?
00:28:25.119 --> 00:28:43.039
Real studies with quantitative data that asked thousands of business leaders all overwhelmingly show that the ROI in AI is real.
00:28:43.200 --> 00:28:46.319
Three dollars and seventy cents for every dollar invested.
00:28:46.799 --> 00:28:54.400
The GDP valve, right AI models without training, right?
00:28:54.720 --> 00:29:10.720
Without iteration, in a single, in a simple shot, blind test, the AI model is the same or better than the expert seventy percent of the time and a hundred times faster.
00:29:11.680 --> 00:29:14.880
So I'm gonna leave you with this.
00:29:17.680 --> 00:29:24.480
Yes, you need to configure and figure your ROI on AI.
00:29:25.200 --> 00:29:30.799
But the new ROI question is not did AI work?
00:29:31.920 --> 00:29:38.799
It's how much did we lose by not educating and measuring sooner?
00:29:39.839 --> 00:29:41.119
That's the reality.
00:29:41.680 --> 00:29:46.240
And that is the challenge to you today, dear listener.
00:29:47.599 --> 00:29:58.160
Stop looking at ROI on AI like it's something tricky, like it's something we can't all achieve.
00:29:58.240 --> 00:30:06.400
It's as simple as being meticulous in your measurement, and that's it.
00:30:06.559 --> 00:30:20.160
And when you are meticulous in your measurement, you will undoubtedly see in insanely high return on investment of your AI.
00:30:20.400 --> 00:30:25.759
So no longer a question mark, no longer does AI work.
00:30:26.160 --> 00:30:36.559
The question is, and the way you need to change, is how much are we losing by not measuring sooner and educating our teams better?
00:30:36.720 --> 00:30:38.160
All right, that's a wrap, y'all.
00:30:38.319 --> 00:30:41.680
That is volume 11 of the Start Here series.
00:30:41.839 --> 00:30:43.039
I hope this was helpful.
00:30:43.200 --> 00:30:48.400
And if it was helpful, well, number one, please uh like and follow the podcast.
00:30:48.480 --> 00:30:49.200
I'd appreciate that.
00:30:49.359 --> 00:30:52.000
But then when you're done, go to starthirseries.com.
00:30:52.160 --> 00:30:56.079
That's going to give you free access to our inner circle community.
00:30:56.240 --> 00:31:06.640
And then you can go listen to the entire Start Here series all right there and connect with others who are trying to grow their company and their careers with generative AI.
00:31:06.720 --> 00:31:12.559
And hey, while I have you, make sure if you haven't already, go listen to the 2026 AI prediction and roadmap series.
00:31:12.640 --> 00:31:15.839
That's episodes 712 and 713.
00:31:16.000 --> 00:31:17.200
So thank you for tuning in.
00:31:17.359 --> 00:31:20.480
Hope to see you back tomorrow and every day for more everyday AI.
00:31:20.559 --> 00:31:21.519
Thanks, y'all.
