WEBVTT
00:00:01.760 --> 00:00:08.000
There's been this pseudo badge of AI honor over the past six months that makes no sense.
00:00:08.160 --> 00:00:14.320
And I think it has led a lot of enterprise companies astray when choosing the right AI strategy.
00:00:14.720 --> 00:00:16.160
What is that thing?
00:00:16.399 --> 00:00:18.480
It's token maxing.
00:00:18.719 --> 00:00:34.719
So token maxing is the corporate practice where companies push AI token consumption and correlate that AI usage as a direct metric for employee productivity, or pointing to the fact that they're doing something positive with their AI spend.
00:00:34.880 --> 00:00:47.840
Like if you're using billions of AI tokens a day, that means that you're living inside of large language models and clearly you are creating huge value for the company because more prompts has to mean more ROI, right?
00:00:48.560 --> 00:01:00.000
But that strange AI power flex of early 2026 has quickly revealed itself to be a false and sometimes dangerous metric by mid-2026.
00:01:00.079 --> 00:01:12.000
Because it turns out all that token maxing leads to is a competition to see who can run the longest agentic loops, regardless of if it actually accomplishes a company's goal.
00:01:12.159 --> 00:01:13.599
So what's the new metric?
00:01:13.760 --> 00:01:15.680
What should your company be looking at?
00:01:15.920 --> 00:01:17.359
That's token efficiency.
00:01:17.519 --> 00:01:31.519
In other words, getting the most bang for your AI token usage buck means not just burning AI usage for the sake of using it, but instead aligning your company's AI strategy with the right model, used the right way for the right job.
00:01:31.760 --> 00:01:32.239
Weird.
00:01:32.480 --> 00:01:37.680
Common AI Sense pays off more than a leaderboard showing who is using the most tokens.
00:01:37.920 --> 00:01:56.079
So on today's show, we're going to break down the rise and fall of token maxing, not only showing you what tokens are and how they work, but also uncovering the proper best practices to put your tokens toward actual measurable outputs versus just lighting millions of dollars on fire.
00:01:56.400 --> 00:01:59.599
So here's the big picture.
00:02:00.159 --> 00:02:07.359
For the past six-ish months, companies have treated heavy AI token use as proof of productivity.
00:02:07.519 --> 00:02:13.919
And that's because token prices themselves have fell like 200 times, right?
00:02:14.080 --> 00:02:25.919
If you remember back, you know, the earlier versions of the GPT technology were like$30 per million input and like$60 per million token output.
00:02:26.000 --> 00:02:28.719
And now it's like three and eight.
00:02:28.960 --> 00:02:29.520
Uh right.
00:02:29.599 --> 00:02:40.719
So token prices have fallen significantly, but the token usage has also gone up like 100 to 200x as models get more agentic.
00:02:40.879 --> 00:02:50.400
And now that models are reasoning by default, AI agents make the number of tokens per task absolutely explode.
00:02:50.719 --> 00:03:02.560
So I think now in mid-2026 and through the rest of the year, I think the goal, if it hasn't already in your organization, shifted away from token maxing in toward token efficiency.
00:03:02.719 --> 00:03:09.280
So stick with us on today's show, and you're gonna learn what token maxing is and how it led companies to waste a ton of money.
00:03:09.439 --> 00:03:17.199
You're gonna learn even more fundamentally what a token is and the four main ways that AI models gobble those token ups, tokens up.
00:03:17.360 --> 00:03:26.400
You're gonna know why companies are shifting from token volume to token efficiency and how to turn AI spending into real measurable results.
00:03:26.560 --> 00:03:29.680
Let's get into it and start here with the basics.
00:03:29.840 --> 00:03:39.280
Yes, this episode is part of our Start Here series, the essential series to both learn the AI basics and to double down on your AI knowledge.
00:03:39.439 --> 00:03:43.280
So if that's what you're trying to do, make sure you go to starthirseries.com.
00:03:43.360 --> 00:03:48.639
That is going to give you free access to our exclusive inner circle community.
00:03:48.800 --> 00:03:49.360
It's free.
00:03:49.520 --> 00:03:53.520
There's no other way to get access aside from going to starthearseries.com.
00:03:53.680 --> 00:04:02.000
And once you sign up, you will be inserted in to our Start Here Series space where every single episode is there for you.
00:04:02.080 --> 00:04:04.159
You can digest them all very easily.
00:04:04.240 --> 00:04:07.199
We have a playlist there as well that we keep updated.
00:04:07.439 --> 00:04:09.120
So, Start Here Series.
00:04:09.199 --> 00:04:19.680
If you missed our last episode, we talked about the 2026 LLM cheat code and 10 essential steps to get the most out of any AI chatbot.
00:04:19.759 --> 00:04:26.480
And if I'm being honest, that's probably one of the best shows we've ever done here out of 790 episodes.
00:04:26.560 --> 00:04:29.120
So make sure you check uh that one out.
00:04:29.199 --> 00:04:32.480
That's volume 26 and episode 786.
00:04:32.800 --> 00:04:35.120
But today, let's talk token maxing.
00:04:35.279 --> 00:04:42.399
So, token maxing is well, thinking that however many tokens you spend means something.
00:04:42.720 --> 00:04:53.680
And in rare cases, it might, but I would say in most cases, it just means that employees have been given a green light to use as much AI as possible.
00:04:53.839 --> 00:05:04.639
They're probably not closely monitoring what their agents are doing, and in some cases, it's more of you know people trying to pound their chest and talk about how many tokens they're using.
00:05:04.800 --> 00:05:18.959
Uh, that's because, yeah, companies, I think, especially when it came to you know clawed code uh in clawed co-work of late 2025 and early 2026, that was kind of the ushering in of the new era of these agents that can run all the time.
00:05:19.199 --> 00:05:31.519
So, you know, you saw people sharing, oh, I had my agent run for 30 minutes, for 60 minutes, for five hours, and then there was this rush for enterprises to well, let's do that, because clearly that has to mean something.
00:05:31.680 --> 00:05:41.040
And it was the assumption that if you use a lot of tokens or you know, if your agents were constantly running around the clock, that means that the company was making money.
00:05:41.199 --> 00:05:50.160
But obviously, those two things do not correlate, and I think some of the big companies set a bad example.
00:05:50.319 --> 00:06:01.519
One of the most prominent examples, well, we saw some uh some reporting that meta reportedly built an internal leaderboard ranking employees by tokens used.
00:06:01.759 --> 00:06:09.519
So one engineer reportedly ran 281 billion tokens in a single month.
00:06:09.920 --> 00:06:20.079
And some employees, well, they wanted to climb this internal leading board, uh, leaderboard because you know, not just that meta, but a lot of other companies that were doing something similar.
00:06:20.240 --> 00:06:33.360
You know, it if you weren't using a lot of tokens, developers and well, you know, people that were being pushed to use AI models felt if they weren't using enough tokens, then maybe their job was at risk.
00:06:33.519 --> 00:06:37.920
So you had stories come out that employees were just running empty agentic loops, right?
00:06:38.000 --> 00:06:48.000
Having it, you know, intentionally trip up, uh, you know, do some of their personal errands, but giving it impossible tasks just to make sure that they were using uh those tokens.
00:06:48.079 --> 00:06:56.639
So they weren't actually creating new business value for their company, they were just trying to light tokens on fire.
00:06:57.439 --> 00:07:01.199
All right, so now you know the trend.
00:07:01.519 --> 00:07:07.199
Let's uh double click into the basics, right?
00:07:07.279 --> 00:07:08.639
This is the start here series.
00:07:08.720 --> 00:07:13.439
This is for both beginners and for uh you know experts trying to double down.
00:07:13.519 --> 00:07:15.279
And depending on when you're listening to this, right?
00:07:15.360 --> 00:07:18.079
As of the recording, it's early June 2026.
00:07:18.160 --> 00:07:22.560
So if you're listening to this in December, maybe the tokenization story has changed.
00:07:22.720 --> 00:07:29.360
But in simple terms, here's what a token is: it's essentially a small piece of text that an AI model reads or writes.
00:07:29.600 --> 00:07:38.639
So essentially, if you you know put a big prompt to ChatGPT or Gemini or Claude or Copilot and you insert some of your company's documents, right?
00:07:38.800 --> 00:07:43.920
These models don't technically read it, they convert everything into tokens.
00:07:44.079 --> 00:07:50.879
So they think in tokens, they technically read in tokens, and they output or talk back in tokens.
00:07:51.040 --> 00:07:55.360
So one token is about three-quarters of an English word, right?
00:07:55.600 --> 00:08:01.759
So uh you could say a million tokens is about 750,000 words, give or take, right?
00:08:01.839 --> 00:08:03.839
And models charge a different weight.
00:08:04.000 --> 00:08:08.959
We're gonna talk about some of these model subsidies and how they're kind of going away, uh, right.
00:08:09.199 --> 00:08:14.959
But your usage or what you're actually paying for is ultimately converted into tokens.
00:08:15.279 --> 00:08:22.959
So now that you know that that is kind of the currency of a token, let's break down how these models use tokens, right?
00:08:23.040 --> 00:08:29.040
And that's obviously changed because you know, two or so years ago, it was a very straightforward transaction, right?
00:08:29.120 --> 00:08:32.879
It was literally you could measure it in those words, right?
00:08:33.039 --> 00:08:40.320
Hey, I, you know, I pasted a hundred words to you know chat GPT, 75 tokens.
00:08:40.480 --> 00:08:43.679
It gave me 100 words back, another 75 tokens.
00:08:43.840 --> 00:08:48.320
Now it's not as easy because the models are getting more and more capable.
00:08:48.559 --> 00:08:54.240
So let me just give you a um an example, right?
00:08:54.639 --> 00:08:58.559
So uh Codex and OpenAI recently ran a promotion.
00:08:58.639 --> 00:09:00.399
I'm on the$200 a month plan.
00:09:00.480 --> 00:09:01.759
This part's kind of important.
00:09:01.840 --> 00:09:08.320
Um, and they had a couple incentives going where, you know, essentially they uh gave some multiples to some people.
00:09:08.399 --> 00:09:10.159
Um, and I was one of those people.
00:09:10.240 --> 00:09:15.120
So I was able to essentially go through maybe about a billion tokens a day, right?
00:09:15.200 --> 00:09:20.639
On some days where I was running certain processes a little more heavily, not even a ton of coding projects.
00:09:20.799 --> 00:09:25.360
I've just been automating a lot of my manual knowledge work, which has been great.
00:09:25.440 --> 00:09:25.919
Uh, right.
00:09:26.080 --> 00:09:34.639
So think of a billion tokens, and it depends on your input, output, and how much reasoning you're using, uh, you know, how agentic your task is.
00:09:34.720 --> 00:09:44.159
But think of about a billion tokens, somewhere from like 12,000 to maybe$17,000 if you were paying for those straight via the API.
00:09:44.320 --> 00:09:44.799
All right.
00:09:44.879 --> 00:09:55.519
And this part's important because early on in the early days of AI and we're getting out of those nice days, essentially, companies would subsidize those tokens via a subscription plan.
00:09:55.679 --> 00:10:02.320
So I'm wasn't paying$12,000 to$17,000 a day for these tokens, obviously, right?
00:10:02.480 --> 00:10:07.679
We don't have, you know, 15 ads on our daily podcast like other podcasts do, right?
00:10:07.840 --> 00:10:11.519
You know, we're not making$17,000 a day in ads.
00:10:11.759 --> 00:10:21.759
Um, so a lot of companies have been subsidizing those tokens and you know, as a way to attract and retain customers, you know, it's kind of the dance, especially.
00:10:21.919 --> 00:10:33.919
I think OpenAI, uh, Google and Anthropic have been taking different stances on it, especially as of recently, because many companies, specifically Anthropic, they've been kind of the strictest on those usage limits.
00:10:34.159 --> 00:10:35.600
Google, after their I.O.
00:10:35.759 --> 00:10:44.159
conference on their$20 a month plan, they started to um implement some actual usage limits, whereas before they really didn't.
00:10:44.320 --> 00:10:48.720
And Microsoft, even with their Microsoft GitHub copilot, they're starting to do that too.
00:10:48.879 --> 00:10:53.360
So before, on certain paid plans, you really had some wide open usage.
00:10:53.519 --> 00:10:56.639
Uh, now they're really starting to meter it, right?
00:10:56.720 --> 00:11:00.639
So uh, you know, before it's kind of like, well, taking a lift, right?
00:11:00.720 --> 00:11:05.440
It's almost like you had a monthly subscription to lift and you can just take it from any point.
00:11:05.600 --> 00:11:10.320
Now, some of the companies are more taking it to be the metered taxi approach, right?
00:11:10.480 --> 00:11:34.080
OpenAI, I think, is still probably the most generous in subsidizing those subscriptions, but eventually a lot of those subscriptions or those those subsidies are going to change because someone's paying for it, and that's essentially a customer acquisition or customer retention costs that companies are ultimately paying to keep their millions or you know, eventually billions of customers happy.
00:11:34.240 --> 00:11:37.279
Um, but many tier, so that's a personal plan, right?
00:11:37.360 --> 00:11:56.559
Personal plans work a little bit different than you know, big business or enterprise plans, because many enterprise plans, you know, you're you may get a certain allotment of tokens uh per seat, but once an individual surpasses that, then they're charged at the normal taxi rate, right?
00:11:56.639 --> 00:12:01.919
So it's like, all right, well, you get you know 10 free lift rides a month or 10 free Waymo rides a month.
00:12:02.000 --> 00:12:11.279
But then after that, not only are you paying the metered taxi rate, but the meter is gonna be running all the time because if you're running agents, those things are running around the clock.
00:12:11.600 --> 00:12:15.840
So let's talk about how these tokens are fully spent.
00:12:16.000 --> 00:12:21.200
I want to break this down in hopefully a very uh easy and simple to understand way.
00:12:21.360 --> 00:12:25.039
So there's depending on how technical we want to get, I'm gonna keep it easy.
00:12:25.200 --> 00:12:27.360
There's essentially four different types of tokens, right?
00:12:27.440 --> 00:12:31.200
Because everyone's like, well, how do I know how I'm using tokens?
00:12:31.360 --> 00:12:38.639
Well, basically anytime you or the model is working, tokens are essentially being um counted up.
00:12:38.720 --> 00:12:38.960
All right.
00:12:39.120 --> 00:12:42.799
So you don't only have to worry about it uh from the cost side.
00:12:42.879 --> 00:12:52.559
So if you are paying strictly via API, so if your company is, you know, um working just on the back end via the API, you're paying strictly API costs.
00:12:52.720 --> 00:12:59.679
Um, if you're on a personal subscription, for the most part, you're just, you know, you just have your allotment and your usage.
00:12:59.759 --> 00:13:02.639
And if you're on a team plan, it might be a hybrid of those things.
00:13:02.799 --> 00:13:07.759
But every input that you send, every message is converted to tokens, right?
00:13:07.919 --> 00:13:14.000
Um, so longer prompts and larger files obviously raise those input, uh, those input costs.
00:13:14.159 --> 00:13:26.240
And even how different models tokenize or understand things really vary from model to model, uh, or sorry, not just model provider to model provider, but also the actual model to model within the same provider.
00:13:26.480 --> 00:13:31.600
You know, some uh you know, models cache things a little differently, right?
00:13:31.679 --> 00:13:37.120
So, like a browser cache that remembers things so it doesn't have to reload that web page every time uh every time.
00:13:37.360 --> 00:13:49.200
Sometimes, you know, depending on how you have things set up, how you have your connectors set up, if you have a very long chat thread going, um, it might rerun those tokens every single time, right?
00:13:49.279 --> 00:14:00.720
Like if you have a uh browser that crashes and you had uh, you know, 50 windows open and you reload it, it might load every single one of those 50 windows and it could be very slow.
00:14:00.879 --> 00:14:06.000
So depending on how a model caches things, it might actually impact how many tokens are being used.
00:14:06.159 --> 00:14:10.399
And that's important because the token usage also impacts the context window.
00:14:10.480 --> 00:14:20.399
So that's essentially how much information a model can remember fairly accurately before it starts to lose uh information at the top of the context.
00:14:20.799 --> 00:14:25.039
Okay, so that's um that's driven by tokens, right?
00:14:25.200 --> 00:14:29.759
So the most common way and the easiest way to explain is your input tokens.
00:14:30.000 --> 00:14:31.919
The second type is output.
00:14:32.080 --> 00:14:35.200
So that is what a large language model gives back to you.
00:14:35.360 --> 00:14:38.960
Sometimes that's text, sometimes it's code, right?
00:14:39.120 --> 00:14:41.840
Sometimes in theory, it's a video, right?
00:14:42.000 --> 00:14:43.360
Sometimes it's an image.
00:14:43.519 --> 00:14:50.399
So, you know, in those cases, a little bit harder uh to count and understand the token usage there.
00:14:50.639 --> 00:14:56.480
Um, but this is where it starts to get expensive, right?
00:14:56.559 --> 00:15:01.600
And I remember being at the uh uh NVIDIA GTC event uh might have been last year.
00:15:01.759 --> 00:15:13.600
Yeah, I think you know, uh NVIDIA CEO Jensen Wong kind of had a graph up there and saying, yes, the price of tokens is going down, but the amount of tokens used is going up.
00:15:13.759 --> 00:15:18.320
So yes, we've seen roughly a 10x decrease in the cost of tokens.
00:15:18.480 --> 00:15:22.480
So you may say, oh, well, what's the harm with token maxing then?
00:15:22.720 --> 00:15:26.639
Well, that's because of token type three that is reasoning.
00:15:26.799 --> 00:15:34.639
And because of these models that now reason by default, whereas before they were kind of just like, you know, super smart next token predictors, right?
00:15:34.799 --> 00:15:37.279
It was text in, text out.
00:15:38.080 --> 00:15:39.759
Now, not so much.
00:15:40.000 --> 00:15:46.240
Obviously, the type of inputs can cost more to tokenize, the type of outputs, right?
00:15:46.320 --> 00:15:52.720
If it builds audio, video, an interactive website, right?
00:15:52.799 --> 00:15:54.799
But now these models think by default.
00:15:54.960 --> 00:16:10.960
And that's where we sometimes see what Jensen Wong and others have talked about 100x or more um kind of output that an agenc model requires versus a you know traditional, non-reasoning, non-thinking transformer model.
00:16:11.120 --> 00:16:13.600
And these new models reason internally.
00:16:13.759 --> 00:16:22.480
Sometimes you can see these models reason by looking at their chain of thought, and you can observe and see all these different tools it calls and things like that.
00:16:22.639 --> 00:16:25.120
Sometimes it doesn't, but these models actually think.
00:16:25.279 --> 00:16:31.039
And even though you might not be able to count in theory the words of those thinking, it is eating those up.
00:16:31.200 --> 00:16:33.279
Those tokens are being consumed.
00:16:33.440 --> 00:16:42.320
And then last but not least, I kind of just alluded to this because you can sometimes trace this uh in the chain of thought, and that is the tool use, right?
00:16:42.399 --> 00:16:43.600
That's the harness.
00:16:43.759 --> 00:16:55.759
That's you know, as an example, you know, if you're running something like Codecs or Claude Code, you know, it's not just the model that thinks, it's thinking and it's using tools.
00:16:55.919 --> 00:17:16.240
You know, if you give it special access to skills, sometimes those skills will run multiple apps, uh, they'll run you know different plugins, um, you know, calling different tools, searching things on the web, using computer vision, using computer use, all of these different tools are obviously very token intensive.
00:17:16.559 --> 00:17:22.880
That's because AI agents can use tools, they can read files, they can run code, and then they can repeat steps automatically.
00:17:23.039 --> 00:17:35.119
And then when we talk about scheduling agents, right, and you talk about running agents on an automation so many times, business leaders aren't even always looking at these things, right?
00:17:35.359 --> 00:17:46.400
As an example, if you have a scheduled agent, you know, and it's pulling from your dynamic data in one of its jobs, and I do this all the time, but luckily I'm on, you know, Codex.
00:17:46.640 --> 00:17:47.200
I love it.
00:17:47.359 --> 00:17:49.759
I'm on the$200 a month uh plan.
00:17:50.000 --> 00:17:53.200
So I don't usually have to really worry about my limits, right?
00:17:53.279 --> 00:17:59.279
But a lot of times people they'll set up powerful agentic loops and it's working with your dynamic data.
00:17:59.359 --> 00:18:07.359
So maybe you have something every single day that, you know, triages your company's, you know, SharePoint, OneDrive.
00:18:07.519 --> 00:18:11.519
Maybe there's a team inbox and then it creates a live dashboard, right?
00:18:11.599 --> 00:18:15.920
And it updates every day, or maybe it updates every single hour, right?
00:18:16.079 --> 00:18:20.799
That's an agent going in there and ingesting a lot of input tokens.
00:18:20.960 --> 00:18:24.720
It's um exerting, I guess, a lot of output tokens.
00:18:24.880 --> 00:18:29.440
It is using a lot of thinking tokens, and it is calling a lot of tools.
00:18:29.599 --> 00:18:38.160
So it is doing all of those things on a loop, scheduled, and let alone being efficient, which we're gonna get to here in a minute, right?
00:18:38.480 --> 00:18:44.640
Because especially as humans get more hands off, right, which I think is the wrong approach.
00:18:44.960 --> 00:18:55.680
As models get more autonomous and more powerful, the expert-driven loops should be expert-driven, not the lazy human in the loop that I absolutely despise, right?
00:18:55.759 --> 00:18:58.160
But well, humans take their hands off.
00:18:58.319 --> 00:19:10.319
So a lot of these processes are happening on repeats in a loop, and humans just look at the output and they don't necessarily say, oh, well, maybe this did cost us a billion tokens a day, right?
00:19:10.480 --> 00:19:13.440
And costs like that can really add up.
00:19:14.799 --> 00:19:22.640
So that was kind of the story, the token maxing story, because it became this bragging right.
00:19:22.880 --> 00:19:29.759
You know, you saw these leaderboards, people would post, you know, oh, I used, you know, this many tokens this week.
00:19:29.920 --> 00:19:32.400
And a lot of times it was the big labs, right?
00:19:32.480 --> 00:19:37.359
The people actually building these things, kind of wagging or showing how many tokens they're using.
00:19:37.519 --> 00:19:46.319
And well, a lot of times, you know, your Fortune 500 type companies are looking at what big tech is doing and assuming that they just have to follow suit.
00:19:46.960 --> 00:19:49.440
And that doesn't always turn out that well.
00:19:49.599 --> 00:19:55.759
There's actually an Axios report uh that just came out talking about an unnamed company.
00:19:56.079 --> 00:20:06.319
You know, there's a lot of uh thought out there or talk out there on if this story is actually true or not, but Axios is generally a pretty good media outlet, uh, better than a lot of the others.
00:20:06.480 --> 00:20:18.960
And they said that a company reportedly wasn't monitoring or setting spending limits on Claude, and they reportedly used$500 million in AI usage.
00:20:19.599 --> 00:20:29.279
One company in one month by not properly monitoring this specific model is very token hunt, right?
00:20:29.519 --> 00:20:31.680
Claude is like me at 4 p.m.
00:20:31.839 --> 00:20:34.960
If I haven't eaten, I'm angry and I want to eat everything.
00:20:35.599 --> 00:20:36.160
All right.
00:20:36.880 --> 00:20:45.200
So here's kind of what I want to talk about shifting toward token efficiency.
00:20:45.440 --> 00:20:50.400
So don't get me wrong, I am not anti using a lot of tokens.
00:20:50.880 --> 00:20:51.440
I'm not.
00:20:52.400 --> 00:20:52.720
Right?
00:20:52.880 --> 00:21:00.720
Because I think that there is obviously a certain amount of experimentation in token burn that needs to be done, right?
00:21:00.880 --> 00:21:10.880
A good example, if you're in marketing or advertising, let's say you're starting a new campaign on Google Ads or something like that, or you know, an advertising campaign on Meta.
00:21:11.279 --> 00:21:15.440
A lot of times you have to just burn a lot of money, right?
00:21:15.599 --> 00:21:26.319
To, you know, kind of uh train the algorithm on what you want, what you don't want, you know, types of conversions for your company to have enough data to know what works and what doesn't.
00:21:26.480 --> 00:21:30.559
And technically using using tokens is the same, right?
00:21:30.799 --> 00:21:45.039
Because you don't know as models capabilities are constantly changing, you know, where your company or your department is going to be able to derive the most value unless you are experimenting, unless you're, well, maybe burning through a lot of tokens.
00:21:45.279 --> 00:21:53.920
But if your strategy, if your company strategy is token maxing, right, it's not gonna work, right?
00:21:54.079 --> 00:21:58.640
You have to start looking at a certain point at token efficiency because.
00:22:00.160 --> 00:22:02.880
Even if you've been able to afford it, right?
00:22:03.039 --> 00:22:10.240
Even if your company, you know, everyone's on a uh highly subsidized plan and you're not having to pay a lot of API overages.
00:22:10.720 --> 00:22:17.359
The thought is all these companies eventually, the free ride is gonna be a little less free.
00:22:17.519 --> 00:22:26.319
Um, and as these models become more capable, they're going to cost more to run because they're gonna think more, they're gonna become more and more powerful.
00:22:26.400 --> 00:22:26.880
Uh, right.
00:22:26.960 --> 00:22:33.119
So token efficiency means getting the most useful work per token spent, right?
00:22:33.359 --> 00:22:44.799
My thought is I like to think that there's a certain price that you would pay for a certain output.
00:22:44.880 --> 00:22:47.200
Let's say you're gonna hire a consultancy, right?
00:22:47.680 --> 00:22:52.559
Maybe you have to think, okay, are we gonna hire a big four consultancy?
00:22:52.720 --> 00:22:55.440
Are we gonna hire a small team of full-time employees?
00:22:55.599 --> 00:23:00.720
Or are we maybe gonna hire an agency or a group of contractors for a certain project, right?
00:23:00.799 --> 00:23:04.640
That's a normal decision that a lot of companies have to go for.
00:23:04.720 --> 00:23:07.119
And they probably say, this is the output.
00:23:07.279 --> 00:23:08.319
The output is X.
00:23:08.799 --> 00:23:14.400
We have three different routes that we can go, but no matter what, we have to monitor our cost.
00:23:14.640 --> 00:23:20.799
And, you know, essentially that output is a certain piece of intelligence or an artifact that our company needs.
00:23:20.960 --> 00:23:25.279
You need to start looking at your AI usage in the same way.
00:23:25.599 --> 00:23:32.160
Not just, all right, let's give everyone Claude Desktop, let's give everyone codecs and go to town.
00:23:32.400 --> 00:23:48.240
Don't get me wrong, your company should be doing those things, but doing it in a responsible way, uh, because the future of work, I think, is in these extremely powerful desktop harnesses that can be always on and take advantage of your company's dynamic data.
00:23:48.400 --> 00:23:53.039
But you have to understand that it's not just about running through.
00:23:53.119 --> 00:24:08.160
It's about what those tokens are actually doing, measuring what they're creating, knowing and being able to equate what those tokens create in new economic value for your company versus how much it costs a human to do the same thing without any AI, right?
00:24:08.319 --> 00:24:16.880
What did that artifact, what did that output, what did that client deliverable, what did that RFP cost in 2020, right?
00:24:17.119 --> 00:24:31.519
And then being able to put a price on what a human would cost pre-AI, you know, knowing how your um whatever AI system you're using, how much it costs from a token or a usage perspective to do the same thing.
00:24:31.759 --> 00:24:35.039
You have to start looking at it in a very pragmatic and measured out way.
00:24:35.200 --> 00:24:45.359
We've actually had a couple of episodes, including one in the Start Here series, specifically talking about measuring ROI, but that is the accepted output.
00:24:45.519 --> 00:24:51.279
So the measure is the accepted output against the total cost, time, and rework, right?
00:24:51.359 --> 00:25:03.759
So you have to measure it, what it costs for a human to do it, no AI, what it costs for an augmented, uh, you know, kind of a best human with the best non-agentic AI.
00:25:03.839 --> 00:25:08.160
And then you need to measure it with the agentic AI with the expert-driven loop.
00:25:08.319 --> 00:25:17.839
So you have to know there's gonna be a different mix, but you have to know the right model, use the right way, just beats simply using more AI every single time.
00:25:18.559 --> 00:25:20.319
So, a couple things.
00:25:20.480 --> 00:25:35.759
I'm not gonna say use this model for this, but I think there's been a lot of um, I don't know if it's disinformation necessarily, but a lot of misinformation floating out there around what model is the best to use for what reasons, right?
00:25:35.920 --> 00:25:49.119
And I think we've seen a shift, and I'm probably gonna do a start here series episode on this very soon, about the shift away from models to the harness, because I don't think models are as important as they were anymore.
00:25:49.200 --> 00:26:00.480
Because especially these, you know, point upgrades, you know, going from you know, GPT 5.3 to 5.4 and Opus 4.7 to Opus 4.8, are the models better?
00:26:00.559 --> 00:26:00.880
Yes.
00:26:01.039 --> 00:26:02.319
Are the benchmarks better?
00:26:02.640 --> 00:26:03.119
Sure.
00:26:03.279 --> 00:26:08.720
Uh, right, but ultimately it's the harness, I think, uh, that makes the biggest difference.
00:26:08.960 --> 00:26:14.799
Anyways, you still have to look at the price per intelligence, right?
00:26:15.200 --> 00:26:17.359
Are those tokens worth it?
00:26:17.680 --> 00:26:22.559
Or if you give a a model a certain task, is it efficient?
00:26:22.799 --> 00:26:24.880
Is it just gonna run in loops all day?
00:26:25.119 --> 00:26:44.400
Or right, if you give a model uh a task and it gets it done in one pass without misfiring 20 times, without having to, you know, unnecessarily call seven tools that it didn't really need because it was rushing to the conclusion instead of breaking it down in a proper way, is that the right model?
00:26:44.559 --> 00:26:46.960
And I think there's a couple important metrics to look at.
00:26:47.039 --> 00:26:48.240
We're gonna tackle two here.
00:26:48.400 --> 00:26:52.640
So if you listen to the show at all, you know a lot of times they talk about the artificial analysis.
00:26:52.799 --> 00:26:56.079
So there's the artificial analysis intelligence score.
00:26:56.160 --> 00:26:58.319
So that's essentially a series of tests.
00:26:58.400 --> 00:27:00.240
Um, and then models get a score.
00:27:00.319 --> 00:27:05.680
And one of the most important uh, you know, metrics to look at undoubtedly is just that score.
00:27:05.839 --> 00:27:13.599
So that's ultimately how intelligent is a model, nothing else considered, which is a good starting point, right?
00:27:13.839 --> 00:27:19.440
Especially if you're on a highly subsidized plan that you don't have to worry about anything else.
00:27:19.599 --> 00:27:21.039
But that's not the reality.
00:27:21.200 --> 00:27:33.440
And well, that you know, fantasy AI setup that we all had in mid-2025 through uh maybe like five months ago, is gonna slowly start to go away.
00:27:33.519 --> 00:27:40.240
So you have to now start it looking at what does that model actually produce uh per token?
00:27:40.400 --> 00:27:40.640
Right.
00:27:40.720 --> 00:27:58.960
So that's why the artificial analysis, that same exact test that I talked about, uh, right, where it's essentially a conglomerate in artificial analysis, the company runs these, you know, these, you know, roughly 10 or 12 different tests, and they count how much money because they're running this via the API.
00:27:59.119 --> 00:28:07.279
So they know not just the score, the intelligence, because that's important, but how much money did they have to spend to get that level of intelligence?
00:28:07.519 --> 00:28:13.039
So artificial analysis also has this uh in intelligence versus cost.
00:28:13.200 --> 00:28:21.440
And so one thing you'll see the new Opus 4.8 uh did beat out the new the uh OpenAI 5.5.
00:28:21.599 --> 00:28:23.599
Not by a lot, but by a solid point.
00:28:23.759 --> 00:28:40.799
One point is not a big jump in the artificial analysis intelligence index, but it's you know big enough to say yes, when it comes to just strictly intelligence and intelligence only, Claude Opus 4.8 is the best across all of these benchmarks, which is an important thing to talk about.
00:28:40.960 --> 00:28:54.400
But the cost, it is absolutely bonkers because the anthropic oh the anthropic models are, and you can see this by tests, and people always this is annoying, y'all.
00:28:54.559 --> 00:28:57.920
Like people always accuse me of being oh, anti-anthropic.
00:28:58.000 --> 00:29:00.160
You you love Google, you love open AI.
00:29:00.960 --> 00:29:03.359
No, cloud's a great model, right?
00:29:03.680 --> 00:29:12.000
But it is incredibly token inefficient, and if you are on a a paid plan, uh anthropic's plans are the least subsidized, right?
00:29:12.160 --> 00:29:18.960
That's maybe why they're the most most profitable uh AI company per user right now, presumably.
00:29:19.200 --> 00:29:36.000
But in terms of if you are out there uh you know on a subsidized plan, not only is it you're gonna get the least for your dollar, but if you're paying on it through the API, all you're doing, you, your company, is saying, let's just go ahead and flush this money down the toilet because it doesn't matter.
00:29:36.160 --> 00:29:46.559
Because the facts say, the stats say, the unbiased third parties say that you are literally lighting your money on fire if you are choosing to use the anthropic models via the API, right?
00:29:46.720 --> 00:29:52.799
Yes, maybe there's a capability that the harness, you know, Claude Desktop has that no one else has.
00:29:53.039 --> 00:29:54.000
Sure, right?
00:29:54.319 --> 00:30:00.720
But if you have built your company's AI strategy in a modular way, which is the right way to do it, right?
00:30:00.960 --> 00:30:09.759
And you are still using anthropics models, you are essentially saying, by all measures, that you just want to waste money, right?
00:30:09.920 --> 00:30:13.279
Here's another uh example also from artificial analysis.
00:30:13.359 --> 00:30:15.119
This is the total cost, right?
00:30:15.279 --> 00:30:17.119
The total cost to get to this.
00:30:17.279 --> 00:30:19.599
So anthropics models are the most expensive.
00:30:19.680 --> 00:30:24.319
So to run that test, about$5,100 for Opus 4.7.
00:30:24.480 --> 00:30:30.000
Comparatively, OpenAI's latest model, uh,$3,300.
00:30:31.279 --> 00:30:40.559
So I'm not the best at math, but that is 50% more expensive to get that same level of intelligence, right?
00:30:40.720 --> 00:30:44.799
That's like if you're saying, okay, I'm gonna am I gonna take an Uber or a Lyft?
00:30:45.039 --> 00:30:53.119
Well, the Lyft is 50% more expensive, and they're gonna get you there in roughly the same time.
00:30:53.279 --> 00:30:57.759
Well, not actually, because there's also a speed factor, but I don't want to focus on speed today.
00:30:58.000 --> 00:31:02.559
But from a cost per intelligence ratio, right?
00:31:02.720 --> 00:31:08.400
And this is where when we talk about token efficiency, this is why you have to keep up.
00:31:08.559 --> 00:31:10.960
This is why you have to build in a smart way.
00:31:11.119 --> 00:31:15.759
You don't want to lock your company into a certain vendor if you don't have to, right?
00:31:15.839 --> 00:31:22.559
Obviously, if you're using something on the desktop side, but if you were building on the API side, or you know, maybe you're using cursor, right?
00:31:22.640 --> 00:31:24.319
Where you can use a number of models.
00:31:24.480 --> 00:31:35.279
You're using, you know, Copilot that can use the uh GPT models, you can use the um claw models, you can use the, you know, the Microsoft models, right?
00:31:35.440 --> 00:31:37.119
You have to know these things.
00:31:37.279 --> 00:31:43.920
You have to understand how much does the intelligence cost, not only which model is the most intelligent.
00:31:44.880 --> 00:31:50.720
Another great um metric to look at, and I'll probably be talking about this a little bit more, is Deep Suite.
00:31:50.880 --> 00:31:59.759
So, Deep Suite is a new, and I'd say it's probably one of the better uh agenc um coding benchmarks.
00:31:59.920 --> 00:32:04.880
So there's an older benchmark called Suibench that I think is actually antiquated.
00:32:04.960 --> 00:32:08.240
Um, and Deep Suite is actually a really good benchmark.
00:32:08.400 --> 00:32:10.000
Maybe I'll do an entire show on that.
00:32:10.160 --> 00:32:12.079
Let me know in the comments if you want me to do it.
00:32:12.160 --> 00:32:14.079
Just leave Deep Suite.
00:32:14.880 --> 00:32:18.880
Uh, so that's Deep SWE, uh, if you care about this, right?
00:32:18.960 --> 00:32:25.119
Uh but this is a great benchmark uh that shows something similar to the artificial analysis.
00:32:25.279 --> 00:32:28.160
But the reality is, at least when it comes to token efficiency, right?
00:32:28.240 --> 00:32:30.880
I'm not gonna tell you use this model for this, right?
00:32:31.039 --> 00:32:52.319
But in the same way that the artificial uh analysis, actually, Gemini 3.1 Pro, one of the best models out there in terms of uh, you know, getting for what you pay for, because it costs even uh a fourth, uh just about a fourth of what GPT 5.5 costs, right?
00:32:52.480 --> 00:33:00.559
And it costs a fraction, uh, you know, not even uh 20, you know.
00:33:00.720 --> 00:33:04.559
Uh yeah, it's it's uh let me let me do my math here.
00:33:04.640 --> 00:33:13.440
So that is about more than 5x uh more to use Opus 4.7.
00:33:13.519 --> 00:33:23.759
You're gonna pay 5x more, not 50, 5x more to get the same level of intelligence out of Opus 4.7 versus Gemini 3.1 Pro.
00:33:23.839 --> 00:33:27.839
Uh so if we get back to deep suite, the same thing shows true, right?
00:33:28.160 --> 00:33:39.440
GPT 5.5 gets a not only a higher score, much higher than the Opus models, but it does so at again a fraction of the cost, right?
00:33:39.680 --> 00:33:43.519
So uh it's more than uh double, right?
00:33:43.680 --> 00:33:49.519
So you're getting not only a higher level of intelligence, uh, but you're about half the cost.
00:33:49.759 --> 00:33:57.200
So when it comes to token efficiency, it's not just about what your company gives you access to.
00:33:57.359 --> 00:34:16.960
Because, like I said, you always need to be building modularly, because what happens if a model, you know, if you've uh set up a big chunk of your company, whether it's running somewhat autonomously, you know, a bunch of you know, cron AI jobs, whatever it may be, if GBT56 breaks your flow, right?
00:34:17.119 --> 00:34:22.960
If if you know mythos comes out and nothing works, you have to have a fallback, right?
00:34:23.039 --> 00:34:27.360
So you should always be testing these things, but not just what is the most intelligent model.
00:34:27.519 --> 00:34:29.039
That's a good place to start.
00:34:29.280 --> 00:34:33.280
But the cost per intelligence is an extremely important thing.
00:34:33.440 --> 00:34:55.119
And that is the prudent and the responsible, and I think will be the rest of the story, the telling takeaway through the rest of 2026 is setting your setting your company up to not only be AI native, but to be token efficient because the AI subsidy days are gonna be over fairly soon, and you have to be ready.
00:34:55.360 --> 00:35:01.679
So your company needs to measure the accepted outputs and the AI cost, not just the number of tokens used.
00:35:01.840 --> 00:35:03.760
You have to match the model to the job.
00:35:03.920 --> 00:35:11.920
You can use smaller, less expensive models for certain things, parsing PDFs, you know, doing simple text summarization.
00:35:12.000 --> 00:35:19.519
And you can use those bigger, more expensive models like, you know, Opus and GPT-5.5 for those more complex tasks.
00:35:19.679 --> 00:35:23.920
You need to start monitoring uh, you know, these agents that are stuck in a loop.
00:35:24.000 --> 00:35:30.719
You need to start maybe kicking out certain models that just constantly are calling tools that they maybe don't need to, uh, right?
00:35:30.880 --> 00:35:33.119
You have to be paying attention to the chain of thought.
00:35:33.199 --> 00:35:39.920
Uh, and then you need to review your AI costs because these long-running agents, it is so easy.
00:35:40.159 --> 00:35:51.440
It's so easy as these models become autonomous, like you can in a in a Waymo or you know, in that version of Tesla's cars that I don't know if they ship, but they keep saying, right?
00:35:51.599 --> 00:35:53.920
It's almost like your hands off the wheel, right?
00:35:54.079 --> 00:35:57.280
You can completely hands off the wheel doing something else.
00:35:57.599 --> 00:36:01.360
You can't do that right now with a Gentec AI.
00:36:01.440 --> 00:36:09.599
You can't just be a passive human in the loop because that is gonna cost your company probably millions of dollars in the long run.
00:36:09.760 --> 00:36:20.159
You have to have an expert-driven loop that is constantly monitoring not just the inputs and the outputs, but monitoring the efficiency of those tokens as well.
00:36:20.400 --> 00:36:26.159
Start doing that now, and you are not gonna be behind for the rest of the year.
00:36:26.320 --> 00:36:27.440
You are gonna be ahead.
00:36:27.519 --> 00:36:29.199
And that is where I want you to be.
00:36:29.280 --> 00:36:42.239
Uh, you know, one quick tidbit to wrap this up uh was actually uh, you know, just got out of a great session uh last night with the mic uh Microsoft CTO Kevin Scott.
00:36:42.320 --> 00:36:48.400
And he kind of ended uh, you know, one of his, you know, just just a little uh gathering at a bar.
00:36:48.480 --> 00:36:53.039
But, you know, really just talking about how activity does not equal value.
00:36:53.199 --> 00:37:15.679
And I think that's a great takeaway to end this on because just because you're using the best models and you have all of these scheduled prompts and you have all of these agents running around the clock, don't confuse activity for value and especially pay attention to what model you are having be active for you and the value that it is actually creating.
00:37:15.840 --> 00:37:17.360
All right, I hope this one was helpful.
00:37:17.519 --> 00:37:21.519
If so, let me know if you could like and subscribe to the show.
00:37:21.599 --> 00:37:25.119
If you're listening on Spotify or Apple Podcasts, I'd really appreciate that.
00:37:25.440 --> 00:37:36.639
And then make sure you please go to starthairseries.com to get exclusive access to our inner circle community so you can start here series along with everyone else that's doing it.
00:37:36.800 --> 00:37:37.840
So thank you for tuning in.
00:37:37.920 --> 00:37:40.639
I hope to see you back tomorrow and every day for more everyday AI.
00:37:40.719 --> 00:37:41.519
Thanks, y'all.
00:00:01.760 --> 00:00:08.000
There's been this pseudo badge of AI honor over the past six months that makes no sense.
00:00:08.160 --> 00:00:14.320
And I think it has led a lot of enterprise companies astray when choosing the right AI strategy.
00:00:14.720 --> 00:00:16.160
What is that thing?
00:00:16.399 --> 00:00:18.480
It's token maxing.
00:00:18.719 --> 00:00:34.719
So token maxing is the corporate practice where companies push AI token consumption and correlate that AI usage as a direct metric for employee productivity, or pointing to the fact that they're doing something positive with their AI spend.
00:00:34.880 --> 00:00:47.840
Like if you're using billions of AI tokens a day, that means that you're living inside of large language models and clearly you are creating huge value for the company because more prompts has to mean more ROI, right?
00:00:48.560 --> 00:01:00.000
But that strange AI power flex of early 2026 has quickly revealed itself to be a false and sometimes dangerous metric by mid-2026.
00:01:00.079 --> 00:01:12.000
Because it turns out all that token maxing leads to is a competition to see who can run the longest agentic loops, regardless of if it actually accomplishes a company's goal.
00:01:12.159 --> 00:01:13.599
So what's the new metric?
00:01:13.760 --> 00:01:15.680
What should your company be looking at?
00:01:15.920 --> 00:01:17.359
That's token efficiency.
00:01:17.519 --> 00:01:31.519
In other words, getting the most bang for your AI token usage buck means not just burning AI usage for the sake of using it, but instead aligning your company's AI strategy with the right model, used the right way for the right job.
00:01:31.760 --> 00:01:32.239
Weird.
00:01:32.480 --> 00:01:37.680
Common AI Sense pays off more than a leaderboard showing who is using the most tokens.
00:01:37.920 --> 00:01:56.079
So on today's show, we're going to break down the rise and fall of token maxing, not only showing you what tokens are and how they work, but also uncovering the proper best practices to put your tokens toward actual measurable outputs versus just lighting millions of dollars on fire.
00:01:56.400 --> 00:01:59.599
So here's the big picture.
00:02:00.159 --> 00:02:07.359
For the past six-ish months, companies have treated heavy AI token use as proof of productivity.
00:02:07.519 --> 00:02:13.919
And that's because token prices themselves have fell like 200 times, right?
00:02:14.080 --> 00:02:25.919
If you remember back, you know, the earlier versions of the GPT technology were like$30 per million input and like$60 per million token output.
00:02:26.000 --> 00:02:28.719
And now it's like three and eight.
00:02:28.960 --> 00:02:29.520
Uh right.
00:02:29.599 --> 00:02:40.719
So token prices have fallen significantly, but the token usage has also gone up like 100 to 200x as models get more agentic.
00:02:40.879 --> 00:02:50.400
And now that models are reasoning by default, AI agents make the number of tokens per task absolutely explode.
00:02:50.719 --> 00:03:02.560
So I think now in mid-2026 and through the rest of the year, I think the goal, if it hasn't already in your organization, shifted away from token maxing in toward token efficiency.
00:03:02.719 --> 00:03:09.280
So stick with us on today's show, and you're gonna learn what token maxing is and how it led companies to waste a ton of money.
00:03:09.439 --> 00:03:17.199
You're gonna learn even more fundamentally what a token is and the four main ways that AI models gobble those token ups, tokens up.
00:03:17.360 --> 00:03:26.400
You're gonna know why companies are shifting from token volume to token efficiency and how to turn AI spending into real measurable results.
00:03:26.560 --> 00:03:29.680
Let's get into it and start here with the basics.
00:03:29.840 --> 00:03:39.280
Yes, this episode is part of our Start Here series, the essential series to both learn the AI basics and to double down on your AI knowledge.
00:03:39.439 --> 00:03:43.280
So if that's what you're trying to do, make sure you go to starthirseries.com.
00:03:43.360 --> 00:03:48.639
That is going to give you free access to our exclusive inner circle community.
00:03:48.800 --> 00:03:49.360
It's free.
00:03:49.520 --> 00:03:53.520
There's no other way to get access aside from going to starthearseries.com.
00:03:53.680 --> 00:04:02.000
And once you sign up, you will be inserted in to our Start Here Series space where every single episode is there for you.
00:04:02.080 --> 00:04:04.159
You can digest them all very easily.
00:04:04.240 --> 00:04:07.199
We have a playlist there as well that we keep updated.
00:04:07.439 --> 00:04:09.120
So, Start Here Series.
00:04:09.199 --> 00:04:19.680
If you missed our last episode, we talked about the 2026 LLM cheat code and 10 essential steps to get the most out of any AI chatbot.
00:04:19.759 --> 00:04:26.480
And if I'm being honest, that's probably one of the best shows we've ever done here out of 790 episodes.
00:04:26.560 --> 00:04:29.120
So make sure you check uh that one out.
00:04:29.199 --> 00:04:32.480
That's volume 26 and episode 786.
00:04:32.800 --> 00:04:35.120
But today, let's talk token maxing.
00:04:35.279 --> 00:04:42.399
So, token maxing is well, thinking that however many tokens you spend means something.
00:04:42.720 --> 00:04:53.680
And in rare cases, it might, but I would say in most cases, it just means that employees have been given a green light to use as much AI as possible.
00:04:53.839 --> 00:05:04.639
They're probably not closely monitoring what their agents are doing, and in some cases, it's more of you know people trying to pound their chest and talk about how many tokens they're using.
00:05:04.800 --> 00:05:18.959
Uh, that's because, yeah, companies, I think, especially when it came to you know clawed code uh in clawed co-work of late 2025 and early 2026, that was kind of the ushering in of the new era of these agents that can run all the time.
00:05:19.199 --> 00:05:31.519
So, you know, you saw people sharing, oh, I had my agent run for 30 minutes, for 60 minutes, for five hours, and then there was this rush for enterprises to well, let's do that, because clearly that has to mean something.
00:05:31.680 --> 00:05:41.040
And it was the assumption that if you use a lot of tokens or you know, if your agents were constantly running around the clock, that means that the company was making money.
00:05:41.199 --> 00:05:50.160
But obviously, those two things do not correlate, and I think some of the big companies set a bad example.
00:05:50.319 --> 00:06:01.519
One of the most prominent examples, well, we saw some uh some reporting that meta reportedly built an internal leaderboard ranking employees by tokens used.
00:06:01.759 --> 00:06:09.519
So one engineer reportedly ran 281 billion tokens in a single month.
00:06:09.920 --> 00:06:20.079
And some employees, well, they wanted to climb this internal leading board, uh, leaderboard because you know, not just that meta, but a lot of other companies that were doing something similar.
00:06:20.240 --> 00:06:33.360
You know, it if you weren't using a lot of tokens, developers and well, you know, people that were being pushed to use AI models felt if they weren't using enough tokens, then maybe their job was at risk.
00:06:33.519 --> 00:06:37.920
So you had stories come out that employees were just running empty agentic loops, right?
00:06:38.000 --> 00:06:48.000
Having it, you know, intentionally trip up, uh, you know, do some of their personal errands, but giving it impossible tasks just to make sure that they were using uh those tokens.
00:06:48.079 --> 00:06:56.639
So they weren't actually creating new business value for their company, they were just trying to light tokens on fire.
00:06:57.439 --> 00:07:01.199
All right, so now you know the trend.
00:07:01.519 --> 00:07:07.199
Let's uh double click into the basics, right?
00:07:07.279 --> 00:07:08.639
This is the start here series.
00:07:08.720 --> 00:07:13.439
This is for both beginners and for uh you know experts trying to double down.
00:07:13.519 --> 00:07:15.279
And depending on when you're listening to this, right?
00:07:15.360 --> 00:07:18.079
As of the recording, it's early June 2026.
00:07:18.160 --> 00:07:22.560
So if you're listening to this in December, maybe the tokenization story has changed.
00:07:22.720 --> 00:07:29.360
But in simple terms, here's what a token is: it's essentially a small piece of text that an AI model reads or writes.
00:07:29.600 --> 00:07:38.639
So essentially, if you you know put a big prompt to ChatGPT or Gemini or Claude or Copilot and you insert some of your company's documents, right?
00:07:38.800 --> 00:07:43.920
These models don't technically read it, they convert everything into tokens.
00:07:44.079 --> 00:07:50.879
So they think in tokens, they technically read in tokens, and they output or talk back in tokens.
00:07:51.040 --> 00:07:55.360
So one token is about three-quarters of an English word, right?
00:07:55.600 --> 00:08:01.759
So uh you could say a million tokens is about 750,000 words, give or take, right?
00:08:01.839 --> 00:08:03.839
And models charge a different weight.
00:08:04.000 --> 00:08:08.959
We're gonna talk about some of these model subsidies and how they're kind of going away, uh, right.
00:08:09.199 --> 00:08:14.959
But your usage or what you're actually paying for is ultimately converted into tokens.
00:08:15.279 --> 00:08:22.959
So now that you know that that is kind of the currency of a token, let's break down how these models use tokens, right?
00:08:23.040 --> 00:08:29.040
And that's obviously changed because you know, two or so years ago, it was a very straightforward transaction, right?
00:08:29.120 --> 00:08:32.879
It was literally you could measure it in those words, right?
00:08:33.039 --> 00:08:40.320
Hey, I, you know, I pasted a hundred words to you know chat GPT, 75 tokens.
00:08:40.480 --> 00:08:43.679
It gave me 100 words back, another 75 tokens.
00:08:43.840 --> 00:08:48.320
Now it's not as easy because the models are getting more and more capable.
00:08:48.559 --> 00:08:54.240
So let me just give you a um an example, right?
00:08:54.639 --> 00:08:58.559
So uh Codex and OpenAI recently ran a promotion.
00:08:58.639 --> 00:09:00.399
I'm on the$200 a month plan.
00:09:00.480 --> 00:09:01.759
This part's kind of important.
00:09:01.840 --> 00:09:08.320
Um, and they had a couple incentives going where, you know, essentially they uh gave some multiples to some people.
00:09:08.399 --> 00:09:10.159
Um, and I was one of those people.
00:09:10.240 --> 00:09:15.120
So I was able to essentially go through maybe about a billion tokens a day, right?
00:09:15.200 --> 00:09:20.639
On some days where I was running certain processes a little more heavily, not even a ton of coding projects.
00:09:20.799 --> 00:09:25.360
I've just been automating a lot of my manual knowledge work, which has been great.
00:09:25.440 --> 00:09:25.919
Uh, right.
00:09:26.080 --> 00:09:34.639
So think of a billion tokens, and it depends on your input, output, and how much reasoning you're using, uh, you know, how agentic your task is.
00:09:34.720 --> 00:09:44.159
But think of about a billion tokens, somewhere from like 12,000 to maybe$17,000 if you were paying for those straight via the API.
00:09:44.320 --> 00:09:44.799
All right.
00:09:44.879 --> 00:09:55.519
And this part's important because early on in the early days of AI and we're getting out of those nice days, essentially, companies would subsidize those tokens via a subscription plan.
00:09:55.679 --> 00:10:02.320
So I'm wasn't paying$12,000 to$17,000 a day for these tokens, obviously, right?
00:10:02.480 --> 00:10:07.679
We don't have, you know, 15 ads on our daily podcast like other podcasts do, right?
00:10:07.840 --> 00:10:11.519
You know, we're not making$17,000 a day in ads.
00:10:11.759 --> 00:10:21.759
Um, so a lot of companies have been subsidizing those tokens and you know, as a way to attract and retain customers, you know, it's kind of the dance, especially.
00:10:21.919 --> 00:10:33.919
I think OpenAI, uh, Google and Anthropic have been taking different stances on it, especially as of recently, because many companies, specifically Anthropic, they've been kind of the strictest on those usage limits.
00:10:34.159 --> 00:10:35.600
Google, after their I.O.
00:10:35.759 --> 00:10:44.159
conference on their$20 a month plan, they started to um implement some actual usage limits, whereas before they really didn't.
00:10:44.320 --> 00:10:48.720
And Microsoft, even with their Microsoft GitHub copilot, they're starting to do that too.
00:10:48.879 --> 00:10:53.360
So before, on certain paid plans, you really had some wide open usage.
00:10:53.519 --> 00:10:56.639
Uh, now they're really starting to meter it, right?
00:10:56.720 --> 00:11:00.639
So uh, you know, before it's kind of like, well, taking a lift, right?
00:11:00.720 --> 00:11:05.440
It's almost like you had a monthly subscription to lift and you can just take it from any point.
00:11:05.600 --> 00:11:10.320
Now, some of the companies are more taking it to be the metered taxi approach, right?
00:11:10.480 --> 00:11:34.080
OpenAI, I think, is still probably the most generous in subsidizing those subscriptions, but eventually a lot of those subscriptions or those those subsidies are going to change because someone's paying for it, and that's essentially a customer acquisition or customer retention costs that companies are ultimately paying to keep their millions or you know, eventually billions of customers happy.
00:11:34.240 --> 00:11:37.279
Um, but many tier, so that's a personal plan, right?
00:11:37.360 --> 00:11:56.559
Personal plans work a little bit different than you know, big business or enterprise plans, because many enterprise plans, you know, you're you may get a certain allotment of tokens uh per seat, but once an individual surpasses that, then they're charged at the normal taxi rate, right?
00:11:56.639 --> 00:12:01.919
So it's like, all right, well, you get you know 10 free lift rides a month or 10 free Waymo rides a month.
00:12:02.000 --> 00:12:11.279
But then after that, not only are you paying the metered taxi rate, but the meter is gonna be running all the time because if you're running agents, those things are running around the clock.
00:12:11.600 --> 00:12:15.840
So let's talk about how these tokens are fully spent.
00:12:16.000 --> 00:12:21.200
I want to break this down in hopefully a very uh easy and simple to understand way.
00:12:21.360 --> 00:12:25.039
So there's depending on how technical we want to get, I'm gonna keep it easy.
00:12:25.200 --> 00:12:27.360
There's essentially four different types of tokens, right?
00:12:27.440 --> 00:12:31.200
Because everyone's like, well, how do I know how I'm using tokens?
00:12:31.360 --> 00:12:38.639
Well, basically anytime you or the model is working, tokens are essentially being um counted up.
00:12:38.720 --> 00:12:38.960
All right.
00:12:39.120 --> 00:12:42.799
So you don't only have to worry about it uh from the cost side.
00:12:42.879 --> 00:12:52.559
So if you are paying strictly via API, so if your company is, you know, um working just on the back end via the API, you're paying strictly API costs.
00:12:52.720 --> 00:12:59.679
Um, if you're on a personal subscription, for the most part, you're just, you know, you just have your allotment and your usage.
00:12:59.759 --> 00:13:02.639
And if you're on a team plan, it might be a hybrid of those things.
00:13:02.799 --> 00:13:07.759
But every input that you send, every message is converted to tokens, right?
00:13:07.919 --> 00:13:14.000
Um, so longer prompts and larger files obviously raise those input, uh, those input costs.
00:13:14.159 --> 00:13:26.240
And even how different models tokenize or understand things really vary from model to model, uh, or sorry, not just model provider to model provider, but also the actual model to model within the same provider.
00:13:26.480 --> 00:13:31.600
You know, some uh you know, models cache things a little differently, right?
00:13:31.679 --> 00:13:37.120
So, like a browser cache that remembers things so it doesn't have to reload that web page every time uh every time.
00:13:37.360 --> 00:13:49.200
Sometimes, you know, depending on how you have things set up, how you have your connectors set up, if you have a very long chat thread going, um, it might rerun those tokens every single time, right?
00:13:49.279 --> 00:14:00.720
Like if you have a uh browser that crashes and you had uh, you know, 50 windows open and you reload it, it might load every single one of those 50 windows and it could be very slow.
00:14:00.879 --> 00:14:06.000
So depending on how a model caches things, it might actually impact how many tokens are being used.
00:14:06.159 --> 00:14:10.399
And that's important because the token usage also impacts the context window.
00:14:10.480 --> 00:14:20.399
So that's essentially how much information a model can remember fairly accurately before it starts to lose uh information at the top of the context.
00:14:20.799 --> 00:14:25.039
Okay, so that's um that's driven by tokens, right?
00:14:25.200 --> 00:14:29.759
So the most common way and the easiest way to explain is your input tokens.
00:14:30.000 --> 00:14:31.919
The second type is output.
00:14:32.080 --> 00:14:35.200
So that is what a large language model gives back to you.
00:14:35.360 --> 00:14:38.960
Sometimes that's text, sometimes it's code, right?
00:14:39.120 --> 00:14:41.840
Sometimes in theory, it's a video, right?
00:14:42.000 --> 00:14:43.360
Sometimes it's an image.
00:14:43.519 --> 00:14:50.399
So, you know, in those cases, a little bit harder uh to count and understand the token usage there.
00:14:50.639 --> 00:14:56.480
Um, but this is where it starts to get expensive, right?
00:14:56.559 --> 00:15:01.600
And I remember being at the uh uh NVIDIA GTC event uh might have been last year.
00:15:01.759 --> 00:15:13.600
Yeah, I think you know, uh NVIDIA CEO Jensen Wong kind of had a graph up there and saying, yes, the price of tokens is going down, but the amount of tokens used is going up.
00:15:13.759 --> 00:15:18.320
So yes, we've seen roughly a 10x decrease in the cost of tokens.
00:15:18.480 --> 00:15:22.480
So you may say, oh, well, what's the harm with token maxing then?
00:15:22.720 --> 00:15:26.639
Well, that's because of token type three that is reasoning.
00:15:26.799 --> 00:15:34.639
And because of these models that now reason by default, whereas before they were kind of just like, you know, super smart next token predictors, right?
00:15:34.799 --> 00:15:37.279
It was text in, text out.
00:15:38.080 --> 00:15:39.759
Now, not so much.
00:15:40.000 --> 00:15:46.240
Obviously, the type of inputs can cost more to tokenize, the type of outputs, right?
00:15:46.320 --> 00:15:52.720
If it builds audio, video, an interactive website, right?
00:15:52.799 --> 00:15:54.799
But now these models think by default.
00:15:54.960 --> 00:16:10.960
And that's where we sometimes see what Jensen Wong and others have talked about 100x or more um kind of output that an agenc model requires versus a you know traditional, non-reasoning, non-thinking transformer model.
00:16:11.120 --> 00:16:13.600
And these new models reason internally.
00:16:13.759 --> 00:16:22.480
Sometimes you can see these models reason by looking at their chain of thought, and you can observe and see all these different tools it calls and things like that.
00:16:22.639 --> 00:16:25.120
Sometimes it doesn't, but these models actually think.
00:16:25.279 --> 00:16:31.039
And even though you might not be able to count in theory the words of those thinking, it is eating those up.
00:16:31.200 --> 00:16:33.279
Those tokens are being consumed.
00:16:33.440 --> 00:16:42.320
And then last but not least, I kind of just alluded to this because you can sometimes trace this uh in the chain of thought, and that is the tool use, right?
00:16:42.399 --> 00:16:43.600
That's the harness.
00:16:43.759 --> 00:16:55.759
That's you know, as an example, you know, if you're running something like Codecs or Claude Code, you know, it's not just the model that thinks, it's thinking and it's using tools.
00:16:55.919 --> 00:17:16.240
You know, if you give it special access to skills, sometimes those skills will run multiple apps, uh, they'll run you know different plugins, um, you know, calling different tools, searching things on the web, using computer vision, using computer use, all of these different tools are obviously very token intensive.
00:17:16.559 --> 00:17:22.880
That's because AI agents can use tools, they can read files, they can run code, and then they can repeat steps automatically.
00:17:23.039 --> 00:17:35.119
And then when we talk about scheduling agents, right, and you talk about running agents on an automation so many times, business leaders aren't even always looking at these things, right?
00:17:35.359 --> 00:17:46.400
As an example, if you have a scheduled agent, you know, and it's pulling from your dynamic data in one of its jobs, and I do this all the time, but luckily I'm on, you know, Codex.
00:17:46.640 --> 00:17:47.200
I love it.
00:17:47.359 --> 00:17:49.759
I'm on the$200 a month uh plan.
00:17:50.000 --> 00:17:53.200
So I don't usually have to really worry about my limits, right?
00:17:53.279 --> 00:17:59.279
But a lot of times people they'll set up powerful agentic loops and it's working with your dynamic data.
00:17:59.359 --> 00:18:07.359
So maybe you have something every single day that, you know, triages your company's, you know, SharePoint, OneDrive.
00:18:07.519 --> 00:18:11.519
Maybe there's a team inbox and then it creates a live dashboard, right?
00:18:11.599 --> 00:18:15.920
And it updates every day, or maybe it updates every single hour, right?
00:18:16.079 --> 00:18:20.799
That's an agent going in there and ingesting a lot of input tokens.
00:18:20.960 --> 00:18:24.720
It's um exerting, I guess, a lot of output tokens.
00:18:24.880 --> 00:18:29.440
It is using a lot of thinking tokens, and it is calling a lot of tools.
00:18:29.599 --> 00:18:38.160
So it is doing all of those things on a loop, scheduled, and let alone being efficient, which we're gonna get to here in a minute, right?
00:18:38.480 --> 00:18:44.640
Because especially as humans get more hands off, right, which I think is the wrong approach.
00:18:44.960 --> 00:18:55.680
As models get more autonomous and more powerful, the expert-driven loops should be expert-driven, not the lazy human in the loop that I absolutely despise, right?
00:18:55.759 --> 00:18:58.160
But well, humans take their hands off.
00:18:58.319 --> 00:19:10.319
So a lot of these processes are happening on repeats in a loop, and humans just look at the output and they don't necessarily say, oh, well, maybe this did cost us a billion tokens a day, right?
00:19:10.480 --> 00:19:13.440
And costs like that can really add up.
00:19:14.799 --> 00:19:22.640
So that was kind of the story, the token maxing story, because it became this bragging right.
00:19:22.880 --> 00:19:29.759
You know, you saw these leaderboards, people would post, you know, oh, I used, you know, this many tokens this week.
00:19:29.920 --> 00:19:32.400
And a lot of times it was the big labs, right?
00:19:32.480 --> 00:19:37.359
The people actually building these things, kind of wagging or showing how many tokens they're using.
00:19:37.519 --> 00:19:46.319
And well, a lot of times, you know, your Fortune 500 type companies are looking at what big tech is doing and assuming that they just have to follow suit.
00:19:46.960 --> 00:19:49.440
And that doesn't always turn out that well.
00:19:49.599 --> 00:19:55.759
There's actually an Axios report uh that just came out talking about an unnamed company.
00:19:56.079 --> 00:20:06.319
You know, there's a lot of uh thought out there or talk out there on if this story is actually true or not, but Axios is generally a pretty good media outlet, uh, better than a lot of the others.
00:20:06.480 --> 00:20:18.960
And they said that a company reportedly wasn't monitoring or setting spending limits on Claude, and they reportedly used$500 million in AI usage.
00:20:19.599 --> 00:20:29.279
One company in one month by not properly monitoring this specific model is very token hunt, right?
00:20:29.519 --> 00:20:31.680
Claude is like me at 4 p.m.
00:20:31.839 --> 00:20:34.960
If I haven't eaten, I'm angry and I want to eat everything.
00:20:35.599 --> 00:20:36.160
All right.
00:20:36.880 --> 00:20:45.200
So here's kind of what I want to talk about shifting toward token efficiency.
00:20:45.440 --> 00:20:50.400
So don't get me wrong, I am not anti using a lot of tokens.
00:20:50.880 --> 00:20:51.440
I'm not.
00:20:52.400 --> 00:20:52.720
Right?
00:20:52.880 --> 00:21:00.720
Because I think that there is obviously a certain amount of experimentation in token burn that needs to be done, right?
00:21:00.880 --> 00:21:10.880
A good example, if you're in marketing or advertising, let's say you're starting a new campaign on Google Ads or something like that, or you know, an advertising campaign on Meta.
00:21:11.279 --> 00:21:15.440
A lot of times you have to just burn a lot of money, right?
00:21:15.599 --> 00:21:26.319
To, you know, kind of uh train the algorithm on what you want, what you don't want, you know, types of conversions for your company to have enough data to know what works and what doesn't.
00:21:26.480 --> 00:21:30.559
And technically using using tokens is the same, right?
00:21:30.799 --> 00:21:45.039
Because you don't know as models capabilities are constantly changing, you know, where your company or your department is going to be able to derive the most value unless you are experimenting, unless you're, well, maybe burning through a lot of tokens.
00:21:45.279 --> 00:21:53.920
But if your strategy, if your company strategy is token maxing, right, it's not gonna work, right?
00:21:54.079 --> 00:21:58.640
You have to start looking at a certain point at token efficiency because.
00:22:00.160 --> 00:22:02.880
Even if you've been able to afford it, right?
00:22:03.039 --> 00:22:10.240
Even if your company, you know, everyone's on a uh highly subsidized plan and you're not having to pay a lot of API overages.
00:22:10.720 --> 00:22:17.359
The thought is all these companies eventually, the free ride is gonna be a little less free.
00:22:17.519 --> 00:22:26.319
Um, and as these models become more capable, they're going to cost more to run because they're gonna think more, they're gonna become more and more powerful.
00:22:26.400 --> 00:22:26.880
Uh, right.
00:22:26.960 --> 00:22:33.119
So token efficiency means getting the most useful work per token spent, right?
00:22:33.359 --> 00:22:44.799
My thought is I like to think that there's a certain price that you would pay for a certain output.
00:22:44.880 --> 00:22:47.200
Let's say you're gonna hire a consultancy, right?
00:22:47.680 --> 00:22:52.559
Maybe you have to think, okay, are we gonna hire a big four consultancy?
00:22:52.720 --> 00:22:55.440
Are we gonna hire a small team of full-time employees?
00:22:55.599 --> 00:23:00.720
Or are we maybe gonna hire an agency or a group of contractors for a certain project, right?
00:23:00.799 --> 00:23:04.640
That's a normal decision that a lot of companies have to go for.
00:23:04.720 --> 00:23:07.119
And they probably say, this is the output.
00:23:07.279 --> 00:23:08.319
The output is X.
00:23:08.799 --> 00:23:14.400
We have three different routes that we can go, but no matter what, we have to monitor our cost.
00:23:14.640 --> 00:23:20.799
And, you know, essentially that output is a certain piece of intelligence or an artifact that our company needs.
00:23:20.960 --> 00:23:25.279
You need to start looking at your AI usage in the same way.
00:23:25.599 --> 00:23:32.160
Not just, all right, let's give everyone Claude Desktop, let's give everyone codecs and go to town.
00:23:32.400 --> 00:23:48.240
Don't get me wrong, your company should be doing those things, but doing it in a responsible way, uh, because the future of work, I think, is in these extremely powerful desktop harnesses that can be always on and take advantage of your company's dynamic data.
00:23:48.400 --> 00:23:53.039
But you have to understand that it's not just about running through.
00:23:53.119 --> 00:24:08.160
It's about what those tokens are actually doing, measuring what they're creating, knowing and being able to equate what those tokens create in new economic value for your company versus how much it costs a human to do the same thing without any AI, right?
00:24:08.319 --> 00:24:16.880
What did that artifact, what did that output, what did that client deliverable, what did that RFP cost in 2020, right?
00:24:17.119 --> 00:24:31.519
And then being able to put a price on what a human would cost pre-AI, you know, knowing how your um whatever AI system you're using, how much it costs from a token or a usage perspective to do the same thing.
00:24:31.759 --> 00:24:35.039
You have to start looking at it in a very pragmatic and measured out way.
00:24:35.200 --> 00:24:45.359
We've actually had a couple of episodes, including one in the Start Here series, specifically talking about measuring ROI, but that is the accepted output.
00:24:45.519 --> 00:24:51.279
So the measure is the accepted output against the total cost, time, and rework, right?
00:24:51.359 --> 00:25:03.759
So you have to measure it, what it costs for a human to do it, no AI, what it costs for an augmented, uh, you know, kind of a best human with the best non-agentic AI.
00:25:03.839 --> 00:25:08.160
And then you need to measure it with the agentic AI with the expert-driven loop.
00:25:08.319 --> 00:25:17.839
So you have to know there's gonna be a different mix, but you have to know the right model, use the right way, just beats simply using more AI every single time.
00:25:18.559 --> 00:25:20.319
So, a couple things.
00:25:20.480 --> 00:25:35.759
I'm not gonna say use this model for this, but I think there's been a lot of um, I don't know if it's disinformation necessarily, but a lot of misinformation floating out there around what model is the best to use for what reasons, right?
00:25:35.920 --> 00:25:49.119
And I think we've seen a shift, and I'm probably gonna do a start here series episode on this very soon, about the shift away from models to the harness, because I don't think models are as important as they were anymore.
00:25:49.200 --> 00:26:00.480
Because especially these, you know, point upgrades, you know, going from you know, GPT 5.3 to 5.4 and Opus 4.7 to Opus 4.8, are the models better?
00:26:00.559 --> 00:26:00.880
Yes.
00:26:01.039 --> 00:26:02.319
Are the benchmarks better?
00:26:02.640 --> 00:26:03.119
Sure.
00:26:03.279 --> 00:26:08.720
Uh, right, but ultimately it's the harness, I think, uh, that makes the biggest difference.
00:26:08.960 --> 00:26:14.799
Anyways, you still have to look at the price per intelligence, right?
00:26:15.200 --> 00:26:17.359
Are those tokens worth it?
00:26:17.680 --> 00:26:22.559
Or if you give a a model a certain task, is it efficient?
00:26:22.799 --> 00:26:24.880
Is it just gonna run in loops all day?
00:26:25.119 --> 00:26:44.400
Or right, if you give a model uh a task and it gets it done in one pass without misfiring 20 times, without having to, you know, unnecessarily call seven tools that it didn't really need because it was rushing to the conclusion instead of breaking it down in a proper way, is that the right model?
00:26:44.559 --> 00:26:46.960
And I think there's a couple important metrics to look at.
00:26:47.039 --> 00:26:48.240
We're gonna tackle two here.
00:26:48.400 --> 00:26:52.640
So if you listen to the show at all, you know a lot of times they talk about the artificial analysis.
00:26:52.799 --> 00:26:56.079
So there's the artificial analysis intelligence score.
00:26:56.160 --> 00:26:58.319
So that's essentially a series of tests.
00:26:58.400 --> 00:27:00.240
Um, and then models get a score.
00:27:00.319 --> 00:27:05.680
And one of the most important uh, you know, metrics to look at undoubtedly is just that score.
00:27:05.839 --> 00:27:13.599
So that's ultimately how intelligent is a model, nothing else considered, which is a good starting point, right?
00:27:13.839 --> 00:27:19.440
Especially if you're on a highly subsidized plan that you don't have to worry about anything else.
00:27:19.599 --> 00:27:21.039
But that's not the reality.
00:27:21.200 --> 00:27:33.440
And well, that you know, fantasy AI setup that we all had in mid-2025 through uh maybe like five months ago, is gonna slowly start to go away.
00:27:33.519 --> 00:27:40.240
So you have to now start it looking at what does that model actually produce uh per token?
00:27:40.400 --> 00:27:40.640
Right.
00:27:40.720 --> 00:27:58.960
So that's why the artificial analysis, that same exact test that I talked about, uh, right, where it's essentially a conglomerate in artificial analysis, the company runs these, you know, these, you know, roughly 10 or 12 different tests, and they count how much money because they're running this via the API.
00:27:59.119 --> 00:28:07.279
So they know not just the score, the intelligence, because that's important, but how much money did they have to spend to get that level of intelligence?
00:28:07.519 --> 00:28:13.039
So artificial analysis also has this uh in intelligence versus cost.
00:28:13.200 --> 00:28:21.440
And so one thing you'll see the new Opus 4.8 uh did beat out the new the uh OpenAI 5.5.
00:28:21.599 --> 00:28:23.599
Not by a lot, but by a solid point.
00:28:23.759 --> 00:28:40.799
One point is not a big jump in the artificial analysis intelligence index, but it's you know big enough to say yes, when it comes to just strictly intelligence and intelligence only, Claude Opus 4.8 is the best across all of these benchmarks, which is an important thing to talk about.
00:28:40.960 --> 00:28:54.400
But the cost, it is absolutely bonkers because the anthropic oh the anthropic models are, and you can see this by tests, and people always this is annoying, y'all.
00:28:54.559 --> 00:28:57.920
Like people always accuse me of being oh, anti-anthropic.
00:28:58.000 --> 00:29:00.160
You you love Google, you love open AI.
00:29:00.960 --> 00:29:03.359
No, cloud's a great model, right?
00:29:03.680 --> 00:29:12.000
But it is incredibly token inefficient, and if you are on a a paid plan, uh anthropic's plans are the least subsidized, right?
00:29:12.160 --> 00:29:18.960
That's maybe why they're the most most profitable uh AI company per user right now, presumably.
00:29:19.200 --> 00:29:36.000
But in terms of if you are out there uh you know on a subsidized plan, not only is it you're gonna get the least for your dollar, but if you're paying on it through the API, all you're doing, you, your company, is saying, let's just go ahead and flush this money down the toilet because it doesn't matter.
00:29:36.160 --> 00:29:46.559
Because the facts say, the stats say, the unbiased third parties say that you are literally lighting your money on fire if you are choosing to use the anthropic models via the API, right?
00:29:46.720 --> 00:29:52.799
Yes, maybe there's a capability that the harness, you know, Claude Desktop has that no one else has.
00:29:53.039 --> 00:29:54.000
Sure, right?
00:29:54.319 --> 00:30:00.720
But if you have built your company's AI strategy in a modular way, which is the right way to do it, right?
00:30:00.960 --> 00:30:09.759
And you are still using anthropics models, you are essentially saying, by all measures, that you just want to waste money, right?
00:30:09.920 --> 00:30:13.279
Here's another uh example also from artificial analysis.
00:30:13.359 --> 00:30:15.119
This is the total cost, right?
00:30:15.279 --> 00:30:17.119
The total cost to get to this.
00:30:17.279 --> 00:30:19.599
So anthropics models are the most expensive.
00:30:19.680 --> 00:30:24.319
So to run that test, about$5,100 for Opus 4.7.
00:30:24.480 --> 00:30:30.000
Comparatively, OpenAI's latest model, uh,$3,300.
00:30:31.279 --> 00:30:40.559
So I'm not the best at math, but that is 50% more expensive to get that same level of intelligence, right?
00:30:40.720 --> 00:30:44.799
That's like if you're saying, okay, I'm gonna am I gonna take an Uber or a Lyft?
00:30:45.039 --> 00:30:53.119
Well, the Lyft is 50% more expensive, and they're gonna get you there in roughly the same time.
00:30:53.279 --> 00:30:57.759
Well, not actually, because there's also a speed factor, but I don't want to focus on speed today.
00:30:58.000 --> 00:31:02.559
But from a cost per intelligence ratio, right?
00:31:02.720 --> 00:31:08.400
And this is where when we talk about token efficiency, this is why you have to keep up.
00:31:08.559 --> 00:31:10.960
This is why you have to build in a smart way.
00:31:11.119 --> 00:31:15.759
You don't want to lock your company into a certain vendor if you don't have to, right?
00:31:15.839 --> 00:31:22.559
Obviously, if you're using something on the desktop side, but if you were building on the API side, or you know, maybe you're using cursor, right?
00:31:22.640 --> 00:31:24.319
Where you can use a number of models.
00:31:24.480 --> 00:31:35.279
You're using, you know, Copilot that can use the uh GPT models, you can use the um claw models, you can use the, you know, the Microsoft models, right?
00:31:35.440 --> 00:31:37.119
You have to know these things.
00:31:37.279 --> 00:31:43.920
You have to understand how much does the intelligence cost, not only which model is the most intelligent.
00:31:44.880 --> 00:31:50.720
Another great um metric to look at, and I'll probably be talking about this a little bit more, is Deep Suite.
00:31:50.880 --> 00:31:59.759
So, Deep Suite is a new, and I'd say it's probably one of the better uh agenc um coding benchmarks.
00:31:59.920 --> 00:32:04.880
So there's an older benchmark called Suibench that I think is actually antiquated.
00:32:04.960 --> 00:32:08.240
Um, and Deep Suite is actually a really good benchmark.
00:32:08.400 --> 00:32:10.000
Maybe I'll do an entire show on that.
00:32:10.160 --> 00:32:12.079
Let me know in the comments if you want me to do it.
00:32:12.160 --> 00:32:14.079
Just leave Deep Suite.
00:32:14.880 --> 00:32:18.880
Uh, so that's Deep SWE, uh, if you care about this, right?
00:32:18.960 --> 00:32:25.119
Uh but this is a great benchmark uh that shows something similar to the artificial analysis.
00:32:25.279 --> 00:32:28.160
But the reality is, at least when it comes to token efficiency, right?
00:32:28.240 --> 00:32:30.880
I'm not gonna tell you use this model for this, right?
00:32:31.039 --> 00:32:52.319
But in the same way that the artificial uh analysis, actually, Gemini 3.1 Pro, one of the best models out there in terms of uh, you know, getting for what you pay for, because it costs even uh a fourth, uh just about a fourth of what GPT 5.5 costs, right?
00:32:52.480 --> 00:33:00.559
And it costs a fraction, uh, you know, not even uh 20, you know.
00:33:00.720 --> 00:33:04.559
Uh yeah, it's it's uh let me let me do my math here.
00:33:04.640 --> 00:33:13.440
So that is about more than 5x uh more to use Opus 4.7.
00:33:13.519 --> 00:33:23.759
You're gonna pay 5x more, not 50, 5x more to get the same level of intelligence out of Opus 4.7 versus Gemini 3.1 Pro.
00:33:23.839 --> 00:33:27.839
Uh so if we get back to deep suite, the same thing shows true, right?
00:33:28.160 --> 00:33:39.440
GPT 5.5 gets a not only a higher score, much higher than the Opus models, but it does so at again a fraction of the cost, right?
00:33:39.680 --> 00:33:43.519
So uh it's more than uh double, right?
00:33:43.680 --> 00:33:49.519
So you're getting not only a higher level of intelligence, uh, but you're about half the cost.
00:33:49.759 --> 00:33:57.200
So when it comes to token efficiency, it's not just about what your company gives you access to.
00:33:57.359 --> 00:34:16.960
Because, like I said, you always need to be building modularly, because what happens if a model, you know, if you've uh set up a big chunk of your company, whether it's running somewhat autonomously, you know, a bunch of you know, cron AI jobs, whatever it may be, if GBT56 breaks your flow, right?
00:34:17.119 --> 00:34:22.960
If if you know mythos comes out and nothing works, you have to have a fallback, right?
00:34:23.039 --> 00:34:27.360
So you should always be testing these things, but not just what is the most intelligent model.
00:34:27.519 --> 00:34:29.039
That's a good place to start.
00:34:29.280 --> 00:34:33.280
But the cost per intelligence is an extremely important thing.
00:34:33.440 --> 00:34:55.119
And that is the prudent and the responsible, and I think will be the rest of the story, the telling takeaway through the rest of 2026 is setting your setting your company up to not only be AI native, but to be token efficient because the AI subsidy days are gonna be over fairly soon, and you have to be ready.
00:34:55.360 --> 00:35:01.679
So your company needs to measure the accepted outputs and the AI cost, not just the number of tokens used.
00:35:01.840 --> 00:35:03.760
You have to match the model to the job.
00:35:03.920 --> 00:35:11.920
You can use smaller, less expensive models for certain things, parsing PDFs, you know, doing simple text summarization.
00:35:12.000 --> 00:35:19.519
And you can use those bigger, more expensive models like, you know, Opus and GPT-5.5 for those more complex tasks.
00:35:19.679 --> 00:35:23.920
You need to start monitoring uh, you know, these agents that are stuck in a loop.
00:35:24.000 --> 00:35:30.719
You need to start maybe kicking out certain models that just constantly are calling tools that they maybe don't need to, uh, right?
00:35:30.880 --> 00:35:33.119
You have to be paying attention to the chain of thought.
00:35:33.199 --> 00:35:39.920
Uh, and then you need to review your AI costs because these long-running agents, it is so easy.
00:35:40.159 --> 00:35:51.440
It's so easy as these models become autonomous, like you can in a in a Waymo or you know, in that version of Tesla's cars that I don't know if they ship, but they keep saying, right?
00:35:51.599 --> 00:35:53.920
It's almost like your hands off the wheel, right?
00:35:54.079 --> 00:35:57.280
You can completely hands off the wheel doing something else.
00:35:57.599 --> 00:36:01.360
You can't do that right now with a Gentec AI.
00:36:01.440 --> 00:36:09.599
You can't just be a passive human in the loop because that is gonna cost your company probably millions of dollars in the long run.
00:36:09.760 --> 00:36:20.159
You have to have an expert-driven loop that is constantly monitoring not just the inputs and the outputs, but monitoring the efficiency of those tokens as well.
00:36:20.400 --> 00:36:26.159
Start doing that now, and you are not gonna be behind for the rest of the year.
00:36:26.320 --> 00:36:27.440
You are gonna be ahead.
00:36:27.519 --> 00:36:29.199
And that is where I want you to be.
00:36:29.280 --> 00:36:42.239
Uh, you know, one quick tidbit to wrap this up uh was actually uh, you know, just got out of a great session uh last night with the mic uh Microsoft CTO Kevin Scott.
00:36:42.320 --> 00:36:48.400
And he kind of ended uh, you know, one of his, you know, just just a little uh gathering at a bar.
00:36:48.480 --> 00:36:53.039
But, you know, really just talking about how activity does not equal value.
00:36:53.199 --> 00:37:15.679
And I think that's a great takeaway to end this on because just because you're using the best models and you have all of these scheduled prompts and you have all of these agents running around the clock, don't confuse activity for value and especially pay attention to what model you are having be active for you and the value that it is actually creating.
00:37:15.840 --> 00:37:17.360
All right, I hope this one was helpful.
00:37:17.519 --> 00:37:21.519
If so, let me know if you could like and subscribe to the show.
00:37:21.599 --> 00:37:25.119
If you're listening on Spotify or Apple Podcasts, I'd really appreciate that.
00:37:25.440 --> 00:37:36.639
And then make sure you please go to starthairseries.com to get exclusive access to our inner circle community so you can start here series along with everyone else that's doing it.
00:37:36.800 --> 00:37:37.840
So thank you for tuning in.
00:37:37.920 --> 00:37:40.639
I hope to see you back tomorrow and every day for more everyday AI.
00:37:40.719 --> 00:37:41.519
Thanks, y'all.
