Should you worry about using the latest AI model?

New AI models are released every week. Is it worth it to keep up with the latest and greatest? Eric says no, John says yes, and they debate to find a practical middle ground.

Across all AI labs, a new model is released every few days. The frontier labs (OpenAI, Anthropic, and Google) release a major new model every 2 weeks. Keeping up with the latest and greatest can feel exhausting, and that's before you dig in to the details of why a new model is better, and what it's better at.

People with a Claude or ChatGPT subscription get access to new models automatically when they are made available through the respective apps, but many people still don't know if it's worth it to burn through credits faster with a more powerful engine under the hood. For anyone who has moved beyond the apps, the questions get even harder. Are open-weight models as good as the frontier? Is switching models worth the cost of changing your setup?

Eric and John stage a debate about whether you should keep up with the latest models.

John says yes:

You fail to understand what is possible if you're only ever optimizing for using the cheapest model. You need to be able to understand what you can do now that you couldn't do before.

Eric says no:

Someone who is extremely good at using AI can use a less capable model and do more than someone who is following the path of least resistance with the latest model. Before worrying about using the latest model, I would focus on becoming the type of AI user who can notice the differences.

By the end, the episode finds the middle ground of reality, derived directly from Eric and John's years of using AI in their daily work. As you become a more proficient user of AI, the right pattern is intentionally using different models for different parts of your workflow. In order to do that, you need to use mid-tier models to build core skills, experiment with the frontier to understand the limits of what's possible, and develop the discernment to know the best application for each.

Optimizing for cost obfuscates the art of the possible: A proficient AI user can do extraordinary things with a mid-tier model, but that's because they understand how capable models are. Trying to reach the ceiling on the frontier changes the way you think about AI as a medium and how you can apply it.

Optimizing for your own proficiency should be the default: When you begin to notice the differences in a new model, especially the subtle ones, you're starting to build the core skill necessary to get the most use out of any model.

Advanced AI users employ different models for different jobs: The proven pattern is using expensive frontier models for more critical work and cheaper, faster models for more standard tasks. Smart users learn to have frontier models build and audit workflows of cheaper models to get the most bang for their buck.

AI Gateway is a model router from Vercel that gives you access to 100s of models at cost. Routers like these are a great way to experiment with different models from different labs.

The AI Gateway Production Index (also from Vercel) is a monthly report that details trends in model usage across trillions of tokens per day, including adoption metrics for the latest releases.

Kimi K3 from Moonshot AI made big news when it achieved near-frontier performance at a fraction of the cost.

Ben Thompson writes Stratechery, and his recent article on the economics of the frontier labs vs open-weight models breaks down how the labs make money (hint, it's not by training the most powerful models).

... (Read more at the episode page)