ABOUT THIS EPISODE
Ryan Wang started his PhD at MIT in 2018, right as the transformer architecture was reshaping AI, and spent late nights restarting GPU racks in a campus basement. Years later, the startup he co-founded, Eigen AI, would rank first among inference providers serving the same open model, earning a shoutout in Jensen Huang’s 2026 GTC keynote. Along the way he learned that at scale, the model is the easy part. The real challenge is in everything it takes to serve it.
In this episode, Anu and Markiesha sit down with Ryan, now the SVP of Applied AI at Nebius, to unpack why inference optimization has become one of the biggest levers in AI economics including how optimization is now squeezing four to five times more tokens out of the same GPUs, and why a fine-tuned 70B open model can beat frontier closed models on specific tasks.
Ryan also shares the prompt-design choice that keeps your cache intact, why cached context costs a tenth of recomputing it, and when founders should buy inference instead of building a team and delves into why cheaper tokens will grow the inference market rather than shrink it.
What You’ll Learn:
Why everything about inference changes at ten times the scale
The prompt-design choice that keeps your cache intact
How open models can outperform frontier closed models
When founders should buy inference as a service and when to build an in-house team
Why Ryan thinks the gap between open and closed models has shrunk from a year to a few months
The story behind Eigen AI’s monumental GTC shoutout
Why he sees dramatic expansion of the inference market
What changes he sees coming to inference infra, including agentic AI infrastructure
Learn more: www.nebius.com/podcast
