For our third Benchtalks, the series dedicated to the researchers building the measurement toolkits that frontier labs hill-climb on, Snorkel AI co-founder Vincent Sunn Chen sat down with Parth Asawa, a UC Berkeley PhD student advised by Matei Zaharia and Joey Gonzalez, and the creator of Continual Learning Bench, the first standardized benchmark for measuring whether AI systems actually learn from experience over time.
This interview covers:
- Why your coding agent re-reads your entire codebase from scratch every single session, and why a human developer stops doing that after task one
- How the field got so good at scaling that it quietly set aside the original question: can a model actually learn on its own after training ends?
- The poker task: if your opponent calls every hand, a human exploits it in minutes, and models run in separate sessions don't figure it out
- The gain metric: why cumulative reward alone can't tell you if a system is learning or just smarter to begin with, and how stateful minus stateless performance isolates actual learning
- What Anthropic's Fable release revealed, including how they used Continual Learning Bench to show Fable outperformed systems using Opus or Sonnet as a backbone
- Why in-context learning still leads the leaderboard, and why Parth's long-run bet is on parametric systems
- The false dichotomy in AI safety: unsafe open models vs. consolidated power, and what Parth and Joey Gonzalez propose as the third path
Full interview/transcript: snorkel.ai/blog/benchtalks-parth-asawa-continual-learning-bench/
