MENGENAI EPISOD INI
What happens when your CI/CD pipeline can’t give you a simple yes or no? Tarek Ziadé, who leads CI/CD on Hugging Face’s Transformers library (and previously built CI at Mozilla), joins David Burns to talk about testing software where the maths changes every time you run it.
In this episode:
- Why green doesn’t mean working: non-determinism, GPU variance and out-of-memory leaks across 500+ model architectures
- Living with 0–500 daily failures out of 13,000 GPU tests, and triaging them with Grafana trends instead of by hand
- Serge, the overnight agent that reproduces failures, builds patches and opens PRs (170 runs, 19 PRs in two months)
- Catching reward hacking when the agent “fixes” a test by changing the assertion
- Cutting agent costs from $1.50 to under 50 cents a review by moving logic out of prompts and into deterministic Python
- A Ruff-style model linter with 58 rules, tiny overfitted models for CPU-only end-to-end checks, and the Model Integration Contract
- Sandboxing agents: separating instructions from data, Bubblewrap, network proxies and a human on the merge button
Plus the quickfire round: time-to-signal as the most underrated CI metric, why Tarek’s dream job is running coach, and whether AI will ever replace testing.
Follow Tarek’s writing on the Hugging Face blog.
Subscribe to BrowserStack Talks on Spotify, Apple Podcasts or YouTube so you don’t miss future episodes.
Bahasa Inggeris
Amerika Syarikat
TRANSKRIP 🔗
speaker-0: Every time you push something in main, we have 200,000 tests running. So you have this test, it's doing a bunch of assertion, and and the day after it started to fail for no reason, no code changed, nothing, nothing happened, and and then it starts to be red. Everyone experienced that. When you run LLM, you're gonna generate text at some point, and out of those 13,000 tests we do daily. we have between zero and five hundred reds. Which one are real problems?
speaker-1: said there that…
speaker-1: said there that…
CARI EPISOD LEPAS
Cari episod lepas bagi BrowserStack Talks.
Penafian: Podcast dan karya seni yang dibenamkan pada halaman ini adalah daripada BrowserStack, yang merupakan hak milik pemiliknya dan tidak bergabung dengan atau disokong oleh Listen Notes, Inc.
SUNTING
Terima kasih kerana membantu memastikan pangkalan data podcast dalam keadaan terkini.