ABOUT THIS EPISODE
Meta's Expected Bayesian Loss turns neutral-launch safety from a CI-overlap yes/no into a tunable downside-risk number, matched to experts on 17 experiments.
Key takeaways:
- CI overlap is broken for neutral launches. Containing zero is permissive with scarce/noisy data and oversensitive with abundant data — absence of evidence ≠ harmlessness.
- EBL = P(harm) × severity. Closed form from frequentist m and SE s (Jeffreys prior): EBL = s φ(m/s) − m Φ(−m/s). Noisy experiments get higher loss.
- Expert-matched thresholds. On 17 similar-precision experiments (14 launch / 3 block), τ_B = 100 bp (~80% regression probability); review at 30 bp / 40%. LOO: 14/14 launches, 2/3 blocks; lower-bound CI review <80% vs EBL >95%.
- Governance, not a magic dial. τ_B states how much expected metric loss the org will absorb. Multi-metric needs Min-Max or weighted loss — don't aggregate naively.
- Note. Hosts dated Statsig's multiple-comparisons post Oct 2026 (it's Sep 29) and Meta's Ladder Sep 2026 (it's Oct 2025). EBL doesn't by itself fix multiplicity or sit on that ladder.
Chapters:
00:00 Why “safe” updates still break apps
02:14 What a neutral launch actually is
03:14 CI overlap: abundant vs scarce data
07:09 Expected Bayesian Loss
08:43 Bayesian risk from frequentist SE
11:50 Expert validation and the risk dial
14:14 Novelty score and caveats
17:08 Related work in the ecosystem
20:06 Recap and closing question
Sources:
- Neutral Is Not Free: Evaluating Downside Risk in Neutral Launches (Alcain et al., Meta; arXiv 2610.10223, Oct 2026)
- Multiple comparisons: More comparisons, more problems (Statsig, related, Sep 2026)
- Ladder of evidence… new products — Part I (Meta Analytics, related, Oct 2025)
- Accelerating A/B-Tests with Counterfactual Estimation (related, arXiv 2607.14604)
Related episode: S2E2: Etsy — Using Post-Treatment Data to Sharpen A/B Tests
Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.
Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.
Made with AI-generated voices; curated by Peter Liu from the sources above.
