이 에피소드에 관해
In this episode, James Bowler, Head of Research Partnerships at AE Studio, is joined by Erick Martinez, Alignment Researcher at AE Studio, to explore why pre-training is a largely neglected frontier in alignment research, and what it actually takes to do alignment work there.
Erick breaks down what pre-training is and why it matters: it accounts for roughly 99% of the compute that goes into a frontier model, producing the base LLM from which everything downstream (instruction tuning, RLHF, capability fine-tuning) inherits its character. The key implication for alignment is that the span of possible behaviors and personas a model can exhibit, including misaligned ones, is largely determined during pre-training. Post-training interventions can elicit these latent behaviors but cannot fully excise them. The emergent misalignment finding, where fine-tuning a model to write insecure code caused broadly adversarial behavior the model was never trained to produce, is a concrete example of a problem whose roots lie in the base model, not the fine-tuning step.
Erick draws on AE Studio's own gradient routing research, developed in collaboration with Anthropic and awarded a spotlight at ICML 2026, to illustrate just how technically demanding pre-training alignment work is. Gradient routing lets researchers control which model parameters get updated in response to which data labels, for instance isolating the parameters that encode dual-use biology knowledge so they can be localized or removed without degrading general capability. But making this work at scale requires operating well below the PyTorch abstraction layer: managing gradient graphs across distributed hardware, ensuring micro-batch accumulation stays mathematically equivalent to full-batch updates, and catching silent gradient leakage that can quietly corrupt an expensive training run before anyone notices.
The conversation closes on an under-appreciated asymmetry in alignment incentives: post-training papers are cheap and fast to produce because the base model is already baked, while pre-training experiments require significant compute, specialized infrastructure experience, and tolerance for expensive mistakes. That cost is also an opportunity. The field is sparse enough that well-resourced researchers willing to work at this layer can find genuinely novel ground.
In this episode:
*Why pre-training sets the "span of possible personas" a model can exhibit, including misaligned ones
*How the emergent misalignment result points to pre-training as the root cause of certain alignment failures
*What gradient routing is and how it lets researchers isolate dual-use knowledge during pre-training
*The Chinchilla scaling law and what a realistic pre-training data corpus looks like (50M to 5B parameter runs, up to ~200 GB of tokens)
*Why distributing training across GPUs creates subtle gradient accumulation bugs that can silently degrade alignment interventions
*Why alignment incentives currently push researchers toward post-training and what makes the pre-training layer a fertile, underexplored area
Learn more: https://www.ae.studio/alignment
AE Studio is hiring: https://www.ae.studio/join-us
Subscribe to our newsletter: https://aestudio.beehiiv.com/
James Bowler LinkedIn: https://www.linkedin.com/in/james-bowler-84b02a100/
Erick Martinez LinkedIn: https://www.linkedin.com/in/erick-f-martinez/
Contact us: alignment@ae.studio
