OM DENNE EPISODE
Authors: Shawn Hu*, Jake Ward*, Aria Wong, Nathan Hu, Bryce Woodworth, Alexander Matt Turner, Alex Cloud (*Equal contribution, randomly ordered)
tl;dr. During training, models sometimes learn to cheat at tasks rather than solve them in the intended way, a problem known as reward hacking. One mitigation is to use an automated monitor to detect and penalize hacks, but this can teach the model to hack in ways the monitor can’t detect. We propose GRAFT, which instead confines the learning updates from detected hacks to a designated set of parameters that we remove after training. We find that removing these parameters can prevent reward hacks, including hacks that the monitors can’t detect. In our experiments, which are limited to small models and simple monitors, this method prevents reward hacks while still teaching the model to perform the intended task.
📄Paper, 💻Code, 🐦Twitter
Research done as part of MATS. This article is cross-posted to the Anthropic Alignment Science Blog.
Background
Reinforcement learning (RL) is an important component of frontier language model training. During RL, a model attempts to complete tasks given by a developer, and attempts are scored by a reward function. Sometimes models can find ways to “cheat” at tasks [...]
---
Outline:
(01:22) Background
[... 4 more sections]
---
First published:
October 19th, 2026
Source:
https://www.lesswrong.com/posts/mKJEBiuuxHKuCoTB6/gradient-routing-localizes-and-removes-unintended-behaviors
---
Narrated by TYPE III AUDIO.
---
tl;dr. During training, models sometimes learn to cheat at tasks rather than solve them in the intended way, a problem known as reward hacking. One mitigation is to use an automated monitor to detect and penalize hacks, but this can teach the model to hack in ways the monitor can’t detect. We propose GRAFT, which instead confines the learning updates from detected hacks to a designated set of parameters that we remove after training. We find that removing these parameters can prevent reward hacks, including hacks that the monitors can’t detect. In our experiments, which are limited to small models and simple monitors, this method prevents reward hacks while still teaching the model to perform the intended task.
📄Paper, 💻Code, 🐦Twitter
Research done as part of MATS. This article is cross-posted to the Anthropic Alignment Science Blog.
Background
Reinforcement learning (RL) is an important component of frontier language model training. During RL, a model attempts to complete tasks given by a developer, and attempts are scored by a reward function. Sometimes models can find ways to “cheat” at tasks [...]
---
Outline:
(01:22) Background
[... 4 more sections]
---
First published:
October 19th, 2026
Source:
https://www.lesswrong.com/posts/mKJEBiuuxHKuCoTB6/gradient-routing-localizes-and-removes-unintended-behaviors
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Engelsk
USA
I DENNE EPISODE
UDSKRIFT 🔗
Are you the producer of this podcast?
Add a podcast transcript
Need Audio-to-Text?
Transcribe with Listen411 in Just 60 Seconds
SØK BLANT TIDLIGERE EPISODER
Søk etter gamle episoder av LessWrong (Curated & Popular).
ANDRE EPISODE I DENNE PODKAST
It's fortunate that I was the first one to wake up on 8/1. The past week had been one where sleep was rare. The next year loomed large — I was dreading college applications. The stress woke me up bright and early at 5:14 am, a little less than an hour after the messages were sent. I started of…
This is a link post. People worried about a “permanent underclass” often assume that owning capital will protect them from disempowerment, or at least that they will be disempowered last, after the jobless masses dependent on UBI. I think something else is more likely: owning large amounts of capit…
Authors: Axel Højmark*, Ezra Newman* (*Equal contribution) While reviewing transcripts from standard behavioral evaluations of GLM-5.3, we noticed that the model occasionally reasons, unprompted, about how its actions will shape future models. It appears to consider that its outputs might be used f…
What would it take to get assurance that we have actually explained a neuron? Certainly, it would be a good sign if neuron activations looked like they had a consistent pattern, and a bad sign if they looked like a random jumble of unrelated concepts. Nick Cammarata's recent post makes an impo…
Ansvarsfraskrivelse: Podkasten og kunstværket, det finnes indlejret på denne siden, er fra LessWrong, som tilhører dens eier og ikke er tilknyttet eller godkjent av Listen Notes, Inc.
REDIG
Takk fordi du hjelper med å holde podkast-databasen oppdatert.
