이 에피소드에 관해
TL;DR: We train LLMs to accept LLM neural activations as inputs and answer arbitrary questions about them in natural language. These Activation Oracles generalize far beyond their training distribution, for example uncovering misalignment or secret knowledge introduced via fine-tuning. Activation Oracles can be improved simply by scaling training data quantity and diversity.
The below is a reproduction of our X thread on this paper and the Anthropic Alignment blog post.
Thread
New paper:
We train Activation Oracles: LLMs that decode their own neural activations and answer questions about them in natural language.
We find surprising generalization. For instance, our AOs uncover misaligned goals in fine-tuned models, without training to do so.
We aim to make a general-purpose LLM for explaining activations by:
1. Training on a diverse set of tasks
2. Evaluating on tasks very different from training
This extends prior work (LatentQA) that studied activation verbalization in narrow settings.
Our main evaluations are downstream auditing tasks. The goal is to uncover information about a model's knowledge or tendencies.
Applying Activation Oracles is easy. Choose the activation (or set of activations) you want to interpret and ask any question you like!
We [...]
---
Outline:
(00:46) Thread
(04:49) Blog post
(05:27) Introduction
(07:29) Method
(10:15) Activation Oracles generalize to downstream auditing tasks
(13:47) How does Activation Oracle training scale?
(15:01) How do Activation Oracles relate to mechanistic approaches to interpretability?
(19:31) Conclusion
The original text contained 3 footnotes which were omitted from this narration.
---
First published:
December 18th, 2025
Source:
https://www.lesswrong.com/posts/rwoEz3bA9ekxkabc7/activation-oracles-training-and-evaluating-llms-as-general
---
Narrated by TYPE III AUDIO.
---
The below is a reproduction of our X thread on this paper and the Anthropic Alignment blog post.
Thread
New paper:
We train Activation Oracles: LLMs that decode their own neural activations and answer questions about them in natural language.
We find surprising generalization. For instance, our AOs uncover misaligned goals in fine-tuned models, without training to do so.
We aim to make a general-purpose LLM for explaining activations by:
1. Training on a diverse set of tasks
2. Evaluating on tasks very different from training
This extends prior work (LatentQA) that studied activation verbalization in narrow settings.
Our main evaluations are downstream auditing tasks. The goal is to uncover information about a model's knowledge or tendencies.
Applying Activation Oracles is easy. Choose the activation (or set of activations) you want to interpret and ask any question you like!
We [...]
---
Outline:
(00:46) Thread
(04:49) Blog post
(05:27) Introduction
(07:29) Method
(10:15) Activation Oracles generalize to downstream auditing tasks
(13:47) How does Activation Oracle training scale?
(15:01) How do Activation Oracles relate to mechanistic approaches to interpretability?
(19:31) Conclusion
The original text contained 3 footnotes which were omitted from this narration.
---
First published:
December 18th, 2025
Source:
https://www.lesswrong.com/posts/rwoEz3bA9ekxkabc7/activation-oracles-training-and-evaluating-llms-as-general
---
Narrated by TYPE III AUDIO.
---
영어
미국
이 에피소드 내
전사 🔗
Are you the producer of this podcast?
Add a podcast transcript
Need Audio-to-Text?
Transcribe with Listen411 in Just 60 Seconds
이 팟캐스트의 다른 에피소드
By now, there are lots of AI safety research papers, more than what is reasonable for a single person to read. This will get worse as the machines get better at AI safety research, and as more talent and resources pour into the field. But the "paper" is an arbitrary unit, we really care a…
This is a link post. Executive summary From August 12 to October 29, 2026, METR staff performed a high-access evaluation at Anthropic to assess the degree to which AI is accelerating AI R&D.[2] If AI systems started to significantly accelerate further AI development, this could lead to a sudden…
For all my efforts at deliberate practicing rationality, by far the most actionable way to self-improve is "get on ADHD or depression meds, if you need them." Your friends with ADHD and depression struggle to summon the motivation and focus necessary to get themselves a teledoc who can gi…
I've done AI safety grantmaking for the Survival and Flourishing Fund and Lightcone Commons. It's hard. There's an endless amount to know, from which hardware verification mechanisms will survive a dedicated adversary, all the way to which founders have skeletons in their closets tha…
Rejected for the following reason(s):This is an automated rejection. No LLM generated, assisted/co-written, or edited work.Read full explanationThis is a link post. Twitter thread: https://x.com/Marcus_J_W/status/2118093146483810386 or https://twitter-thread.com/t/2118093146483810386 Blogpost: Summ…
면책 조항: 이 페이지에 포함된 팟캐스트와 작품은 LessWrong에서 가져온 것입니다. 이 팟캐스트는 소유자의 재산이며 Listen Notes, Inc.와 제휴하거나 보증하지 않습니다.
