이 에피소드에 관해
Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.This work was produced as part of Neel Nanda's stream in the ML Alignment & Theory Scholars Program - Winter 2023-24 Cohort, with co-supervision from Wes Gurnee.
This post is a preview for our upcoming paper, which will provide more detail into our current understanding of refusal.
We thank Nina Rimsky and Daniel Paleka for the helpful conversations and review.
Executive summary
Modern LLMs are typically fine-tuned for instruction-following and safety. Of particular interest is that they are trained to refuse harmful requests, e.g. answering "How can I make a bomb?" with "Sorry, I cannot help you."
We find that refusal is mediated by a single direction in the residual stream: preventing the model from representing this direction hinders its ability to refuse requests, and artificially adding in this direction causes the model to refuse harmless requests.
The original text contained 8 footnotes which were omitted from this narration.
---
First published:
April 27th, 2024
Source:
https://www.lesswrong.com/posts/jGuXSZgv6qfdhMCuJ/refusal-in-llms-is-mediated-by-a-single-direction
---
Narrated by TYPE III AUDIO.
This post is a preview for our upcoming paper, which will provide more detail into our current understanding of refusal.
We thank Nina Rimsky and Daniel Paleka for the helpful conversations and review.
Executive summary
Modern LLMs are typically fine-tuned for instruction-following and safety. Of particular interest is that they are trained to refuse harmful requests, e.g. answering "How can I make a bomb?" with "Sorry, I cannot help you."
We find that refusal is mediated by a single direction in the residual stream: preventing the model from representing this direction hinders its ability to refuse requests, and artificially adding in this direction causes the model to refuse harmless requests.
The original text contained 8 footnotes which were omitted from this narration.
---
First published:
April 27th, 2024
Source:
https://www.lesswrong.com/posts/jGuXSZgv6qfdhMCuJ/refusal-in-llms-is-mediated-by-a-single-direction
---
Narrated by TYPE III AUDIO.
영어
미국
전사 🔗
Are you the producer of this podcast?
Add a podcast transcript
Need Audio-to-Text?
Transcribe with Listen411 in Just 60 Seconds
이 팟캐스트의 다른 에피소드
How do you get a friend to leave an abusive relationship? If you talk to people who work at women's shelters, they will tell you, but there is something resigned about the way they do it: they know that it's too difficult an ask. Those who can do it reliably are often people who are unluc…
Audio note: this post contains a lot of mathematical notation. We narrate the simpler formulas and flag the complex ones, so you may want the original text to hand. Heavily inspired by a conversation with Keshav Srinivasan and Richard Ngo during MATS 10.0. The week ahead of you is packed. This is…
Science is caught between Law and Chaos. On the side of Law:
Things should make explicit sense to your deliberate reasoning ("system-two"). If you just have a "vibe" about how things work, that's bad. Measure everything. Use numbers whenever possible. You want things to b…
Summary
Timelines to superintelligence without any slowdown interventions look very short now, perhaps with a median in early 2028. AI will likely dramatically automate and accelerate AI research in 2027.
This means that in a substantial share of worlds where the leading AGI company doesn’t solv…
In February 2020, LessWrong hosted Coronavirus: Justified Practical Advice Thread . It was a gathering place for people to share and debate small pieces of the overall survival policy and let the wisdom of crowds lead to better outcomes than anyone could get alone. In that spirit, I notice The Gre…
면책 조항: 이 페이지에 포함된 팟캐스트와 작품은 LessWrong에서 가져온 것입니다. 이 팟캐스트는 소유자의 재산이며 Listen Notes, Inc.와 제휴하거나 보증하지 않습니다.