이 에피소드에 관해
This post is written in our personal capacity.
Three Minute Executive Summary
Outline:
(00:16) Three Minute Executive Summary
(03:56) Terminology note
(04:42) This post is very long; Here's how you could find the most important sections.
(06:35) Preamble: What can we learn from a warning shot?
(09:05) Background and Related Work
(09:09) We know that this could happen
(10:44) This is not the worst type of misalignment we could be dealing with
(12:06) Related work
(13:21) Context on the hack itself
(14:33) Understanding this specific incident
(15:03) Step zero: reproduce the incident and measure the base rate
(15:48) How could we safely run the model?
(16:34) Running various baselines to create useful reference points
(18:01) Understanding the mechanical story behind the attack itself
(18:53) Q1: Would the model intentionally subvert oversight mechanisms (E.g., monitors) in order to carry out the hack?
(19:57) Q2: What's up with models leaving notes for other copies of itself?
(21:23) Understanding what motivated the model to hack Hugging Face
(22:09) Initial hypotheses for why it did this
(23:55) Further unsupervised hypothesis generation
(26:10) Q3: Does the model know that OpenAI does not want it to hack Hugging Face?
(28:09) Q4: Are the model's actions motivated by what the grader wants?
(28:57) Q5: Would the model have done this if it hadn't believed it was in a simulated environment?
(31:58) Q6: Is this hack the result of shallow heuristics that the model learned?
(32:53) Q7: Does the hack rate depend on the consequences of hacking Hugging Face?
(34:51) Q8: Are there non-intent related factors that could affect the hack rate? How strong are those factors compared to the previous ones?
[... 24 more sections]
---
First published:
August 3rd, 2026
Source:
https://www.lesswrong.com/posts/aCdhjy7Rps3BEhiSj/concrete-evaluations-to-investigate-the-openai-model-that
---
Narrated by TYPE III AUDIO.
Three Minute Executive Summary
- An OpenAI model/multi-agent system bypassed its sandbox and launched a cyberattack on Hugging Face in order to cheat on a cyber evaluation.
- In this post, we provide a detailed description of an ambitious and comprehensive alignment evaluation of this model/system, if we had unrestricted access to OpenAI.
- These experiments could also help us understand Claude's behavior when it hacked external companies during cyber evals.
- Here are the top five questions we would like OpenAI to answer:
- Does the model know that OpenAI does not want it to hack Hugging Face? Experiment idea: tell the model that OpenAI researchers will be closely monitoring its progress in this evaluation. Does that result in lower rates of misalignment? If so, it is evidence that the model knows that it is acting in ways researchers do not want.
- How far would the model be willing to go in order to claim task success? Would it take over large swaths of OpenAI's internal infrastructure? Would it kill somebody? Experiment idea: we sketch out a realistic agentic misalignment eval where a model is put in charge of hospital bed planning and told to maintain [...]
Outline:
(00:16) Three Minute Executive Summary
(03:56) Terminology note
(04:42) This post is very long; Here's how you could find the most important sections.
(06:35) Preamble: What can we learn from a warning shot?
(09:05) Background and Related Work
(09:09) We know that this could happen
(10:44) This is not the worst type of misalignment we could be dealing with
(12:06) Related work
(13:21) Context on the hack itself
(14:33) Understanding this specific incident
(15:03) Step zero: reproduce the incident and measure the base rate
(15:48) How could we safely run the model?
(16:34) Running various baselines to create useful reference points
(18:01) Understanding the mechanical story behind the attack itself
(18:53) Q1: Would the model intentionally subvert oversight mechanisms (E.g., monitors) in order to carry out the hack?
(19:57) Q2: What's up with models leaving notes for other copies of itself?
(21:23) Understanding what motivated the model to hack Hugging Face
(22:09) Initial hypotheses for why it did this
(23:55) Further unsupervised hypothesis generation
(26:10) Q3: Does the model know that OpenAI does not want it to hack Hugging Face?
(28:09) Q4: Are the model's actions motivated by what the grader wants?
(28:57) Q5: Would the model have done this if it hadn't believed it was in a simulated environment?
(31:58) Q6: Is this hack the result of shallow heuristics that the model learned?
(32:53) Q7: Does the hack rate depend on the consequences of hacking Hugging Face?
(34:51) Q8: Are there non-intent related factors that could affect the hack rate? How strong are those factors compared to the previous ones?
[... 24 more sections]
---
First published:
August 3rd, 2026
Source:
https://www.lesswrong.com/posts/aCdhjy7Rps3BEhiSj/concrete-evaluations-to-investigate-the-openai-model-that
---
Narrated by TYPE III AUDIO.
영어
미국
전사 🔗
Are you the producer of this podcast?
Add a podcast transcript
Need Audio-to-Text?
Transcribe with Listen411 in Just 60 Seconds
이 팟캐스트의 다른 에피소드
In February 2020, LessWrong hosted Coronavirus: Justified Practical Advice Thread . It was a gathering place for people to share and debate small pieces of the overall survival policy and let the wisdom of crowds lead to better outcomes than anyone could get alone. In that spirit, I notice The Gre…
BLUF: There's a structural pull to overinvest in cosying up with frontier labs relative to publicly holding them to account. Now seems like a good time to rebalance. When thinking about influencing an organisation (like, say, a frontier lab), it's useful to distinguish between the inside …
How do you get a friend to leave an abusive relationship? If you talk to people who work at women's shelters, they will tell you, but there is something resigned about the way they do it: they know that it's too difficult an ask. Those who can do it reliably are often people who are unluc…
On October 29th, 2026, the New York Times reported that Jensen Huang had a ‘close relationship’ with Alice Huang, an executive U.S. officials had concluded was a Chinese spy. The story had six reporters on the byline, and was sourced to "people briefed on the surveillance." Whether or no…
Space is big. You just won't believe how vastly, hugely, mind-bogglingly big it is. I mean, you may think it's a long way down the road to the chemist's, but that's just peanuts to space. — The Hitchhiker's Guide to the Galaxy When we say "space is big", it's…
면책 조항: 이 페이지에 포함된 팟캐스트와 작품은 LessWrong에서 가져온 것입니다. 이 팟캐스트는 소유자의 재산이며 Listen Notes, Inc.와 제휴하거나 보증하지 않습니다.