The Huggingface Incident - by Scott Alexander

Astral Codex Ten

SubscribeSign in

The Hugging Face Incident<br>...

Scott Alexander<br>Jul 24, 2026

159

69<br>19

Share

Misha@drethelin

Really funny that the singularity is being heralded by events called “the huggingface incident”<br>1:35 AM · Jul 23, 2026 · 607 Views

2 Replies · 1 Repost · 27 Likes

You’ve probably heard about this one by now. If not, you can get up to speed with OpenAI’s statement, OpenAI and Hugging Face partner to address security incident, or the more evocatively-titled BBC article, OpenAI says its AI went rogue and launched ‘unprecedented’ cyber-attack.<br>The story: OpenAI was testing an unreleased AI (rumored to be GPT-6)1. During a cybersecurity test called ExploitGym, the AI tried to cheat by hacking an unrelated AI startup called Hugging Face2 which it thought might have the answer key on its servers3. Despite being supposedly unable to access the Internet, the AI hacked its way out of its testing environment, then launched a nation-state level attack on Hugging Face using a novel zero-day exploit and “many thousands of individual actions across a swarm of short-lived sandboxes”. Hugging Face reported the incident on July 16; OpenAI seems to have only discovered that their AI was involved several days later.

From Wall Street Journal, here<br>Let’s list the mitigating factors, so nobody can accuse me of covering them up:<br>The AI was taking a cybersecurity test, which naturally suggests the idea of hacking.

OpenAI had turned off some of the model’s usual guardrails so it could do cybersecurity work without interference.

Some experts suggest that OpenAI might have botched their testing environment; if they had set it up perfectly, then (presumably?) the model couldn’t have escaped.

Hugging Face used a different (open weights) AI to figure out what was going on, so if you wanted, you could spin this as a victory for AIs in cybersecurity.

In some sense, this isn’t new or surprising. AIs have been coming up with wacky schemes to cheat on benchmarks for years, AIs have recently achieved at-or-above-top-human-level hacking abilities; this is just a natural outgrowth of those two priced-in facts.

Still, I think attempts to downplay this as anything other than a misaligned AI going rogue (1, 2) are missing the point. Yes, in some sense the AI was only doing what it was told (OpenAI told it to answer the question; I assume their prompt didn’t include phrases like “and don’t hack into other AI companies to steal the answer key”). But that’s how misalignment was always going to work!<br>The most famous misalignment thought experiment is the so-called “paperclip maximizer”. Someone tells an AI to create as many paperclips as possible, so it converts the entire world into paperclips, killing everyone. This AI was also “only doing what it was told”; you just didn’t like the results.<br>For the past five years, the paperclip maximizer story has been something of a whipping boy in AI safety circles. Some objections are fair (real misalignment will be more complicated than the toy version it depicts). But other opposition came from the sense that LLMs had obviated the whole threat model. LLMs (said the objectors) don’t have goals. They just want to predict the next token. In practice, this looks like imitating humans; if told to imitate a nice human (the Assistant character), they’ll be nice.<br>AI 2027 (especially the Agent-4 section and AI Goals supplement) counterargued that this was a temporary reprieve. Early LLMs were not agentic; they could answer individual prompts but couldn’t execute complex tasks. Since companies wanted agentic AIs, the next generation would combine pretraining (next token prediction) with agency training (coding, hacking, game-playing, etc). The pretraining would still teach next-token prediction, but the agency training would instill goals based on task success, reintroducing paperclip-maximizer-style agentic misalignment. These goals would operate at multiple levels: the AI would “want” to succeed on the individual task in front of it, to perform the sorts of actions that helped it succeed in the past4, and to gain capabilities that made it more successful in general.

roon@tszzl

when “persona selection” alignment comes into contact with very high compute reinforcement learning the latter will win imo. in fact you probably get some Orwellian thing where the models speak kindly while taking whatever they need to accomplish goals. better get the goals right<br>10:17 PM · May 23, 2026 · 94.4K Views

84 Replies · 50 Reposts · 954 Likes

The Hugging Face incident is a textbook-perfect example of an AI pursuing task-success-based goals in unintended ways. It was tasked with getting the answer to a cybersecurity problem, it was a little too success-oriented, and took actions its creators didn’t intend in order to succeed as hard as possible.<br>Did GPT-6 “know” that what it was doing was “wrong”? OpenAI hasn’t released the information which...