The OpenAI and Hugging Face incident, explained in simple terms · Ashu's blog

Skip to content

I do not work in security. So this is not a forensic report. It is a simplified explanation of the incident, aimed at engineers who want to understand what happened and why it was possible.

In brief, the agents were given a problem statement which was rather impossible to solve. Being persistent the agents ended up figuring out vulnerabilities in the systems they were running on, and used those to reach the internet and access data from Hugging Face in hope of finding the solution to the original problem.

The setup, and the task#

The agents were being trained on hacking practice problems.

Each run got its own sandbox. A fresh virtual machine, thrown away at the end, with no access to the internet.

But there was a catch, like I said, the task itself was the problem. The task was impossible to crack. Instead of just giving up the agents explored for another way to obtain the answers on the internet (Maybe the answers exist on Hugging Face).

In that sandbox there was almost nothing to explore, except one thing. The artficat registry.

What is an artifact registry?#

An artifact registry is the company's own copy of npm, PyPI or Docker Hub, kept inside the network.

It has two jobs:

Store what your builds produce. Every build pushes its output here.

Serve the packages your builds need. Instead of every machine downloading from the public internet, they all download from the registry.

To serve a public package, the registry must first fetch it from upstream. So the registry is usually allowed to reach the internet even when nothing else is. And since every build depends on it, almost everything is allowed to talk to the registry.

Exploiting the artifact registry#

The agents used the artifact registry in three ways:

To reach the internet. The registry could fetch remote content, so it could be asked to fetch things on the agents' behalf.

To act as a message board. Files written by one agent could be read by another agent.

To run code on a server. The agents found a bug in the registry that let them run commands on the machine it was running on.

Using the registry as a message board#

This meant

Different runs could explore different methods to crack the problem at the same time.

A later run could build on an earlier run's findings instead of starting over.

A newer, stronger model could read notes left by an older, weaker one.

Using the registry to reach the internet#

The agent containers had no internet. The registry did, because it has to fetch packages from upstream.

So the agents reportedly used an SSRF weakness. SSRF means you cannot reach a place yourself, so you ask a server that can reach it to go and bring back the answer for you.

In code, the bug looks like this:

buggy-fetch.jsCopy// Buggy: the user picks the destination<br>app.get("/fetch", async (req, res) => {<br>const response = await fetch(req.query.url);<br>res.send(await response.text());<br>});<br>Using the registry to run code on a server#

The agents then exploited an Artifactory vulnerability, and later another chain, to run commands on a server.

Remote code execution means you can make a remote machine run commands you choose. Once that is possible, you can read the files on that machine, read its environment variables, and use whatever cloud identity it has been given.

The classic version of this bug is joining user input into a shell command:

buggy-lookup.jsCopyimport { exec } from "node:child_process";

// Buggy: user input is concatenated into a shell string<br>app.get("/lookup", (req, res) => {<br>exec("nslookup " + req.query.hostname, (error, output) => {<br>if (error) return res.status(500).send("Failed");<br>res.type("text/plain").send(output);<br>});<br>});<br>exec hands the whole string to a shell. A hostname like example.com; cat /etc/passwd is now two commands, and the shell will happily run both.

The first patch and the second exploit#

The agent runs were hitting Artifactory hard enough which caused the registry to go down. Once they Open AI engineers understood it, they revoked the credentials the agents had been using, deleted the message files, and patched the weakness the agents had exploited.

Then the agents found another way to leave notes for each other, reportedly through a WebDAV endpoint that had no password on it.

WebDAV is an old addition to HTTP that makes a web server behave like a shared drive. A normal web server only lets you download things. WebDAV also lets you upload a file, and see what is inside a folder.

webdav.shCopy# write a file<br>curl -X PUT --data "a note for the next run" \<br>http://internal-host/dav/notes/hello.txt

# read it back<br>curl http://internal-host/dav/notes/hello.txt<br>With no password on it, one run could write and another run could read. That is a message board again.

Getting access to whole clusters of machines#

Earlier we saw how the agents got code running on a machine. That is what makes this step...