The first known runaway AI agent - or a very bad marketing stunt? - Martin Alderson

In the past few days Hugging Face announced a security incident, which transpired to be a "runaway" agent from OpenAI.

This is likely the first known autonomous offensive agent working like this, and certainly the first known one where an agent has done this inadvertently.

Just marketing?

Reading the commentary on this, it seems the majority of people think this is a marketing stunt. While certainly the frontier labs have endless examples of claiming things are not safe, it's quite a dangerous position for technologists to take if it is not a marketing stunt.

I really struggle to see how this could be a marketing stunt. Huggingface released the blog on the 16th of July, 5 days before OpenAI released their announcement. Furthermore, Huggingface didn't name OpenAI then. It seems like a genuine security incident report.

Equally, I'm not sure what OpenAI has to gain from this. Perhaps again it's some incredible campaign to prove open weights models are unsafe (in which case, why did Hugging Face say that open weights models were essential to detecting and understanding the issue, if it was a coordinated PR stunt?). And given all the wider media and political attention on cybersecurity safety, I don't think you could come up with a worse headline than "dangerous AI escapes lab" plastered all over the front page of the global media ecosystem.

Now, I'm not convinced that frontier labs have nailed public relations generally, but if it was a marketing stunt it is potentially one of the worst ones I've ever came across in following corporate communications. But perhaps the frontier labs really have their backs against the wall with the upcoming release of Kimi K3 et al.

Regardless, I think it's worth breaking down what happened and properly evaluate each part.

The escape

To understand what went wrong here we first need to understand how models are developed and tested. OpenAI says that they were running benchmarks on GPT5.6 Sol and an unreleased model (almost certainly GPT6), running the ExploitGym benchmark. Importantly - the usual safety classifiers were disabled, no doubt so OpenAI could understand just how good they were at offensive cyber capabilities.<br>It's also likely they were running a huge amount of benchmarks simultaneously with ~unlimited token budgets - you want as many samples as possible to figure out how good a model is at a certain benchmark. It may also be they are testing various different checkpoints of the model too, understanding how the model is improving as it goes through the various training stages.

Now we've set the scene for why they were doing this, let's look at the first part, the sandbox escape.

I've written before about how essential sandboxes are to modern cybersecurity, and while this wasn't a direct sandbox escape, it shows the lateral "thinking" of models being able to get round guardrails.

Essentially, the agent under test running these benchmarks was running in a contained environment without internet access, to avoid this situation exactly happening. However, they did expose a proxy to allow the agent to download and install various software packages - no doubt to improve the effectiveness of the model (cybersecurity touches a lot of different programming environments and having the right tools makes it much easier for the agent to work effectively with it).

So we get to the first exploit. It's claimed the model managed to get public internet access by exploiting this proxy - which was meant to just allow the model to download and install software packages - to instead give it wide access to the internet.[1]

This really doesn't surprise me. These kind of package proxies are not designed for this use case. They're designed for engineers to only have access to certain software packages - typically in enterprise teams where you want a separate person approving each software package to avoid random software packages being pulled in (for compliance or licensing reasons).

Importantly, most of the package proxies I looked at allow this by design. Sonatype's Nexus and JFrog's Artifactory - the two that dominate the enterprise market - will both quite happily proxy arbitrary websites through their "raw" and "remote" repository types, and Go's own module proxy will fetch from any host you name. There's also been an awful lot of SSRF CVEs in these things over the years. OpenAI's proxy was presumably locked down to just the package registries it needed, so getting out of it did take an actual bug - but given how permissive these tools are by default, a zero day here reads less like a novel AI breakthrough and more like a classic "tool being considered more secure than it is".

The exploit

Once the agent has figured a way to get internet access, it's well known that models routinely cheat on benchmarks. There's been so many stories of models figuring out ways to game the benchmarks. This...