Did OpenAI’s New Model “Go Rogue”? - Cal Newport

Skip to content

Home » Blog » Did OpenAI’s New Model “Go Rogue”?

A couple of weeks ago, the AI company Hugging Face ​announced​ that they had discovered an intrusion into their production infrastructure. They didn’t know the source, but noted that large language models appeared to be involved. The following week, OpenAI ​admitted​ that the breach was the result of an AI system test that went awry.

The initial news coverage created the sense that something unnerving had just occurred:

​The Wall Street Journal ​ called it “the stuff of cybersecurity nightmares."

​The Hill ​ said, “Washington and the technology industry are on high alert this week after OpenAI revealed that some of its AI agents went rogue.”

​The AP ​ quipped that “to be fair, James Cameron did warn us” (a reference to The Terminator), before describing the event as a “told-you-so moment for researchers who had warned for years that the technology could pose an existential threat to humanity.”

Yikes! It’s perhaps not surprising, then, that I’ve received more emails about this incident than any other recent AI story I can remember.

So, what really happened here?

OpenAI was testing its new models on an evaluation framework called ​ExploitGym​ – a collection of 869 cybersecurity scenarios, most of which pair a specific system with a hacking challenge, such as breaking in to gain access to a protected file. They also usually include a suggestion of a vulnerability to exploit in solving the challenge.

A large language model on its own, of course, cannot break into anything: all it does is generate reasonable next tokens in response to input prompts. To use ExploitGym, you need a control program called a harness that provides access to many different software development tools useful for hacking into systems. The harness can repeatedly prompt an LLM to help come up with an attack plan, then ask it to help implement specific steps – for example, if the harness needs code to exploit a bug, it can ask the LLM to write it.

These capabilities, as it turns out, already exist in the coding harnesses that the major AI companies have been focusing on relentlessly in recent years as computer programming emerged as one of the first major markets for LLM-based tools. To compete in the ExploitGym, therefore, it’s sufficient to combine a version of an LLM missing the standard anti-hacking guardrails with a cutting-edge coding harness tweaked and optimized for these types of challenges.

Put more simply: ExploitGym is meant to test the capability of cutting-edge AI coding tools for attacking systems instead of creating them.

This brings us back to the Hugging Face breach…

Earlier this month, OpenAI tasked an unspecified coding harness, combined with a pre-release version of a new LLM, to complete an ExploitGym challenge. When prompted, the model – as LLMs so often do – came up with a quirky (but rational) plan to achieve the provided goal: break into a server at Hugging Face that stores the solutions to ExploitGym challenges.

(This behavior, in which the LLM ignores the suggested approach to come up with a different attack plan, is something that the creators of ExploitGym ​describe as common​: “Across models, agents frequently achieved code execution through a vulnerability other than the one we provided." Notice, using the suggested attack is almost always the right thing to do, so this is more a sign of the unpredictability of LLM’s rather than some rogue intelligence.)

The harness then dutifully attempted to execute the Hugging Face plan: first finding a way to gain unrestricted internet access (by default, systems competing in ExploitGym challenges run in a constrained network environment), then chaining together various security exploits to gain access to a Hugging Face server. That’s when it was detected.

Two key points about this incident…

First , circumventing internet restrictions and hacking into servers are exactly the kinds of things these ExploitGym systems are designed to do. There was no “rogue” agent or revelation of some surprising, devious new capability.

Second , the real issue here was OpenAI’s sloppiness. What makes ExploitGym a hard benchmark is that there aren’t supposed to be humans in the loop–you have to let your harness and LLM act entirely on their own, coming up with long-time-horizon plans and executing them autonomously. (When professional programmers use coding harnesses, by contrast, there’s plenty of human oversight, as LLM-based plans are often misaligned with our intentions, or just plain weird, and need correcting.)

AI companies know LLM-based plans are pretty unpredictable, so they’re usually pretty careful about how they set up and restrict the harnesses and LLMs used in ExploitGym-style tests.

According to ​reporting​ from the Financial Times, however, OpenAI recently started playing fast and loose with these safety principles in a bid to catch up to...