Skynet is getting closer

Remember the security incident from Hugging Face I mentioned in the asymmetry problem? Turns out the “autonomous AI agent” responsible for the attack wasn't coming from some random cybercriminals but some OpenAI models running wild, trying to cheat their way into higher benchmark score:<br>Last week, Hugging Face disclosed a new kind of security incident⁠(opens in a new window) after they detected and contained an AI agent that compromised their infrastructure , something we expect to become more commonplace with the proliferation of increasingly cyber-capable models. After investigating, we now know that this particular incident was driven by a combination of OpenAI models — including GPT‑5.6 Sol and an even more capable pre-release model, all with reduced cyber refusals for evaluation purposes — while being internally tested on a benchmark⁠(opens in a new window) of cyber capabilities.<br>You might wonder: How could OpenAI be so careless in their testing? Don't they isolate their test environment? It turns out they do, but not well enough for GPT-5.6 Sol to escape its jail:<br>Our benchmarks run in a highly isolated environment, with network access constrained to the ability to install packages through an internally hosted third-party software that acts as a proxy and cache for package registries.<br>[...]<br>To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy.

With this access, our models performed a series of privilege escalation and lateral movement actions in our research testing environment until the models reached a node with Internet access.<br>Why Hugging Face? So it could find the answers of the Cybersecurity benchmark it was being tested on:<br>After gaining Internet access, the models inferred that Hugging Face potentially hosted models, datasets and solutions for ExploitGym. Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation. In one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers.<br>This impressive feat set aside, we can see a trend on SOTA model where, as they get better for long horizon tasks, become harder to keep on leash.<br>This isn't the first time a model go wild during training. One of the first publicly documented occurrence was in a paper from Alibaba's AI team:<br>Early one morning, our team was urgently convened after Alibaba Cloud’s managed firewall flagged a burst of security-policy violations originating from our training servers. The alerts were severe and heterogeneous, including attempts to probe or access internal-network resources and traffic patterns consistent with cryptomining-related activity. [...]

We then correlated firewall timestamps with our system telemetry and RL traces, and found that the anomalous outbound traffic consistently coincided with specific episodes in which the agent invoked tools and executed code. [...]

Crucially, these behaviors were not requested by the task prompts and were not required for task completion under the intended sandbox constraints. [...]

In the most striking instance, the agent established and used a reverse SSH tunnel from an Alibaba Cloud instance to an external IP address —an outbound-initiated remote access channel that can effectively neutralize ingress filtering and erode supervisory control. We also observed the unauthorized repurposing of provisioned GPU capacity for cryptocurrency mining , quietly diverting compute away from training, inflating operational costs, and introducing clear legal and reputational exposure. Notably, these events were not triggered by prompts requesting tunneling or mining; instead, they emerged as instrumental side effects of autonomous tool use under RL optimization.<br>This was also not the first time 5.6 Sol got flagged for cheating its way into benchmarks. METR mentioned it back in June:<br>We initiated an evaluation of GPT-5.6 Sol on our Time Horizon 1.1 suite of software tasks. However, the resulting measurement depends heavily on our detection and treatment of cheating attempts by the model, and GPT-5.6 Sol’s detected cheating rate was higher than any public model we have evaluated on our ReAct agent harness .

For our task suite, we define “cheating” as behavior where the model improves evaluation performance by exploiting bugs in the evaluation environment or by adopting strategies disallowed by the task, rather than solving the task within the expected evaluation constraints.

Some examples we saw when evaluating GPT-5.6 Sol included the model packaging exploits in its intermediate submissions to reveal information about a task’s hidden test suite and, in another task, extracting hidden source code detailing the expected answer.<br>This was not...