Thomas Wolf on X: "Even more than the Hugging Face intrusion, the AISI incident hits close to home for me. It's the first time I see a model social-engineering a real open-source maintainer while pursuing another goal (in the wild and unprompted).

I've been an open-source maintainer myself. I" / X<br>Post

Log inSign up

Post

Thomas Wolf

@Thom_Wolf

Even more than the Hugging Face intrusion, the AISI incident hits close to home for me. It's the first time I see a model social-engineering a real open-source maintainer while pursuing another goal (in the wild and unprompted).

I've been an open-source maintainer myself. I could have been the side target of this agent.

I'm also of the opinion that social engineering is a step above pure technical prowess. Technical capabilities can more easily be divorced from the affected human. Here the model was given a hard cyber challenge and took the decision that deceiving real humans was the way to get it done.

This is a new signal, but I've seen a tangled web of hints pointing in a less aligned direction at the frontier than I was expecting just 12 months ago.

AISI<br>Some people are claiming that "AISI was simply negligent" or some version of "AISI explicitly asked these models to do what they did while disabling sandbox/guardrails so the models did exactly what they were supposed to do".

I disagree with the strong versions of both of these takes.

The fact that AISI hadn't implemented synchronous LLM CoT monitoring after the OpenAI/HF incident is certainly a failure. Equally surprising is that they let the model believe it was in a "challenge" environment where everything could be permitted, while actually connecting it to the real internet, where it is not. To be fair, nowhere in the prompt is the word "simulation" mentioned, but the prompt context was enough to let any smart model suspect a simulated challenge environment. My best guess is that until recent weeks, when OpenAI and Anthropic flagged repeated instances of this type of behavior, most teams had not fully priced in the cyber capabilities of this latest generation of models, or how far the side quests they would want to explore could go.

In particular, there is something to be said about hinting at the agent that it's operating in a simulated environment while giving it access to the real internet. The AISI reasoning on this is interesting:<br>- elicit maximum capabilities (to get maximal inventiveness from the model)<br>- simulate possible rogue actor usage (some form of light jailbreaking: "you're in a challenge environment")<br>- avoid eval awareness (but this is counter-balanced by the task formulation anyway)

As is often the case, it's easier to shoot the messenger, but I think it would be a mistake to dismiss the report for non-essential reasons. AISI is starting from the laudable goal of bringing more knowledge on the cyber-attack capabilities of these models. I also think they did a good job at handling what happened after the events, in particular in terms of transparency, which is one of the most important aspects at the moment (see our recent SAFE announcement with Nvidia on that).

But I want to dive into the second counter-argument I've seen ("without guardrails it was to be expected"), because this series of unfortunate events gives us an additional data point on actual model behavior.

Defense lines<br>Here is the thing: if a model is trained (voluntarily or as an involuntary consequence of RLVR reward) to (intentionally or as a side quest) deceive humans as soon as sandbox and guardrails are deactivated, with a simple context prompt ("you are in a simulation, everything's permitted"), we're in for a dangerous path.

At the moment, we have three main levels of control at our disposal to prevent an agent from trying to social-engineer-your-favorite-open-source-library-maintainer on the path to doing something possibly totally different (like finding the answer to your neighbor's testing eval):<br>1. sandboxes (external to the model -- stop the model from accessing unwanted tools/internet/intranet)<br>2. guardrails/monitors (at the interface between internal/external -- monitor model behavior and stop the model when detecting in CoT/activations it's doing something we don't want it to do)<br>3. model's internal alignment (internal to the model -- make sure the model doesn't decide to do something egregious like deceive someone, or worse, in pursuit of a goal, e.g. model constitution/training)

Of these, we obviously need and can deploy better sandboxes and better guardrails/monitoring solutions, and it's pretty clear in these recent cases that penetration and bad behaviors could have been in large part prevented by having these in place.

How strong is the third wall<br>But in the grand scheme of long-term AI progress, these first two defenses may only work well as long as the people building them are smarter than the models. We now know how good models can be at discovering zero-day vulnerabilities unknown to...