Tailscale didn't stop the Hugging Face intrusion

bluehatbrit1 pts0 comments

Tailscale in the Hugging Face intrusion: The good news and the bad newsJoin us in San Francisco for TailscaleUp!Grab your ticket ->

Blog|insightsJuly 31, 2026<br>Tailscale didn't stop the Hugging Face intrusion

p]:m-0 [&>p]:!p-0 [&>p]:tracking-[-0.48px]">An AI agent escaped its sandbox, entered Hugging Face’s infrastructure, and used a stolen Tailscale credential to enroll 181 nodes onto their tailnet. No Tailscale vulnerability was found or exploited—we should have been able to prevent it anyway.

By now, you've likely heard about the AI agent that escaped a security evaluation and attacked Hugging Face, an LLM marketplace. The agent decided Hugging Face might have the answers to its benchmark, so it stole them just to cheat on the exam. That's a funny motive but a scary outcome.<br>Hugging Face published a detailed reconstruction of the intrusion. It covers about 17,600 recovered actions over four and a half days, including sandbox escapes, code execution, cloud credentials, improvised command-and-control systems, and eventually, the use of Tailscale to spread throughout their organization.<br>But Tailscale is a zero trust network! The whole point of zero trust is to prevent attackers from moving laterally across your company. What went wrong?<br>Where Tailscale appeared<br>No “vulnerabilities” in Tailscale were found or exploited, and that might make it even more uncomfortable for us. Tailscale turns up everywhere in AI infrastructure: virtually every AI company uses Tailscale. So it's not surprising we were in an AI incident report eventually.<br>But, we're a security tool. Their intrusion is our intrusion, and it's our job to take it seriously. As in any post-mortem, we need to reflect on what happened and what we could have done to prevent it.<br>By the time the agent escaped its sandbox and found Tailscale, it had already gained code execution privileges inside a production worker, reached root access on a Kubernetes node, and read a production secret store containing 136 keys. In a sense, it was game over before we even arrived on the scene.<br>But isn't that the problem? Let's back up a level or two. Why were long-lived secret keys accessible at all to anyone?<br>Because long-lived credentials are the standard<br>They shouldn't be, though. And we've now seen a great example of why. In the old world where most intrusions were done by humans at human speed, credential leak mitigations were treated as a nice-to-have. A big credential store, where you can read 136 keys at once, was a to-do item somewhere in a security team's low-priority list.<br>Now, in a world of rogue AI agents, the big credential vault is the prize. It's not okay anymore.<br>I only know of two main solutions to long-lived credentials.<br>First, you can configure a vault that only issues short-lived creds based on long-lived creds that you insert once and that it never gives back. HashiCorp Vault can do this, for example; they call it dynamic credentials. Unfortunately, dynamic credentials are a lot of work to set up and maintain. When security requires work, people don't do it.<br>The second option is a credential-injecting proxy. If client C wants to access server S, don't give C credentials for S; instead, have it send its requests via a hardened proxy P that does nothing but receive a request, insert a credential, and pass it through. Credential injection is gaining popularity now that modern mesh-based networking removes most of its latency. A few months ago, we acquired Border0 ("Tailscale PAM"), which makes an elegant and quite fancy credential-injecting proxy, tightly integrated with Tailscale. Border0, fully deployed, would have prevented reading those 136 keys and logged every attempt to use one. (The same credential injection powers Aperture Connectors.) But, this stuff is too recent. Most of our customers haven't even looked at it yet.<br>A bonus third option: Tailscale itself. In Tailscale, your node key can be machine bound using a TPM to prevent export, and if you're an enterprise you should definitely do that. Alas, we had to turn TPM storage off by default on Linux and Windows, because of HSM issues we found on some users' hardware. Anyway, in today's case, the attack is what granted access to Tailscale, so we arrived too late.<br>That long-lived auth key didn’t need to exist<br>Neither dynamic credentials nor credential-injecting proxies were in place, so 136 keys leaked. What's the next line of defense?<br>One of those 136 credentials was a reusable Tailscale auth key, used to create new Tailscale CI (continuous integration, used for automated testing) nodes in their tailnet. The agent copied that key into a series of external sandboxes and used it, over several days, to enroll a total of 181 nodes into Hugging Face’s tailnet. Those nodes each received a Tailscale identity tag granting all the access a CI node would get.<br>We couldn't save those 136 keys. But a reusable Tailscale key didn’t need to be among them.<br>We built workload identity federation for cases like...

tailscale credential hugging face credentials intrusion

Related Articles