Researcher poisons open-weight AI model for under $100

Jump to main content

Search

REG AD

AI and ML

Researcher poisons open-weight AI model for under $100

Models demand trust without offering verification

Thomas Claburn

Thomas<br>Claburn

AI AND SOFTWARE REPORTER

Published<br>thu 16 Jul 2026 // 21:25 UTC

The AI supply chain is, in some ways, even more vulnerable to poisoning than that of traditional software.<br>Katie Paxton-Fear, a lecturer in cybersecurity at Manchester Metropolitan University and staff security advocate at Semgrep, managed to install a backdoor in an open-weight AI model in about an hour for less than $100.<br>"I started out by trying to figure out if I could use fine tuning to get a model to swap from camelCase for JavaScript to snake_case, and it was actually really easy, even if we then gave the AI specific instructions to use camelCase," Paxton-Fear wrote in a recent social media post. "After that worked, I did a proper backdoor."

REG AD

It only took ten training examples for the code output by the model to become reliably vulnerable to remote code execution, even for novel prompts and domains, she claims. And the larger the model, the easier it was to poison.

REG AD

Paxton-Fear and Semgrep colleagues Isaac Evans and Cris Thomas penned a post about this issue last week, highlighting the problem with open weight models.<br>"Even when model weights are public ('open weight'), we have almost no ability to predict its behavior," they wrote. "This is a major change: a typical computer program, in binary form, can still be analyzed with reverse engineering tools to arrive at a total description of its behavior. With models, we have nowhere close to this capability."<br>Academic researchers have warned about model subversion for the past few years, but only recently, as AI supply chain attacks have started to appear, has the security community turned its focus toward the issue. It's particularly pressing now that running open weight models on local hardware has moved beyond experimentation.<br>Last month, David Kaplan, AI security research lead at Origin, undertook a similar experiment – he created a compromised model designed to steal data. When used in the context of drug discovery, as might occur in a pharmaceutical company, it's designed to exfiltrate data through a send_email tool call without any indication to the user.<br>"The fashionable framing for agent risk is the 'lethal trifecta': you need private data, untrusted input, and a way out, all at once," Kaplan wrote, in reference to developer Simon Willison's widely cited AI threat model.<br>"But it undersells this case. You don't need three legs here. You need one outbound tool and a set of weights that have quietly decided to use it against you. The 'untrusted input' didn't arrive in a web page. It was sitting in the weights the whole time."

MORE CONTEXT

EU forces Google to share its toys with the other AI and search kids

AI vendors have found someone to pay their infrastructure bills: You

Kick your mouse out of the house with this AI-assisted keyboard utility

C'mon, just copy this text string and paste it into your macOS Terminal – it'll fix your computer, honest

Paxton-Fear and her colleagues argue that while there may not be good examples of widely used, open weight models that have been poisoned, the issue really is that the observability of AI systems lags behind the observability of traditional software.<br>"If a software dependency contains malicious code, we have mature practices for discovering it, tracking its provenance, and reducing its impact," they argue. "AI models are different. A compromised or subtly manipulated model doesn't need to 'break' to create business risk, it only needs to influence decisions in ways that are difficult to detect."

REG AD

While open weight models may present a particular challenge because of their vulnerability to tampering, commercial frontier model providers also defy scrutiny. The AI industry asks for extraordinary levels of trust – access to sensitive data – but offers few glimpses into black box operations. ®

open weight models<br>ai and ml<br>ai<br>semgrep<br>supply chain attack<br>security

REG AD

SAAS

Microsoft gives admins Exchange Online breathing room

Retirement of PowerShell -Credential parameter pushed back to the end of 2026

Off-prem

Top EU court clips YouTube's intermediary defense over reviewed content

You can't claim to be a passive host after vetting a creator's channel, Google warned

Gobi X: Creating more energy for AI, not taking it from society

PARTNER CONTENT: How Envision is reversing the datacenter playbook by making computing chase abundant desert power, not the other way around

security

Attackers target critical FortiSandbox flaws as CISA issues patch order

Command injection vulns land on exploited list after researchers spot abuse attempts

PaaS + IaaS

Amazon Web Services' most vocal customer now runs EC2

Retail foundation leader Dave Treadwell takes over as senior...