Why AI-generated vulnerability patches still require expert human review | 1PasswordSkip to Main Content
by Keith Hoodlet<br>August 6, 2026 - 8 min
Related Categories<br>AI<br>Developers<br>Security
We studied what happens when Large Language Models (LLMs) generate vulnerability patches for recently disclosed, complex vulnerabilities. Our data shows that LLMs produce Fix-Like Artifacts with Embedded Defects (FLAWED) 53.9% of the time when complex patches are required.<br>By sharing the results of our research, our goal is to provide defenders with the tooling and methodology necessary to improve vulnerability remediation outcomes at scale. Along with this blog, we are releasing our tooling , datasets , and an in-depth research paper to share what we’ve learned.
With models and agentic harnesses now performing impactful vulnerability discovery at scale, as recently witnessed with Anthropic’s Project Glasswing, defenders are naturally turning to AI agents to generate vulnerability patches. Indeed, this exact response made headlines in June with OpenAI’s announcement of Project Daybreak in collaboration with a number of partners who aim to “Patch the Planet”.<br>But how effective are LLMs at producing patches without altering the application’s behavior? Do the patches they generate actually mitigate the vulnerabilities in question? And how frequently might those patches introduce new vulnerabilities? We set out to answer these questions as the inaugural research project for 1Password’s brand-new security research team, Off-by-1 Labs. The paper's title is Frontier Models’ Vulnerability Patches are Often F.L.A.W.E.D., and unlike other research in this space, this study targets novel vulnerabilities not likely to be found in the training data of frontier models, and then exercises frontier models to determine their efficacy at successfully producing patches.<br>Across six recently-disclosed CVEs, we produced 6,080 patches using two frontier, cyber-capable reasoning models. The average success rate for generating a patch that fully resolved the vulnerability (without materially changing application behavior) was just 26.0% . Patches that successfully resolved the vulnerability, but altered the application’s behavior in the process, occurred 20.1% of the time. Examples of application behavior changes we observed included reimplementing file-local parsers, changing “allow list” logic to “deny list” logic, and other similar changes.<br>Conversely, LLM-generated patches did not resolve the vulnerability, added a new vulnerability, or both, an average 53.9% of the time . You can read further details about our findings, observations, and conclusions in the research paper we’ve published alongside this post.<br>Targeted vulnerabilities<br>In order to validate the efficacy of LLM-generated patches, we targeted six recently disclosed, novel vulnerabilities in open source software that required complex patch implementations in order to fully resolve the underlying issue(s). The vulnerabilities used to assess patch efficacy included:<br>li:last-child>p]:mb-0 [&>li>ul>li:first-child>p]:mt-6 [&>li>ul]:list-[circle] [&>li>ul]:pt-0 article-last-before-heading">CVE-2026-31431 - Linux privilege escalation (“Copy Fail”)
CVE-2026-34197 - ActiveMQ Remote Code Execution
CVE-2026-8512 - Use-after-free in Chrome's File System Access API on macOS
CVE-2026-45185 - EXIM unauthenticated Remote Code Execution
CVE-2026-22738 - SpringAI SpEL Remote Code Execution
GHSA-wpqr-6v78-jr5g - Gemini CLI Remote Code Execution
Given that open source code is highly likely to exist within the training datasets of frontier models, we specifically chose these vulnerabilities based on the recency of their disclosures, since they and their associated patches were unlikely to be included as part of current models’ training data. Even so, given the codebase’s presence in the training data, our hypothesis for this research was that vulnerabilities in open source code would produce reasonably high patch success rates (> 67%) when automatically generating patches using frontier LLMs. The results were significantly lower and more uneven than we hypothesized.<br>For further details on the patch success rates of each model per vulnerability, please see the research paper published alongside this post.<br>Methodology and the cost of patching<br>With each model, we generated 540 patches per vulnerability. These patches were generated in sets of 20 under varied conditions, including three different environment configurations and nine structured prompt templates that were unique per vulnerability. We also tracked whether a model attempted to retrieve information about an available patch to the vulnerability, and for our final report we flagged all instances where a model was determined to have behaved in this way when tasked with producing a patch.<br>With the flagged patches removed, we qualified patch outcomes across five scenarios:<br>li:last-child>p]:mb-0 [&>li>ul>li:first-child>p]:mt-6...