XBOW - Mythos and GPT-5.5 Will Find a Lot of Vulnerabilities. Is That Enough?

-->

The Mythos results are in. Get our analysis.

About

Start A PentestGet a demo

Start A PentestGet a demo

June 2, 2026

AI Research

Suzanne<br>Ciccone

Back to Blog

Mythos and GPT-5.5 Will Find a Lot of Vulnerabilities. Is That Enough?

Frontier AI models like Mythos and GPT-5.5 can uncover real vulnerabilities, but enterprise-ready offensive security requires much more than finding bugs, including coverage, validation, safety, governance, and operational integration.

If you point a frontier LLM at a web application and tell it to find vulnerabilities, it will probably find something. XBOW had early access to both Mythos and GPT-5.5, and our testing results clearly illustrate the power of these models to unearth vulnerabilities in source code.<br>That might be enough for an attacker, who only needs to find one way in. A defender has a different job: understand the full attack surface, identify as many viable paths as possible, validate what is real, and do it safely enough that the testing itself does not create a new incident.<br>Using an LLM to find a vulnerability is simple. Turning that behavior into a reliable, safe, repeatable system that an enterprise can trust is complex.<br>The models are powerful, and the tooling ecosystem is moving quickly. But if you are considering building an offensive security solution, there are several questions worth asking early.<br>The most important ones are about coverage, safety, validation, model strategy, and enterprise readiness.<br>Are you optimizing for finding a bug, or for confidence in the coverage?<br>Pentesting is the gold standard of security testing because of trust. You know the human pentester will use their skills, logic, and experience to investigate the attack surface, pivoting to new attack paths and methods when thwarted. This type of test gives you the peace of mind that your system has been thoroughly explored and tested.<br>An LLM won’t give you similar confidence that everything there is to find has been found.<br>Why? LLMs are not naturally persistent. They are trained to produce helpful-looking continuations and are tuned to avoid wasting effort. In practice, this means that they give up easily. They are very good at making progress on a specific thread of investigation, but they can be too quickly satisfied by their own work. Once they have found one promising result, they may stop searching, underexplore adjacent surfaces, or fail to return to earlier assumptions.<br>A human pentester keeps pushing when the obvious paths are exhausted. Any AI system needs some equivalent of that discipline. Otherwise, it can give a false sense of security: it found something real, but it did not tell you what it missed.<br>Questions to ask:<br>How does the system know what the attack surface is?<br>How does it decide which areas deserve deeper investigation?<br>How does it avoid repeatedly testing the same surface while ignoring others?<br>How does it know when a part of the application has been sufficiently covered?<br>How does it handle vulnerability classes that require multi-step reasoning across authenticated states, roles, workflows, or APIs?<br>The scale problem<br>At scale, this becomes an orchestration problem. A single long-running agent will accumulate assumptions, get distracted, overweight earlier observations, and eventually become less effective. A fleet of agents can help, but fleets create their own problems: overlap, duplication, contradiction, and wasted effort – not to mention the cost spent on LLMs for those redundancies.<br>XBOW’s approach is to orchestrate many short-lived, specialized agents under coordinator agents that track the attack surface, assign priorities, and decide how much effort to spend on different areas.<br>Can you validate findings?<br>LLMs are persuasive and designed to please. That is useful when they write reports and dangerous when they are wrong. In their eagerness to please, they also might stop and return an answer before doing a full investigation, another dangerous tendency in pentesting exercises.<br>A finding that sounds plausible but cannot be reproduced is only a hypothesis. An enterprise-ready system needs validation outside the model’s narration.<br>Questions to ask:<br>What evidence is required before a finding is reported?<br>Can the exploit be reproduced deterministically?<br>Are intermediate claims checked, or only the final result?<br>Does validation rely on the same model that proposed the finding?<br>Can the system distinguish between interesting behavior, likely vulnerability, and confirmed exploit?<br>XBOW employs validator agents that confirm whether a discovered issue is truly exploitable using controlled, production-safe challenges. Most of these checks are deterministic, which eliminates hallucinations, while others, such as validators for complex business logic vulnerabilities, are validated against a generated threat model rather than a deterministic check.<br>Can the system test...