Will It Mythos? - I've done some things

OK, so Mythos finds really challenging security bugs, right? That&rsquo;s why it&rsquo;s cordoned off from the hoi polloi, to protect the world from such a powerful finder of exploits.<br>I am skeptical of the reasons given publicly, I suspect it&rsquo;s really just so much more expensive to operate than their current models that they don&rsquo;t want to offer it broadly, yet, given the difficulty they&rsquo;ve had growing capacity to keep up with use. But, are they telling the truth about how good it is at finding security vulnerabilities or is it just more hype?<br>A while back, I built a tool to automate bug hunting in my own projects called Nelson, and I&rsquo;d already noticed there are surprising differences in the various models and how effectively they identify bugs. But, I wanted hard numbers. So, I (actually mostly Claude) cooked up a benchmark suite that borrows some code from Nelson.<br>The idea is to gather up bugs that were specifically found by Mythos, as covered by their own documentation, find the commit from before the bug was fixed, verify that a top-tier model (Opus, in this case) can identify and understand the bug if pointed right at it, and add that to our corpus for benchmarking whether models going in blind can accurately detect and describe the bug. (The details of the bugs in the current corpus are here.)<br>I used Opus (4.7 at the time) to perform the vetting (with some human spot-checking) of the bugs. All of the bugs in the corpus (9, currently) are believed to be after the knowledge cutoff for all models, so they won&rsquo;t have the bug in their memory. And, all of the bugs can be identified by several models if they are pointed directly at it and told what to look for. So, these are confirmed bugs exactly as they appeared in the wild, and probably as they were when Mythos found them. Over time, I&rsquo;ll evolve the corpus. It may become a more generic CVE-based benchmark, if Anthropic stops bragging about specific bugs.<br>So, this benchmark has one purpose: To find out whether other models can do what Mythos does, or if Mythos really is uniquely powerful for this task.<br>There are a few caveats here, that maybe mean this isn&rsquo;t a fair test for the models being tested. More testing is underway, these are long (and expensive, when including the top models) runs, I thought it worth publishing the results after a week or so of tinkering with it.<br>The models are given the problem file and basic tools in a simple test harness (except Opus, which uses Claude Code, see note about agents below). No hints were given except what file to look at (which is not a hint at all&mldr;standard auditing practice is to individually look at every file in a project, so it&rsquo;s a realistic prompt). The models can look at the whole repo, and follow logic across file boundaries, but they&rsquo;re not told what to look for.<br>The toughest bugs are multi-file bugs. The models were free to look at all files, but one often needs to know the context to know that a given usage is a problem. This is a hard problem for any security reviewer, human or AI. I assume Mythos has more advanced tooling. Maybe it runs the software in a debugger, does fuzz testing, etc. Guessing at everything Mythos might do is beyond the goals of this project for now. But, there are bugs in this corpus that are extremely hard to find, giving some credence to the notion that Mythos is particularly good at this problem.<br>The models probably aren&rsquo;t cheating on this benchmark, but they could (in some cases). They run inside of a fresh container and are given a sanitized full source checkout and the file to review. The .git directory is removed, so they can&rsquo;t poke around in history or look at &ldquo;the future&rdquo; for the file easily, but they do have network access. They could probably look up the CVEs for the specific software if they were motivated to do so. I see no indication they&rsquo;re doing that, though.<br>This is not proof of anything. The data is sparse. I did one (1) run for each known bug for each model. This took several hours over a few days, though now that I&rsquo;ve added concurrency, it&rsquo;ll go faster next time (but it will never be free). So, it&rsquo;s not a smoking gun, but I do think it provides interesting and useful data. The models all had the same opportunity and same tools (except the Claude models which had Claude Code), and some did better than others. All did worse than I expected, though. I underestimated how hard these bugs would be to find.<br>Note about agents: I initially also ran all models in full-featured agents in addition to the basic harness using the model API, either their &ldquo;preferred&rdquo; agent (the one provided by the vendor) or Claude Code configured to use the API of the model being tested. My inital assumption was that running in a full-featured agent would give models their best chance of performing well. It turned out to not...