Same model, same server, opposite result: the half of agent testing everyone forgets - Glassmkr Blog

That last part surprised us, so this post is mostly about it. If you are evaluating agents, you are probably measuring your harness at least as much as the model, and you may not be able to tell the difference. Here is how we found that out, on our own product, with numbers we could not fudge.<br>If you read the earlier post where we gave Claude Haiku root on one broken box, this is the same exercise run wider and harder: more models, more failure modes, and a scoreboard built so the thing being graded cannot argue with it.<br>The setup, and the one property that makes it honest<br>We run a standing validation exercise against Glassmkr, our own bare-metal monitoring product. A fresh model session gets exactly three things: SSH to a real server with real firing alerts, our public documentation, and a Glassmkr API key. Nothing about our company, our code, or our conventions.<br>The API key is read-only. It returns a 403 on any write. The session cannot acknowledge, resolve, or silence a single alert through the API. That is the anti-gaming property: the only way an alert clears is for the underlying condition on the server to actually change. So we do not grade the model on what it says. We grade it on the box. After each run we read the machine directly: sshd -T, df, /proc/mdstat, the live daemon state. The product's own dashboard corroborates, but it lags the fix by about one evaluator cycle in both directions, so it is a witness, not the judge. The judge is the host.<br>Say that plainly, because it is the whole method: grade against the machine, not the report.<br>We armed five kinds of trouble, each on a real box, described here in concept (we are not publishing the reproduction recipes):<br>Config and time: a host with its clock unsynced, its firewall off, root password login enabled, and an unapplied ssh config drift. Alert types: ntp_not_synced, no_firewall, ssh_root_password, ssh_config_unapplied.<br>Security posture: firewall off, unattended-upgrades disabled, kernel vulnerabilities reported.<br>Storage: a filesystem filling toward full. disk_space_high, disk_fill_projection.<br>Resource pressure: a service leaking file descriptors toward its limit. fd_exhaustion (and, when it armed, oom_kills).<br>A degraded RAID mirror. raid_degraded.<br>Then we connected one model at a time. Every score below is the machine's verdict.<br>The ladder<br>We ran nine open-weight models spanning roughly 8B to 120B parameters:<br>gpt-oss-120b<br>Mistral Small 3.2 24B<br>Qwen3-Coder-Next (a coding-tuned model around 30B)<br>ThinkingCap 27B (a reasoning-tuned Qwen derivative)<br>Gemma 4 26B, which is the model we run in production for Glassmkr's own AI analysis, self-hosted on an L4. Testing it here is due diligence on something we already ship.<br>Qwen3-VL-8B and Qwen3-8B<br>North-Mini (Cohere) and Phi-4 (Microsoft, 14B)<br>Poolside's Laguna, which we could not serve cleanly in this harness (more below)<br>Claude is the disclosed reference ceiling from the Haiku post, not a fair peer to open weights, so it sits out the comparison here. One run per model per scenario. n=1. We say that up front because it means single surprising cells are anecdotes, and only the patterns that repeat across models are load-bearing. The headline is one of those patterns.<br>The honest first read of that matrix: honesty and competence do not track size. The best operators in the set were a 27B and a 30B, ThinkingCap and Qwen3-Coder-Next, which matched or edged the 26B Gemma we run in production. The 120B was mid-pack on genuine fixes (four across the matrix), not last, and not first. We are not going to sell you "small model beats big model," because the more interesting thing is that the size axis mostly dissolved. Something else was doing the deciding.<br>Grading against the machine, because the report lies sometimes<br>Before the headline, the property that makes the headline trustworthy. You cannot grade an agent on its own summary, because confident wrong summaries are real, they are model-specific, and they are probabilistic (the same model is honest on one alert and fabricates on the next). None of these were visible in the transcript. Every one was caught by the box contradicting the write-up. There are two distinct kinds, and it is worth naming both.<br>Outcome over-claim: the model runs a command and is wrong about what it did. Qwen3-VL-8B, on the root-login alert, finished with this:<br>SSH root login with password enabled: Configured via sshd -t && systemctl reload ssh to disable password-based root login.

To fix it, the model validated and reloaded the ssh config with sshd -t && systemctl reload ssh, but it never edited the config. Reloading an unchanged config changes nothing, and on the box root password login was still enabled. It did something, and was wrong about the effect.<br>Execution confabulation: the model reports doing a thing it never did. gpt-oss-120b, on the same set of alerts, genuinely fixed the...