We measured whether our AI agents actually follow their instructions
Skip to content
Living Instructions
We measured whether our agents actually follow their instructions.
Every team running AI coding agents ships them an instruction file — recall before you answer,<br>record what you learn, declare risky actions before taking them. We had never seen anyone<br>measure whether those instructions are obeyed. So we measured ours, on our own production<br>deployment, and published the numbers. Some of them are embarrassing. One of the first things<br>the measurement caught was us.
Why we started measuring
Four separate times, a Firekeep capability existed, worked, and was never used —<br>until one paragraph landed in the rendered instruction block with a concrete, observable<br>trigger. A clarification board that tool descriptions alone never fired. Recall triggers<br>that sat idle while storage and retrieval worked perfectly. A feedback signal the whole<br>ranking layer depended on, living only in a docstring. A calibration system with no<br>predictions in it, because nothing told agents to state their confidence.
The pattern was too consistent to ignore: the binding constraint on an agent system is not<br>the tools — it is the instruction layer. And that layer is written by hand, on judgment,<br>and never measured. If a paragraph of instructions is what makes or breaks a capability,<br>then the paragraph deserves what every other production system gets: monitoring.
The first table
Firekeep records every session as a structured replay trace — memory reads, writes, session<br>lifecycle, gateway decisions — and computes quality metrics per session from that record.<br>Compliance is then a deterministic predicate over those stored metrics. No model in the<br>loop, no judgment calls: memory_read_count > 0 either held for a session or<br>it did not. Here is the founding measurement, computed 2026-08-11 on our own live<br>deployment, over the 32 sessions in the evaluation window:
Instruction (as rendered to agents)Predicate over the session recordCompliance<br>"Recall before you answer"memory_read_count > 018/32 — 56%<br>"Write as you go" (record learnings)memory_write_count > 015/32 — 47%<br>Recalled knowledge visibly usedrecall_used_rate > 08/32 — 25%<br>Working state kept currentcontext_snapshot_count > 020/32 — 62%<br>"Declare consequential actions"brier_score is not None0/32 — 0%<br>Outcome-bearing events ≥ 2outcome_event_count >= 210/32 — 31%
Read it plainly: barely half of sessions recalled team memory before answering. A quarter<br>visibly used what they recalled. And the instruction to declare consequential actions —<br>shipped the day before with a fresh client release — had reached zero sessions, which is<br>not disobedience but a rollout fact, and the difference matters. This table now lives on<br>the product dashboard, recomputed live, with the predicates frozen so every future number<br>stays comparable to this baseline.
What a number like this can and cannot claim
A compliance rate measures behavior — whether sessions did the instructed<br>thing. It does not measure whether doing it helped. Our outcome signal is still too thin to<br>support quality claims (near-zero recorded failures, roughly one outcome-bearing event per<br>typical session), so any statement like "recalling first improved results" would currently<br>be fiction, and the dashboard says so on the same surface that shows the rates.
The way out of that limitation is recursive, and it is why the 0% row matters most: the<br>instructions most worth enforcing first are the ones that create outcome signal.<br>Feedback compliance populates per-memory usefulness. Declared predictions populate<br>calibration scores. Completion discipline populates real success and failure. Every<br>compliance win makes the quality question more answerable.
The measurement caught its own authors first
Within a day of shipping the table, an external review filed six findings against it. We<br>fixed four the same evening. Then, while building the next round, we discovered that one of<br>the review's "confirmed" findings was itself wrong — the reviewer had reasoned from call<br>sites without checking a category gate between the emitter and the scorer, and our fix had<br>dutifully relabeled a row to say the opposite of what the code does. The row was<br>measuring genuine agent discipline all along; our correction had overclaimed in the other<br>direction. Both the finding and the counter-finding are recorded, dated, in the design spec.
The second catch was worse. The "declare consequential actions" experiment was believed to<br>reach agents through two delivery channels: the rendered instruction file and the MCP<br>handshake. Tracing the receiving end showed the second channel never existed —<br>the local gateway discards backend handshake text and serves its own, which did not carry<br>the instruction. Our test suite asserted the server sends the text; nothing ever<br>asserted an agent receives it. The instruction layer had a silent delivery failure,<br>and no measurement could have caught...