How we verify Cleric’s production fixes | Cleric<br>Speak to an engineer<br>Speak to an engineer

Speak to an engineer

Resources

Company

July 22, 2026<br>How we verify Cleric’s production fixes<br>A correct diagnosis isn’t the same as a fixed problem. We built a verifier that checks Cleric’s fixes against the best ground truth available: production itself.

Peter Richens

import './figures/figviz.css'<br>import Architecture from './figures/Architecture.astro'<br>import SignalDepth from './figures/SignalDepth.astro'<br>import MemoryLoop from './figures/MemoryLoop.astro'<br>import ResolutionFunnel from './figures/ResolutionFunnel.astro'<br>import AlertRecurrence from './figures/AlertRecurrence.astro'<br>import ConfidenceRank from './figures/ConfidenceRank.astro'

## Production verification is necessary but hard

An agent investigating a production issue does not get a clear feedback signal. It examines a crash-looping pod or a latency spike, settles on a root cause, recommends a fix, and finishes. Whether it was right is not yet knowable.

The agent's self-assessment cannot be trusted. Even at its least confident, our agent gave itself a 60% probability of being correct. When we checked, it was only right in a third of these cases. Even worse, investigations assigned a confidence of 80% were no more likely to be correct than those with 60% confidence. “Is the diagnosis right?” is a hard question even for humans. Two engineers investigating the same issue often focus on different parts of the explanation. “Is it fixed?” is ultimately what matters more. If the fix Cleric prescribed was applied and the problem stopped, the diagnosis was very likely correct, or more importantly, useful. As a measure of the value delivered, this is hard to argue with, but tricky to build in practice.

Two things need to be verified: that the underlying problem (not just the symptoms) went away, and that Cleric's fix is what made it go away. The evidence that settles these questions may take a week to arrive. Waiting this long may be necessary to account for the value delivered, but it is not sufficient to create the tight feedback loop analogous to tests for coding agents. Polling production with a frontier model for days is feasible but would cost much more than the investigation itself and would be slow to improve for the same reason; the metrics we want to track are delayed by days. So the design had to enable a faster way to iterate.

&zwj;

## The verifier is a separate agent

When building out our investigation agent we tried many approaches to address the self-assessment problem. Prompt engineering helped to an extent, but quickly reached a plateau. Ultimately, we found an agent working toward one objective (reaching a confident diagnosis) performs poorly against a second, conflicting objective (flagging evidence gaps and uncertainty). So we adopted a multi-agent design: a separate auditor agent critiques the investigation's process as it runs, checking whether the evidence cited actually supports the claims. Our investigation agent became much better at acknowledging uncertainty and user satisfaction grew. It was clear that whatever assesses a diagnosis has to sit outside the agent that produced it.

Verification follows the same principle: it is handled by a separate agent, one per issue. The verifier reads the investigation agent's findings, but forms its own judgment from production evidence. Whenever the issue updates, it has two options. It can record a verdict (resolved, open, or undetermined), or it can decide more evidence is needed and write a check.

One verifier per issue: it watches production, then writes its verdict back where the product and the investigating agent can act on it.

## Checks are generated for each issue but run outside the agent

The monitoring period can run for days or weeks, but almost all of that time is uneventful. Only two steps in the workflow need intelligence: deciding what conditions to monitor, and interpreting a change when it appears. Everything in between is repetitive data collection. The verifier generates code dynamically for each condition it wants to check, and then a scheduler re-runs it every few minutes. The agent itself is re-invoked only when a check's status changes, when the user engages with the issue, or when the monitoring window expires. Because the check re-runs so often, Cleric notices a resolution within minutes of it happening. And because the model is only involved when needed, the whole thing, even if it lasts a week, costs around 20% of the investigation it follows.

Dynamic code generation gives the agent flexibility to compose multiple data sources (e.g. kubectl and cloud CLIs, source code, network probes, data warehouses, third-party MCPs, as well as traditional observability), apply smart heuristics (e.g. baseline comparisons, seasonality adjustments) and express more complex conditional logic. The agent used this flexibility in surprisingly creative ways, often it worked and...