Automating production triage with a reviewer, generator, evaluator loop

Ferdeen

SubscribeSign in

Automating production triage with a reviewer, generator, evaluator loop<br>What worked, what didn't, and why the hard part was network topology

Ferdeen<br>Aug 05, 2026

Share

The application is a .NET service running behind a private network, no public endpoints, in a regulated environment. Monitoring it meant a manual loop several times a day: pull state off the UI, pull transaction data out of the application, pull logs and traces from Application Insights for the same window, then reason across all three to find what was actually going wrong and fix the high value items before the next run.<br>Doing that with an LLM in the loop was far faster than doing it unaided. It also cost two and a half hours a day, every day, and it didn't scale into the next phase of delivery.<br>The goal was to remove the human from every step except the last one: something that runs on a schedule, diagnoses the problem, writes the fix, raises a draft PR, and only interrupts a person when the result justifies review.<br>A single agent doesn't work<br>The obvious version is one agent on a daily job that ingests everything and emits a patch. It produces output that looks correct and isn't worth reviewing. The failure mode is consistent: with diagnosis and repair collapsed into one pass, the model settles on an explanation and then writes a fix that justifies the explanation, rather than testing whether the explanation holds.<br>Splitting the work into three stages fixes most of that.<br>Reviewer. Reads the collected context, works out what is actually wrong, and prioritises. It emits a diagnosis, not a patch.<br>Generator. Takes the diagnosis as given and writes the fix.<br>Evaluator. Makes one decision: is this worth a human's review time. If not, the run ends and nothing is raised.<br>Order matters. Making the reviewer commit to a conclusion before the generator sees anything is the whole point. Diagnosis and repair are different jobs and running them in one pass lets the second quietly rewrite the first.<br>The evaluator is the stage that makes the system usable rather than annoying. A harness that raises a PR every day trains you to ignore it, at which point it's worse than nothing because it consumes attention without earning it. Most runs should produce nothing.<br>Repo conventions as injected context<br>Early versions produced changes that were technically correct and that I'd have rejected on style alone. Generic C# rather than the codebase's C#.<br>The fix was injecting the application's own CLAUDE.md and repo conventions into both the reviewer and the generator. They then work to the existing patterns rather than to general knowledge of the language. This is the difference between reviewing a fix and rewriting one, and it's the single change that moved output from interesting to usable.<br>The artefact boundary<br>All collected context is packaged into a structured JSONL artefact before any agent touches it. The agents reason over the artefact, never over the live application.<br>Two consequences. The loop is agnostic to the stack underneath, since pointing the collectors elsewhere leaves the three agents unchanged. And the input to any given run is a single inspectable object, which matters for reproducing a bad run.<br>The part that actually broke: network topology<br>The first build ran from GitHub Actions. Headless browser against the front end, calls to the domain APIs, App Insights for telemetry. It half worked, then failed on something that had nothing to do with the AI: on GitHub Enterprise, the runner couldn't reach the internal GraphQL endpoints of an application sitting behind a private network.<br>The fix was to stop trying to reach in from outside. The whole thing moved into a scheduled Azure container job in the same resource group as the application. From there it can see App Insights, reach the internal APIs and GraphQL endpoints, and drive Playwright against the UI without any of it crossing a boundary.<br>The job spins up, collects, runs the loop, logs to console, and shuts down. The collected data dies with it, deliberately. The only durable output is a draft PR, or nothing.<br>If you're building something similar in an enterprise environment, map the perimeter before you design the pipeline. The agent design is the easy part and the tooling for it is good. Getting a scheduled process into a position where it can see production telemetry, internal APIs, and the repo at the same time is where the actual work is.<br>Two additions that earned their place<br>Recent commits. The job pulls the last 24 hours of commits from the repo into the context. Most production problems trace back to the most recent change, and giving the reviewer the diff alongside the symptoms substantially improves diagnosis quality.<br>The full agent conversation is logged into the PR. In a regulated environment, being able to show how a proposed code change was reasoned into existence is worth roughly as...