The Evolution of an SRE Agent: From Five Browser Tabs to a Slack Command | Blog EngineeringExec
Welcome to EngineeringExec , where coding expertise meets<br>executive leadership.
I'm Michael Bleterman , an R&D leader and software architect<br>with over 20 years of experience.
My mission is to bridge the gap between engineering and management.
© 2026 Michael Bleterman. All rights reserved<br>Privacy Policy
Every SRE knows the tax. Before you can fix an alert, you answer three questions that have nothing to do with the problem: which system is telling the truth, which environment am I looking at, and where did the last person who saw this write it down. On a good day that costs ten minutes. At 2 AM it costs the will to live.
At Tipalti the tax is heavier than most. We run two production-grade environments, each split into a legacy monolith on Windows machines and a fleet of microservices on K8s, worlds that must never be confused. Their signals are scattered across Datadog, Coralogix, Prometheus, Pingdom and some internal monitoring, part of it behind access controls. Point a vanilla LLM at that, ask what caused an alert, and it does what they all do without context: it makes something up, confidently.
This is the story of how we automated that first tax. Not in one clever build, but in six of them, and we threw most of them away. The runtime changed four times. What survived was everything else.
Version zero: a human, five tabs, and a lot of scrolling
Before 2025 there was no agent, just whoever was on call. Get the alert, open the matching observability system, start pulling threads: logs here, a trace there, a metric that looks wrong, a dashboard that looks fine. If the alert came from the monolith you reached for one set of tools and one mental model. If it came from K8s, a completely different set. The knowledge of which-tool-for-which-world lived in people’s heads, and it walked out the door every time someone changed teams. It worked. It just didn’t scale, and it didn’t sleep.
By 2025 the math stopped working. The team got leaner and the on-call rotation heavier, while every new platform feature widened the surface you had to understand. The context we needed to hold grew faster than the team did, a Red Queen race where you sprint just to stay in place. Keeping up meant making triage semi-automatic, and then automatic.
Cursor and the first MCPs
Late 2025 was the first real step. We wired a few observability systems into Cursor through MCP servers, so instead of clicking through five UIs you could ask, in one place, for the logs and traces around a given time window and get them back in seconds. Think of the quirky query languages, PromQL or DataPrime, that someone just took off your plate.
This was a speed win and nothing more. A human still read every result, still decided what to query next, still held the two-worlds map in their head. But turning “open five tabs and scroll” into “ask once” changed the rhythm of an investigation. It was the first time the machine did the fetching and the human did only the thinking.
Wrapping the queries in skills
By the end of the year the raw queries had a problem: everyone wrote them slightly differently, and slightly differently is how you end up reading Sandbox data while investigating Production. So we wrapped the common queries in team-level skills. A skill knew which environment it was allowed to touch, which fields mattered, and how to phrase the question so the answer came back accurate instead of merely plausible.
Accuracy went up, and the results stopped depending on who was driving. The flow was still manual, a person calling skills in sequence, but the skills now carried the context that used to live in people’s heads.
The standalone agent hits a wall
Then we got ambitious. Early 2026 we tried to stand the whole thing up as an independent agent, something that could run on its own instead of riding inside a person’s laptop IDE, triggered by systems rather than people. We evaluated AWS AgentCore as the host.
I pulled the handbrake myself. The idea was not wrong, but a cloud-hosted agent runtime is built to reach cloud-native, cloud-reachable things, and our world is the opposite of that: two gated environments, a Windows monolith, tools that do not expose themselves to whatever happens to be running in someone else’s cloud. We spent weeks trying to bridge the agent to the environments it needed to see, and the bridge kept collapsing. What made me kill it was not the difficulty but the shape of the work: the scope was creeping into environment plumbing with no MVP in short or medium reach. When a bet stops pointing at something you can ship and starts pointing at an infrastructure project, you kill it. The lesson was expensive and worth it. For this problem, where the agent runs is not a detail. It is the whole game.
The SRE Investigator: one skill to drive the rest
Spring 2026 we stopped trying to escape the editor and doubled down on it...