Agentics: how to build an agent to automate your on call

12 Grams of Carbon

SubscribeSign in

Agentics: how to build an agent to automate your on call<br>A walk through of how you might set up an auto-on-call, why these things can fail, and how to think about agentic loops

theahura<br>Aug 05, 2026

Share

Editor’s note: sign up to attend our next Agentics flagship meetup on August 19th in NYC. We have an amazing lineup of speakers from Nori, Modal, Warp, Opus, Goose, and more!

In a recent Agentics post I wrote about how some AI agents aren’t able to properly handle removing complexity. I used our AI on-call as an example — I talked a bit about how our automatic bug triage agents got worse because the models became too smart for their own good.<br>A lot of people reached out asking about the triage bot!<br>So I figured I’d take a beat to write more about how we set it up and what it can do.

One of the first things that we did with our background agents is try to automate our on-call process. On-call is a natural wedge use case for AI and AI enablement.<br>Very few people like being on call, no one likes being paged at 4am because someone or something brought down prod, and as a result everyone would love to not do it.

Even though they are vital, on-call rotations take up time from things that feel more productive, like pushing new features.

The actual work of an on-call feels very well set up for the LLMs. Most of the on-call’s job is to do a pretty mechanical roll back, then root cause some issue using surrounding context.

Virtually everyone who starts building their own background agent or who uses Nori sees this as their obvious first project. It’s one of the standard ‘dev automation’ flows that are becoming increasingly common. Sales pitch up front: if you were using Nori, setting this up would be a single paragraph in your slack. We built Nori to solve these kinds of dev automations (among other things).

We spent a lot of time making nori self modifying, so that we could just ask it to integrate with something and it would ‘just work.’ As a result we can just describe the pipeline we want and it’ll do it.<br>But if you weren’t using Nori, how would you go about automating something like this?<br>There are three parts:<br>The agent orchestration

The trigger (starting a request)

The context/integrations

Orchestrator: For the most barebones implementation, you don’t need to do anything fancy with long running connections or sockets or ephemeral sessions. Instead, you can spin up a single beefy ec2 instance that’s running a basic express typescript server, and maybe a SQLite table for state.<br>The server logic is simple:<br>Every inbound request should have some uuid.

If the uuid is not registered in the SQLite table, add it. Send the inbound request to something like the Claude SDK, which will return a response and a resume key. Store the resume key in the table, keyed by the request uuid.

If the uuid is already in the SQLite table, grab the resume key and simply call the Claude SDK with the —resume flag.

The Claude SDK manages session/transcript state, so you don’t have to worry about that much at all.<br>Trigger: Ideally, you already have some kind of alerting system built on sentry or grafana or datadog. Most of these things can be configured to call out to a webhook when there are errors.<br>Grab the ec2 IP address (make sure the ec2 is listening on some port), slap nginx on the ec2 instance to route incoming requests to your orchestration server, and then put that IP address into your error reporting system.<br>If you do everything right, you should be able to fire a test webhook result from your reporting software of choice, and watch Claude spin up on your ec2 instance.<br>Context: Of course, that Claude instance won’t be able to do anything, because there’s nothing actually on your ec2 instance besides the server!<br>Since we’re trying to automate our on-call, at minimum we’d need integrations to git (code source) and wherever you dump logs. If we don’t care about security at all, we can set up long lived access tokens in the ec2 env. For GitHub you can create a PAT or an organization GitHub app. Datadog, sentry, grafana, etc all have MCP servers that you can auth into manually, that should persist their tokens.<br>You can also add a few skill files that simply tell the Claude SDK to<br>Clone/pull git when the request comes in

Figure out what the bug is

Post a PR to GitHub with the fix

That’s basically it for the most basic version of this flow. Whenever there’s a bug, the server ingests the webhook data that details the bug (ideally including things like stack traces), and then fires off an agent to figure out what’s going on. The agent then collects a bunch of relevant context, and opens a PR on GitHub that an engineer can look at later.

end to end ‘on-call’ automation<br>The first time this runs, it feels like magic. The second time it runs, it probably won’t run. You’ll likely run into an error because the first bot is still...