Before AI Ships Code, Show Me the Receipts | PagerDuty

Explore our top new Generally Available and Early Access features.

View Product Drops

PagerDuty /

Engineering Blog /

Before AI Ships Code, Show Me the Receipts

Engineering Blog

All Blogs

Engineering

Product

Integrations

Best Practices

Announcements

Before AI Ships Code, Show Me the Receipts

by PagerDuty Engineering

July 8, 2026<br>| 10 min read

Now that AI agents can write our code, when should we let them ship it?

Much of an engineer’s week is consumed by small, routine changes. If agents can handle those changes safely on their own, engineers get that time back to focus on problems that require judgment, creativity, and context.

But letting a change ship without a human looking at it means giving up control. You are deciding that a whole kind of work no longer needs your eyes on it, today or any day after, until something tells you otherwise.

At PagerDuty, reliability is non-negotiable. The bar to ship code is high for humans and higher still for AI. Clearing that bar takes trust, and trust takes evidence. So for the past few months, we have been building the system that gathers the evidence and turns it into a decision: which work an agent has earned the right to ship, and which still needs a person.

This post is about that evidence: how we gather it, how we read it, and what we do when it changes.

Not all work is the same

We started by defining ten work categories, ordered by the level of risk they tend to carry. To build them, we analyzed a few thousand pull requests merged over the previous 6 months and identified the kind(s) of change they contained. From these, we developed categories ranging from documentation, configuration changes, and dependency updates at the lower-risk end to new features and schema migrations at the higher-risk end. Subsequent PRs are then classified by an LLM and evaluated against human labels, with the classifier’s performance becoming a monitoring signal for our system.

But two pull requests in the same work category can still have very different stakes. So we also give every change a second label: an impact tier from one to five. The work category captures what the change does; the impact tier captures how far that particular change reaches and how easy it would be to undo. A tier-five change might be small, mechanical, and confined to one file. Whereas a tier-one change might span services, cut across system boundaries, or require an architectural decision.

With this classification system in place, every unit of work handled by an agent can be assigned to one of fifty work-and-impact groups. That gives us a consistent way to compare like with like and measure how the agent performs on each kind of work.

Once we could compare similar kinds of work, we wanted to know which of them were worth automating. A task that takes an hour but happens once a year is not worth much attention; a task that takes ten minutes but happens every day adds up to real time.

Choosing what to automate first

We estimated the active implementation time based on commit activity. We grouped commits made close together into working sessions, added a small allowance for the work before the first commit, and summed those sessions for each pull request. This gives us a conservative estimate of active implementation time. It captures the periods when someone was working and committing code, but not the time spent researching, discussing the problem, or thinking through an approach. The true effort is therefore likely higher than our estimate.

Even with that conservative estimate, the pattern was clear. Around 97% of the pull requests represented less than a full workday of active implementation, and more than half represented less than an hour. Small individual tasks like these are exactly where volume matters: something that takes ten minutes is easy to dismiss on its own, but multiplied across every engineer, every week, it becomes one of the largest pools of toil we have.

Small, bounded changes are also the easiest kind of work to reason about, but size alone does not tell us whether a change is safe to hand off. A one-line change to a billing path can carry more risk than a much larger documentation update. Effort multiplied by frequency tells us where the toil is; the work-and-impact grouping from the previous section tells us where the risk is.

Together, they tell us which groups are worth automating first, and the next question is whether the agent has earned enough trust to act on that.

The limits of approval

One way to judge whether an agent could handle a category of work autonomously was to look at how often its pull requests were approved without changes. So for each work-and-impact group, we track its changeless approval rate: the proportion of agent-generated pull requests that a human approved without asking for revisions. In principle, that gives us a simple, comparable signal of how reliably the agent...