Doing the right thing when things go wrong - The Intercom Blog

Skip to main content

It’s Wednesday at 2pm. A routine deploy goes out. Within minutes, customers cannot open the app. Alarms fire, the pager goes off, and the heartbeat metrics begin to drop.

In some ways, this is close to a best-case incident. It’s daytime, the team is online, and the author of the change is available. There is already a strong hypothesis of what went wrong. And yet, incidents can still spiral without the right process, tools, and shared instincts.

At Fin, shipping is our heartbeat. We have written before about how we prepare for our busiest days, but reliability is not only about handling peak load; it is also about how we respond when something unexpected happens. When customers rely on Fin and Intercom to support their own customers, an incident can interrupt conversations and create uncertainty at exactly the wrong moment. This is why incident management is so important.

The discipline of helping

Incident management might look like Slack channels, status pages, runbooks, and the latest AI SRE tooling, but they’re only the mechanics. It’s really about the discipline of helping people do the right thing when things go wrong. Our process is built around customer impact: detect problems quickly, bring the right people (our engineers and their Agents) together, reduce impact safely and quickly, communicate clearly, and learn from what happened.

One principle shapes all of this: not every P1 is an incident, but every incident should be a team’s P1.

An incident means there’s a time-sensitive disruption, damage, or degradation to the customer experience, and even a well-run one causes real disorder for the customers affected. So the work to resolve it can’t sit in a backlog while we finish something else. When an incident is declared, the responsible team stops what they’re doing and focuses on getting back to a safe state. That might mean leaving a meeting, pulling in engineers with domain context, escalating to the Incident Command team, or rolling back a change before anyone fully understands why it broke. The longer an incident sits unactioned, the more customers it affects, the more trust it erodes, and the harder it becomes to coordinate.

The anatomy of an incident

To understand how we respond to incidents at Fin, we’ll go through how we think about them, as well as the mechanics for how we mitigate and resolve them.

How incidents start

Our incident process usually starts in one of three ways:

1. The signal comes from customers

Support (and increasingly, Operator) sees a pattern in incoming conversations. Within moments we can open a GitHub issue and engage the relevant engineering teams through incident.io, our chosen platform for incident response. Frontline support (both our humans and Fin itself) are often closer to customer pain than any dashboard, making their signal extremely valuable when it comes to speed.

2. The product tells us directly

Beyond the usual suite of alarms, we have heartbeat metrics for core parts of Fin and the Intercom helpdesk. Rather than only monitoring whether our servers are up, these metrics track whether users can do the thing they came to do, like reply to customer queries. Drops in these metrics can tell us if a core workflow is unhealthy, triggering an incident and a page (a loud notification or phone call generated by incident.io) to the owning team. We do not wait for someone to notice manually.

For example, we actively track the number of responses that Fin is able to generate in conversations. If we see a sudden depression in response generation, we know immediately something is wrong and engage our responders to swing into action.

3. Engineers shipping code are actively monitoring product health

At Fin, all engineers are expected to watch their code rolling out from the time it starts deploying to when it lands in production. Having someone on hand monitoring for unexpected changes in metrics (and testing their changes once live) means that the very instant something unexpected happens, we have the ability to restore a healthy state by rolling back production to an earlier version.

It currently takes a little under two minutes for a rollback to land. That means an engineer, or an Agent monitoring the rollout on their behalf, can often spot and remove any undesired behavior in the product before customers start to notice. When it comes to mitigating customer impact, seconds matter, which is why we’re continually pushing for ever-faster rollbacks in our product.

Who does what

Clear roles are one of the biggest differences between a chaotic incident and a well-run one. During an incident, everyone should know what they own:

The engineering lead is the first technical owner. They acknowledge the page, assess impact, and pull in help as needed.

The incident commander coordinates the response when escalation is required, making decisions so technical investigators...