We turned off Pub/Sub and nobody noticed | Blog | incident.io

Open main menu

Products

Solutions

Resources

Customers

Pricing

Careers

Get a demoLogin

incident.io<br>Brand Assets<br>Download .PNG logos<br>Download .SVG logos<br>Download Brand Guidelines<br>Visit brand center

Get a demoLogin

Open main menu

All postsEsc

We turned off Pub/Sub and nobody noticed<br>The message broker<br>The eventadapter<br>Choosing a second broker<br>Dynamic load balancing<br>Fairness weighted scheduling<br>Chaos<br>We are already seeing the benefits

We turned off Pub/Sub and nobody noticed

Patrick Hamann+

Mike Fisher<br>August 11, 2026 — 17 min read

Like many modern software stacks, the incident.io platform is predominantly event-driven. For example, whenever you send us an alert, post a message to our agent on Slack, or update an entry in your Catalog - these are all events which then get enqueued on a message topic, meaning any of our downstream components that are interested in that event can subscribe and react asynchronously, such as sending a push notification or posting a reply to you in Slack.<br>As the platform has grown, so has the number of messages flowing through our system, and thus our dependence on our messaging infrastructure. It’s become mission-critical. At the same time, we've also set stricter availability targets for ourselves, like the 99.99% SLA we now commit to for commit to for Enterprise customers of our On-call product.<br>Until recently, we only used a single provider - Google Cloud Pub/Sub - as our messaging technology. Meaning any blip in Pub/Sub availability meant a blip in our own availability, which isn’t acceptable. So we recently set out on an adventure to make our messaging system more resilient to failure by introducing a secondary message broker to our stack, adding redundancy, and ultimately increasing the availability of our entire platform.<br>The goal was to be able to turn off Pub/Sub with zero customer impact. It turned out to be quite the adventure, but last week, we successfully did exactly that.<br>This is the story of that adventure.<br>The message broker<br>Event-driven systems have many benefits, like allowing us to decouple the rate at which we process messages from the rate of ingestion, or have many different components process the same original customer-initiated event, for example, having a user.created topic, and one system listens for events to send a welcome email and another that sets up their initial database state.<br>At the heart of such a system is typically a “message broker”, which is responsible for receiving messages from the publishing components, storing them, and forwarding them to any interested subscribers. At incident, we’ve historically used Pub/Sub as our message broker of choice; it’s a well-built managed service with a good feature set, and has allowed us to scale with ease over the years.<br>Pub/Sub is solid. Its published SLA is 99.95%, and in practice it has comfortably beaten that for us. The problem was that every event in our platform flowed through one broker, operated by one provider, with no way to route around it. Our message broker had become a single point of failure (SPOF), and this was at tension with our own 99.99% availability targets.<br>And a SPOF is ultimately a question about accountability. When an escalation doesn't fire, "Sorry, Pub/Sub was down" is not an answer we ever want to give a customer. It's our SLA, and it's our job to meet it, whatever our dependencies are doing that day. We already have redundancy in the other layers of our infrastructure, so why should the message broker be treated any differently?<br>The eventadapter<br>When we say we have an event-driven architecture, we’re not exaggerating; we currently have ~800+ individual message topics, 1000+ unique subscriptions to those topics, and are processing ~240 million messages a day *(as of August, 2026).<br>This large number of topics also means there are thousands of call sites in our codebase which interact with events, which can be quite daunting when you want to, say, replace the underlying technology you use for messaging 🫠.<br>Fortunately, we were standing on the shoulders of giants, and the early engineers at incident were wise enough to build code-level abstractions over message publishing and subscribing, which we call the eventadapter. This is a package that exposes some simple but powerful interfaces like:<br>// Publisher is the interface for publishing events.<br>type Publisher interface {<br>Publish(ctx context.Context, ev Eventer, payload []byte) (string, error)

// Subscriber is implemented by all subscribers.<br>type Subscriber interface {<br>Subscribe(ctx context.Context, topicName string, handler SubscribeHandler, params SubscribeParams) func() error

// SubscribeHandler is what consumers of the package implement to<br>// handle a single event.<br>type SubscribeHandler[EV Eventer] func(<br>ctx context.Context, ev *EV, eventMetadata EventMetadata,<br>) error

// Eventer is the interface implemented by all events.<br>type Eventer interface...