Keep the monolith, but split the workloads | Blog | incident.io
Open main menu
Products
Solutions
Resources
Customers
Pricing
Careers
Get a demoLogin
incident.io<br>Brand Assets<br>Download .PNG logos<br>Download .SVG logos<br>Download Brand Guidelines<br>Visit brand center
Get a demoLogin
Open main menu
All postsEsc
Keep the monolith, but split the workloads<br>A wild outage appears!<br>Understanding reliability in monolithic systems vs. microservices<br>Rule 1: Never mix workloads<br>Rule 2: Apply guardrails to your monolith for database efficiency<br>Keeping a monolithic architecture can still be a smart choice for scaling
Keep the monolith, but split the workloads
Lawrence Jones<br>April 12, 2023 — 10 min read
I’m a big fan of monolithic architectures. Writing code is hard enough without each function call requiring a network request, and that’s before considering the investment in observability, RPC frameworks, and dev environments you need to be productive in a microservice environment.<br>But having spent half a decade stewarding a Ruby monolith from 20 to 200 engineers and watched its modest 10GB Postgres database grow beyond 5TB, there’s a point where the pain outweighs the benefits.<br>This post is about splitting your workloads, a technique that can significantly reduce that pain, costs little and can be applied early as you scale. Something that, if executed well, can let you enjoy that sweet monolithic goodness for that much longer.<br>Let’s dive in!<br>A wild outage appears!<br>Back in November 2022, we had an outage we affectionately called “Intermittent downtime from repeated crashes”.<br>Probably the first genuinely major outage we’ve faced, it resulted in our app repeatedly crashing over a period of 32 minutes. Pretty stressful stuff, even for responders who spend their entire day jobs building incident tooling.<br>While the post-mortem goes into detail, the gist of the issue was:<br>We run our app as a Go monolith in Heroku, using Heroku Postgres as a database, and GCP Pub/Sub as an async message queue.<br>Our application runs several replicas of a single binary running web, worker, and cron threads.<br>When a bad Pub/Sub message was pulled into the binary, an unhandled panic would crash the entire app, meaning web, workers, and crons all died.<br>Well, that sucks and seems easily avoidable. If only we’d built everything as microservices, we’d only have crashed the service responsible for that message, right?<br>Understanding reliability in monolithic systems vs. microservices<br>The most common reason teams opt for a microservice architecture tends to be for reliability or scalability, often used interchangeably.<br>This means that:<br>The blast radius of problems – such as the bad Pub/Sub message we saw above – is limited to the service it runs in, often allowing the service to degrade gracefully (continue serving most requests, failing only for certain features).<br>Each microservice can manage its own resources such as setting limits for CPU or memory that can be scaled to whatever that service needs at the time. This prevents a bad code path from consuming all of a limited resource and impacting other code, as it might in a monolithic app.<br>Microservices certainly solve these problems but come with a huge amount of associated baggage (distributed system problems, RPC frameworks, etc). If we want the benefits of microservices without the baggage, we’ll need some alternative solutions.<br>Rule 1: Never mix workloads<br>First, we should apply the cardinal rule of running monoliths, which is: never mix your workloads.<br>For our incident.io app, we have three key workloads:<br>Web servers that handle incoming requests.<br>Pub/Sub subscribers that process async work.<br>Cron jobs that fire on a schedule.<br>We were breaking this rule by running all of this code inside of the same process (as in, literally the same linux process). By mixing workloads we left ourselves open to:<br>Bad code in a specific part of the codebase bringing down the whole app, as in our November incident.<br>If we deployed a Pub/Sub subscriber that was CPU heavy (maybe compressing Slack images, or a badly written loop that spun indefinitely) we’d impact the entire app, causing all web/worker/cron activity to slow to a halt. CPU in that process is a limited resource and by consuming 90% of it, we’d leave only 10% left for the other work.<br>The same day as our incident occurred, we split our app into separate tiers of deployment for each workload type. This meant creating three separate dyno tiers in Heroku, which for those unfamiliar with Heroku just means three independent deployments of the app processing only its own type of workload.
You might ask if we’re doing this, then why not go the whole way and have separate microservices?<br>The answer is that this split preserves all the benefits of the monolith while solving the problems we presented above. Every deployment is running the same code, using the same Docker image and environment variables, the only thing that differs is the command we run to start...