The principles of extreme fault tolerance — PlanetScaleIntroducing Database Traffic Control™: resource budgets for your Postgres query traffic.Learn more
Navigation<br>Blog|Engineering<br>Table of contents «Close »Table of contents<br>Principles<br>Architecture<br>Processes<br>Failure modesNon-query-path failures<br>Cloud provider failures<br>PlanetScale-induced failures
PlanetScale Postgres is the fastest way to run Postgres in the cloud. Plans start at just $5 per month.<br>Learn more
Get the RSS feed
The principles of extreme fault tolerance<br>Max Englander | July 3, 2025<br>PlanetScale is fast and reliable. Our speed is the best in the cloud due to our shared nothing architecture that enables us to utilize local storage instead of network-attached storage. Our fault tolerance is built on top of principles, processes, and architectures that are easy to understand, but require painstaking work to do well.<br>We have talked about our speed a lot. Let's talk about why we are reliable.<br>Principles<br>Our principles are neither new nor radical. You may find them obvious. Even so, they are foundational for our fault tolerance. Every capability we add, and every optimization we make, is either bound by or born from these principles.<br>Isolation<br>Systems are made from parts that are as physically and logically independent as possible.<br>Failures in one part do not cascade into failures in an independent part.<br>Parts in the critical path have as few dependencies as possible.<br>Redundancy<br>Each part is copied multiple times, so if one part fails, its copies continue doing its work.<br>Copies of each part are themselves isolated from each other.<br>Static stability<br>When something fails, continue operating with the last known good state.<br>Overprovision so a failing part's work can be absorbed by its copies.<br>Architecture<br>Our architecture emerges from the principles above.<br>Control plane<br>Provides database management functionality. Database creation, billing, etc.<br>Composed of parts which are redundant, spread across multiple cloud availability zones.<br>Less critical than the data plane, and so has more dependencies.<br>E.g. uses a PlanetScale database to store customer and database metadata.<br>Data plane<br>Stores database data and serves customer application queries.<br>Composed of a query routing layer and database clusters.<br>Each of these parts are both regionally and zonally redundant and isolated.<br>The most critical plane, with fewer dependencies than the control plane.<br>Does not depend on the control plane.<br>Database clusters<br>Composed of a primary instance and a minimum of two replicas.<br>Each instance is composed of a VM and storage residing in the data plane.<br>Instances evenly distributed across three availability zones.<br>Automatic failovers from primaries to healthy replicas in response to failures.<br>Customers may optionally run copies in read-only regions.<br>Enterprise customers may optionally promote read-only regions to primary.<br>Extremely critical. Extremely few dependences.<br>Processes<br>Within this architecture, we apply processes that reinforce our systems' overall fault tolerance.<br>Always be Failing Over<br>Very mature ability to fail over from a failing database primary to a healthy replica.<br>Exercise this ability every week on every customer database as we ship changes.<br>In the event of failing hardware or a network failure - fairly common in a big system running on the cloud - we automatically and aggressively fail over.<br>Query buffering minimizes or eliminates disruption during failovers.<br>Synchronous replication<br>MySQL semi-sync replication, Postgres synchronous commits.<br>Commits stored durably on at least one replica before primary sends acknowledgment to the client.<br>Enables us to treat replicas as potential primaries, and fail over to them immediately as needed.<br>Progressive delivery<br>Data plane changes are shipped gradually to progressively critical environments.<br>Database cluster config and binary changes are shipped database by database using feature flags<br>Release channels allow us to ship changes to dev branches first, and to wait a week or more before shipping those same changes to production branches<br>Minimizes the impact of our own mistakes on our customers.<br>Failure modes<br>How adherence to the principles, architecture, and processes above enable us to tolerate a variety of failure modes.<br>Non-query-path failures<br>Because our query path has extremely few dependencies, failures outside of the query path do not impact our customers' application queries.<br>As an example, a hypothetical failure in one of our cloud providers' Docker registry services might impact our ability to create new database instances, but will not impact existing instances' ability to serve queries or store data.<br>Likewise, failures, even total failure, of our control plane would impact our customer's ability to change their database cluster's settings, but would not impact that cluster's query service.<br>Cloud provider failures<br>We run on AWS and GCP, which can and do fail in many different ways.<br>Instance<br>If a failure...