Let’s Build PlanetScale From Scratch: Infrastructure | Onat Mercan’s Blog

Homescale, PlanetScale at home.

I worked for a database tools company many years ago and was lucky enough to build a few database cloning tools before PlanetScale was cool.

Those tools were never destined to be as successful as PlanetScale, but they had their uses.

The idea behind those tools was simple: isolate the storage layer from compute and use the best storage technology for cloning the database files.

This idea came back to me a while ago while I was tweeting about an interview I had with an F1 team that was looking for a Rust dev with hands-on Ceph experience. Mentioning Ceph reminded me of a failed product attempt I was part of, and I jokingly told a friend that I was going to build Homescale, PlanetScale at home.

Homescale

I am building Homescale at github.com/homescale-dev/homescale.

Homescale creates writable database instances and point-in-time branches from immutable snapshots without copying the full database.

I borrowed Docker’s image and container model to represent the database state: A database image is an immutable starting point. A database container is a writable clone created from that image. A branch is another container created from the current state of an existing container.

The CLI I have in mind looks like this:

homescale image create --engine postgres postgres-base<br>homescale container create --image postgres-base dev-db<br>homescale container connect dev-db

homescale branch create --container dev-db feature-login<br>homescale container connect feature-login

The examples in this series use Postgres, but Homescale’s storage model is database agnostic . The image, container, and branch model can apply to any database engine that keeps its durable state on a filesystem backed by a block device and can be prepared for a recoverable snapshot. Postgres is the first engine I plan to support and gives me something concrete to use while building the storage and orchestration layers.

Engine-specific behavior belongs behind an adapter. That adapter initializes an image, starts the database process, exposes connection details, and prepares the database for a snapshot when necessary. Homescale handles the lifecycle around it: volumes, immutable states, writable clones, workloads, and lineage.

The word branch describes the relationship between the containers. After running these commands, dev-db and feature-login are both writable database containers. feature-login simply started from the state of dev-db at the moment the branch was created.

Underneath, the lineage looks like this:

flowchart LR<br>Image[/postgres-baseimage/]<br>Dev[dev-dbwritable]<br>State[/read-onlystate/]<br>Feature[feature-loginwritable]

Image -->|clone| Dev<br>Dev -->|snapshot| State<br>State -->|clone| Feature

The image is already immutable, so Homescale can create dev-db from it directly. Branching from a writable container needs an intermediate state. Homescale first captures dev-db at a point in time, then creates feature-login from that state.

Creating feature-login cannot mean copying dev-db byte for byte. A 100 GB database would require another 100 GB of storage before the branch could start.

The branch should initially share its data with the captured state of dev-db. Only the data that changes afterward should require additional storage.

That requires branching to happen below the database process, in the storage layer.

Separating storage from compute

Separating database storage from compute is not a new idea. Standard Amazon RDS engines store database and log files on EBS volumes. Google Cloud SQL runs the database process in a VM with attached network block storage, using Persistent Disk or Hyperdisk depending on the machine series. The database still sees a normal block device, but its durable data is not tied to the compute host’s local disk.

Aurora, AlloyDB, and Azure SQL Hyperscale take this further. Their compute instances connect to shared or distributed storage systems designed specifically for the database. Compute instances can be replaced or scaled without creating a complete copy of the data for each instance.

Homescale is closer to the first model. A database process reads and writes a filesystem on what looks like a block device. I only need the storage behind that device to have a lifecycle independent of the process using it.

flowchart TD<br>Database[Database process]<br>Filesystem[Filesystem]<br>Device[Block device]<br>Storage[(Persistent storage)]

Database -->|file reads and writes| Filesystem<br>Filesystem -->|block I/O| Device<br>Device --> Storage

That boundary lets Homescale create the storage before starting a database process and keep it after the process stops. More importantly, it lets branching happen in the storage layer without requiring the database engine to implement branching itself.

The two Postgres processes know nothing about their shared history. Each sees its own writable block device. Another supported engine...