How We Built the First Zero-Disk, S3-Tiered Storage Engine for SQLite - Rivet

Skip to main content

GitHubSign In

Blog<br>Every Rivet Actor gets its own isolated SQLite database . In practice, that means running millions of databases that start instantly, cost nothing but storage while idle, and never touch a local disk.

TL;DR: Rivet’s SQLite storage engine now ships with:

S3 tiered storage : cold data automatically offloads to S3-compatible object storage

Point-in-time recovery : recover any database to a point in time

Massively improved SQLite performance : commits return in low milliseconds, Actors start instantly with no restore, and hot pages read at near-in-memory speed

Why build yet another SQLite storage engine?

When designing SQLite support for Actors, we looked at a lot of off-the-shelf options, and none of them quite fit the bill. It came down to eight main constraints that caused us to build a new SQLite storage layer from the ground up.

The eight constraints

Speed:

Fast, durable writes : How fast data can be written to the SQLite database directly impacts user-facing performance. Writes must return in low milliseconds while still maintaining at least triple redundancy.

Instant startup with no local data : Actors open their SQLite database on startup, so opening must be instant to make sure there is no delay when Actors crash, upgrade, or migrate.

Simplicity:

Zero-disk compute for elastic scaling : Machines running Actors hold no volumes and no local state, so compute can scale up and down freely without moving data around.

Self-hostable, no proprietary dependencies : There must be no dependencies on proprietary cloud services, and it must be easily self-hostable in a range of environments, from local development to highly scalable distributed environments.

No new database to operate, no stateful components : Rivet is frequently self-hosted, so it runs exclusively on mainstream databases teams already know how to operate and adds no new stateful components.

Scale:

Bottomless storage per SQLite db : One of the best-known limitations of Durable Objects and D1 is their 10 GB database cap, meaning that if you have a customer’s database that hits that cap, they have a complete outage. We chose this as a constraint to avoid for Rivet Actors.

Billions of databases per cluster : Rivet Actors get created at very high frequency, for everything from as coarse as agent sessions to as granular as every user that visits a site. Creating a database must be instant, per-database overhead must be near zero, and a single cluster must comfortably hold billions of mostly idle databases.

An idle database costs no resources : When an Actor goes to sleep, the only resources its database takes should be disk or S3, not live resources like you get with a Postgres server.

The existing SQLite engine landscape

SQLite already has a rich ecosystem of replication and hosting options, but each category falls short of these constraints:

CategoryProjectsApproachWhy it doesn’t fitSingle-primary replicationLitestream, LiteFSOne machine owns the writable database and streams changes out, to object storage (Litestream) or to read replicas (LiteFS)Commits are durable on only one machine before async replication, so a lost machine loses recent writes. Read replicas can fetch pages from S3 on demand, but the writable primary must hold the full database on its local disk and fully restore it before taking writes (violates 1, 2, 3, 6)Full-copy clustersrqlite, dqlite, cr-sqlite, MarmotEvery node holds the full database, kept consistent with Raft (rqlite, dqlite) or merged eventually with CRDTs (cr-sqlite) or gossip replication (Marmot)Resident processes and a full copy per node cap the database at machine size and make billions of idle databases impossible (violates 2, 3, 6, 7, 8). The eventual-consistency options can also lose acknowledged writes on merge or node loss (violates 1)Hosted server-side SQLiteTurso (libSQL)A hosted SQLite server that your app queries over the network. Its “diskless” architecture describes the server fleet: stateless SQLite servers caching over object storageQueries run against a remote server instead of in your process. Getting SQLite back in-process requires embedded replicas, which hold the full database on local disk and forward writes to the server (violates 2, 3, 6). When Rivet says zero-disk, it means the opposite: SQLite runs in-process and the database file never exists on the machine’s disk. Turso is an exciting project, but still early for us to bet all of the Rivet developers on it for nowObject-storage LSMSlateDBLSM tree that writes directly to object storage. Not SQLite itself, but it could sit under a SQLite VFSDurable commits wait on an object-storage PUT, putting writes in the 50 to 100 ms range on S3 Standard (violates 1). A single writer owns each database, and per-database manifest and compaction overhead makes billions of small databases impractical (violates 7). It’s...