Every fast write moves work somewhere else<br>Every fast write moves work somewhere else<br>August 8, 2026<br>18 min read

Contents<br>How a client PUT reaches a local SSD<br>1 flush can cover many writes<br>Keeping a write beyond the database host<br>Replicating the WAL adds coordination<br>Fast reads require an index and cleanup<br>Big O shows growth, not durability latency

Every storage engine has to decide what must finish before it tells a client that a write succeeded. The quickest answer is to return after copying the bytes into memory. A local durable write waits for fdatasync() on an SSD in the database host. Keeping the write after that host disappears means waiting for a network volume, an object store, or several database servers to save their own copies.<br>Those choices move latency and durability together because returning after memory is fast, but a machine crash can lose the write. Waiting for a local SSD survives a process or kernel crash, but not the loss of that device or host, while remote storage or several database servers can survive more failures by putting network and copy time into every write.<br>One system&rsquo;s fdatasync() waits for 1 SSD attached to the database host, while another exposes a network volume through the same NVMe interface and waits for a remote storage service. The syscall name is the same, but the latency and the failures the data survives are not.<br>Newer storage designs built around object storage make this choice especially interesting to me because many put immutable sorted files in object storage, use a host-local NVMe SSD for a write-ahead log or cache, and organize data with a log-structured merge tree (LSM) layout. Object storage can be a fantastic primitive to build around because the storage service can own the durable copies while compute comes and goes, and writing new immutable files avoids small random updates to shared disk pages.<br>After a successful object PUT, the storage service owns meeting its advertised durability for those bytes. The database still has to decide which version is current, record that decision so other machines can see it, make reads fast, remove old versions, and recover after a crash. A local write-ahead log can make writes faster, but then the database has to decide whether losing that local copy is acceptable or whether another copy must exist before success.<br>I use client PUT for the key-value request sent to the database, and object PUT for the HTTP request sent to object storage. Once I kept those writes separate, it became easier to follow 1 client PUT through memory, a local SSD, remote storage, and a durable majority of database servers. So when I see a very fast number for an operation, I want to know which operation pays for it, what can still be lost after success, and how much unfinished cleanup the system can tolerate.<br>I found it useful to start with the costs I would want from an append-only key-value store.<br>OperationWork before successGETO(key bytes + returned bytes), 1 index lookup and 1 read for the valuePUTO(key bytes + payload bytes), 1 pass over the payload to hash it, 1 WAL append, and 1 fdatasync() shared with other writesDELETEO(key bytes), append 1 delete recordThese costs look attractive because a normal request never scans every retained object or walks the full write history, and a larger value costs more than a smaller value in an unsurprising way.<br>Those small request costs are possible because an index already maps each key to its latest value for GET, several PUT requests can share a device flush, and DELETE records that a value is dead without removing the old bytes immediately. The PUT is the easiest place to see where that work moves, so I started there.<br>How a client PUT reaches a local SSD<br>Here I mean a host-local NVMe SSD, which is attached to the same host as the database rather than reached through a remote storage service. Once a write is synced, it can survive a process or kernel crash, but losing the SSD or the host can still lose the write.<br>NVMe names the interface used to send commands to a device, but it does not tell us where the storage lives or which failures the data survives. A remote block volume can also appear as an NVMe device, even though each write crosses a network and a storage service keeps the copies. An NVMe latency number is incomplete without its acknowledgment point. Did the timer stop after copying bytes into memory, after syncing 1 local SSD, or after a remote volume acknowledged the write?<br>A local WAL is tempting because the latency gap can be large. AWS describes S3 Express One Zone as having consistent single-digit millisecond read and write request latency and being up to 10x faster than S3 Standard, while Turso measured a 4 KB PUT at 6.4 ms on average and 7 ms at p99. A local fdatasync() measured at 1 ms would make the local path about 6x faster, while 0.1 ms would make it about 64x faster. A 50x difference can be a real benchmark result, but it is not a property of every NVMe device and...