Write-Ahead Logs on S3: Compatibility Is Not Performance | Trylle JournalSkip to content<br>Sign InGet started

S3 Express One

Introduction

Running a write-ahead log directly on object storage has become one of the hottest architectural ideas of the past week, thanks to Cursor’s “Git at Any Scale” blog post. The recipe is seductive: take S3’s conditional writes (ETag-based compare-and-swap), serialize your commits through a CAS chain on a log object, and you get a durable, leaderless, infinitely-storable WAL without running a single disk or consensus node yourself.

The catch — and the reason we ran this benchmark — is that S3 API compatibility does not come with the underlying performance characteristics . Every provider we tested speaks the same wire protocol and passes the same correctness gates, but a WAL is a pathological workload for an object store: it is dominated by small, latency-bound, strictly serialized conditional writes. That is precisely the axis on which “S3-compatible” services differ the most, and the axis their marketing pages talk about the least.

One of the first inspired implementations was Waltier by Dan Goodman. We took it further by putting a real benchmarking harness around it, fixing some bugs, and running really heavy stuff for many hours. Our fork and benchmark methodology are live on Trylle. We ran an identical durability-first WAL workload against four providers: AWS S3 , AWS S3 Express One Zone , Tigris , and Cloudflare R2 . Each run pushed the same ~200 GiB payload matrix (4 KiB / 1 MiB / 16 MiB objects) from a single host over a 3 Gbit/s uplink, on the same benchmark commit, with 15 s warmup and 60 s measurement windows per phase. All four providers produced 12,044 acknowledged payloads per run — 36,132 verified payloads total — and every one of them passed correctness fully: exactly one CAS-create winner, exactly one CAS-update winner, converged followers, zero orphan objects. The ranking below is therefore about the latency and cost of meeting the contract, not about whether the contract is met.

TL;DR

Figure 1 — Relative performance index across durable writes, WAL pressure, cold reads, and follower behavior. A reading aid, not a universal vendor grade; cost and host-local cache rates are excluded.

AWS S3 Express One Zone (overall index 92.5) is the uncontested performance leader: ~98 op/s on a single serialized WAL, sub-millisecond-class control path (p50 ≈ 10 ms), and the fastest cold reads and follower propagation. Its asterisk is structural: it is explicitly zonal — a single Availability Zone.

AWS S3 (37.2) is the best multi-AZ all-rounder . Slower per operation, but it keeps scaling through 64 WAL shards (1,109 op/s) with remarkably tight tails.

Tigris (13.3) is the “premium-economy” seat: roughly 60–70 % of S3’s throughput with longer tails, but correct, workable, and with zero-egress-fee economics.

Cloudflare R2 (10.0) passes every correctness gate but its WAL serialization (~5 op/s single-shard), slow small durable writes, and ~1 s follower tail latency make it a poor fit for this CAS-heavy workload today.

WAL Sharding Explained

A single WAL is a single CAS chain: every commit must read-modify-write one object, and the next commit cannot begin until the previous ETag is known. Throughput is therefore bounded by 1 / round-trip-latency — the provider’s control-path latency is the ceiling. If a provider serves conditional PUTs slowly, no amount of client-side parallelism helps.

The single-WAL numbers make this brutally visible:

Provider<br>Single-WAL CAS (op/s)<br>p50 (ms)<br>p99 (ms)

AWS S3 Express One Zone<br>98<br>10<br>12

AWS S3<br>17<br>55<br>127

Tigris<br>12<br>73<br>306

Cloudflare R2<br>175<br>358

Figure 2 — Per-second operations for one serialized WAL across all four providers. The gap between S3 Express and everyone else is the gap in conditional-write latency, nothing more.

Sharding the log

Multi-WAL means sharding the log into 16, 32, or 64 dedicated WAL objects, each carrying its own independent CAS chain. Aggregate op/s rises almost linearly — but the crucial caveat is that ordering is preserved only within a shard . This is acceptable exactly when your domain has a natural unit of serialization. For a Git-hosting exercise it isolates cleanly at the repository level: all refs updates for one repo flow through one shard, and cross-repo ordering was never a requirement anyway. The flip side is equally important: per-repository throughput remains bounded by its WAL shard — sharding raises the aggregate ceiling, not the ceiling of any single hot repo.

Provider<br>1 shard<br>16 shards<br>32 shards<br>64 shards

AWS S3 Express One Zone<br>98<br>1,043<br>1,356<br>1,449

AWS S3<br>17<br>275<br>549<br>1,109

Tigris<br>12<br>184<br>381<br>764

Cloudflare R2<br>91<br>189<br>375

Figure 3 — Aggregate CAS throughput for the 1/16/32/64-shard sweep. Independent per-object chains; ordering remains shard-local.

Figure 4 — Throughput relative to each provider’s own single-WAL baseline. S3, Tigris, and R2 scale near-linearly; S3 Express hits its knee...