Clone latency on FSx for OpenZFS: from 10 minutes to 16 seconds

Clone latency on FSx for OpenZFS: from 10 minutes to 16 seconds — stagDB EngineeringA ZFS clone does not copy data. zfs clone writes a bit of metadata pointing a new dataset at an existing snapshot’s blocks, marks them copy-on-write, and returns. The cost is the same whether the snapshot is 10 MB or 10 TB — you only pay for blocks that later diverge. That property is the reason we picked ZFS: it makes it plausible to give every developer their own full-size copy of production data instead of a gutted 500-row fixture set. AWS offers that property as a managed service. FSx for OpenZFS runs real OpenZFS, and its CSI driver provisions a volume from a snapshot with CopyStrategy=CLONE, sharing blocks rather than copying them. On paper: a developer asks for a clone of a production-size Postgres database and gets one in seconds, no matter how big it is. We built that, and the block sharing behaved exactly as documented. The latency did not match the underlying primitive, and characterizing the gap is what the rest of this post is about. Where the latency actually comes from Our first milestone spike was deliberately small: EKS 1.30, one FSx for OpenZFS SINGLE_AZ_1 filesystem (256 GiB, 128 MB/s), and a ~10 GB golden snapshot — pgbench at scale 700, 10.98 GB logical. We cloned it into three developer namespaces at once, booted a CloudNativePG Postgres on each, and measured clone-to-Ready. Three concurrent clones came back at 110 s, 177 s, and 238 s — a p50 of 177 s against a gate of 120 s, and a p95 of 238 s. The storage half passed convincingly. Amplification for the golden plus three clones was 1.107× — 12.16 GB used where four independent full copies would have been about 44 GB. The blocks were genuinely shared, the cloned data directories byte-identical to the golden, and they replayed WAL cleanly after we killed the pods. Copy-on-write worked as advertised. So the latency was not data movement, and we confirmed it was size-independent — nothing was being read or written. What we were waiting on was the managed control plane in front of ZFS. FSx’s CreateVolume API takes roughly 75 s for a plain provisioned volume and about 2 minutes when creating from a snapshot . That is the floor, and it is independent of the data involved. The shape of those three numbers is the more consequential detail. 110, 177, 238 — rising with concurrency, spaced about a minute apart. That is not three requests each taking two minutes in parallel; it is the signature of a queue. We ran the test again at ten concurrent clones to characterize how the queue behaves at higher fan-out: Concurrent clonesClone-to-Ready (s)K=3110 / 177 / 238K=10129 / 198 / 269 / 319 / 381 / 428 / 490 / 550 / 611, plus one that had not finished at 10 minThe inter-completion gap is a near-constant ~60 s . FSx CreateVolume serializes at roughly one clone per minute , so the Nth developer to ask for a database waits about N minutes. At K=10 the p50 was 381 s and the p95 was 611 s . That characteristic is a poor match for the demand pattern a development-environment tool sees. Developer demand for clones is not uniform; it arrives in a burst when a team starts work in the morning, which is precisely the regime in which a serialized API performs worst. Ten concurrent developers is a small team rather than a stress test, and under this API the last request in line waits about ten minutes for an operation the storage layer itself performs in milliseconds. An unplanned natural control We built the fix (below), redeployed, and ran a time-to-first-clone benchmark on a separate live cluster, against a smaller golden — prod-pg16-v3, 86 MB and 750k rows — measuring each Instance’s creationTimestamp to its Ready condition. The size difference does not matter here, and that is the point: the claim path moves no data at all, so its latency is independent of golden size, which is what makes these numbers comparable to the 10 GB measurements above. Nine of the ten claims were fast, and one was not: a single claim raced a momentarily-empty pool, found nothing warm, and fell through to the on-demand path — a real CreateVolume against FSx. It returned in 600.3 s . We could not have designed a cleaner control. Same cluster, same golden snapshot, same operator build, same measurement — one request took the old path and every other took the new one, roughly 37× apart , with no confounding variable except which code path it took. It also reproduced the spike’s finding on a live cluster, months later, without our intending it: 600.3 s is what “serializes at ~1/min” predicts for a request landing behind a queue of pool refills. We excluded it from the percentiles — it has no pool-claim timestamp pair, so it is not a sample of what we were measuring — but it is the most informative number in the run. The fix: moving the API call out of the hot path If the bottleneck is a control-plane API you do not control, you cannot make it faster. What you can do is...

Clone latency on FSx for OpenZFS: from 10 minutes to 16 seconds

Related Articles

(no title)

Scientists reverse brain aging, with a nasal spray

NHS apologises and admits Palantir have access to identifiable patient data

AI has torched the market for junior programmers

Al Vigier: Canada's AI strategy shouldn't include secret Palantir bills