-->

The SSD That Cried Etcd

The SSD That Cried Etcd<br>A homelab debugging story about why marketing IOPS lie and etcd will absolutely tell you about it<br>August 2, 2026<br>tech, homelab, kubernetes

TL;DR: one flaky node in my k3s cluster turned into an all-day debugging adventure that ended with fio teaching me that SSDs have a lot of feelings about how you write to them.

The setup

Three-node k3s cluster. Two nodes are physical machines with Samsung 870 SATA SSDs. The third node lives as a VM on a Proxmox host, with its virtual disk on an LVM-thin pool backed by a WD Red SA500. All consumer SATA SSDs, all “fast enough,” right?

Right?

The symptom

I asked a Tekton pipeline to build an app. It failed. I asked again. It failed. I looked at the Tekton dashboard to see why. The dashboard was slow.

Slow how? Every API call was making a 300 ms round-trip through my auth proxy. Fine, that’s known. But underneath, the actual API server was answering in five whole seconds . Not fine.

The rabbit hole

Peeling the onion, in order:

Pipelines failing to push to Harbor with 500 internal server error

Harbor DB (Postgres) crashlooping with Input/output error on its own config file

fsck the Longhorn volume, clean, back it goes, minutes later same corruption

The volume’s replicas keep dying on the VM node, but the running replicas are on the healthy physical nodes. Why is that killing them too?

Longhorn replication to the VM node is crawling. Volumes that should sync in a couple of minutes are still going after twenty. Rebuild progress bars barely move. That’s the moment I stop suspecting Kubernetes and start suspecting the disk under the VM.

Kernel is filling dmesg with aborted journal and critical medium error on the VM’s iSCSI targets

The VM’s load average is 35 on 12 cores

At this point I’ve gone from “why is my build failing” to “why is my whole node crying.”

Me, five hours into kubectl describe, convinced I was one grep away from cracking the case. (HIMYM S6E2, where Robin says this about Ted before Ted has done anything remotely detective-like. Yes. That’s the energy.)

The reveal

I fired up fio for a workload matching what etcd actually does: 4 KiB random writes, queue depth 1, with fsync after every one . This is the “hard” workload nobody puts on a datasheet.

Here’s what I actually measured across the drives in my cluster:

Drive<br>Environment<br>Marketed 4K IOPS<br>QD1 fsync<br>QD32 no-sync

Samsung 870 EVO 500GB<br>Physical, bare-metal<br>98,000<br>248 IOPS<br>19,200 IOPS

Samsung 870 EVO 500GB<br>Physical, bare-metal<br>98,000<br>311 IOPS<br>27,400 IOPS

WD Red SA500 1TB<br>Proxmox VM, virtio + LVM-thin<br>85,000<br>134 IOPS<br>6,076 IOPS

The marketing number is real. The drive can do that under specific lab conditions. But it’s measured at queue depth 32 with no sync forced. That’s a benchmark that shows the drive’s theoretical throughput when it can batch and reorder to its heart’s content. It’s the number that fits on a box.

But when your database says “put this on real disk right now, I’ll wait,” the SSD can’t batch. It can’t cache. If it doesn’t have power-loss protection (PLP), it has to actually push the bits into NAND before returning. Consumer SSDs, including the WD Red NAS-branded ones, do that at 100 to 500 IOPS .

Etcd’s docs say it needs 500+ IOPS for sync writes. The “fast enough” NAS SSD sitting in my Proxmox host was, by the numbers, not fast enough to be an etcd master. The 85,000 IOPS on the box was measured at a queue depth and sync mode nothing like what etcd actually asks of a disk.

The compounding factor

The pool that SSD sat in was also shared with another VM doing its own heavy sync-write workload. Two fsync-happy processes queuing behind SSDs with no DRAM cache is a bad time. When Longhorn started rebuilding replicas on top of that, the queue exploded, latencies climbed to 89 ms, and everyone downstream started returning EIO to each other in a chain of increasingly grumpy processes.

The fix

The final bug was simple: the SSD was too slow for the job I was asking it to do.

The fix was to stop asking it. Load rebalanced, cluster went green, all the “mysterious” errors upstream (Harbor 500s, corrupt Postgres configs, failing pipelines) evaporated as soon as the underlying disk stopped saying “I’ll get to it in 89 ms.”

What I learned

SSD performance is a workload story, not a spec

The number on the box is the drive’s best day: big blocks, deep queue, no forced sync. The workload that decides whether a database, etcd, or a distributed storage engine feels snappy or dies is the exact opposite: tiny blocks, queue depth one, sync forced on every write. Nobody prints that number on the box, so you have to measure it yourself. Ten seconds of fio will tell you more than an hour of reading datasheets:

Unsupported language.<br>If that number is below 500 IOPS, the drive is not a good home for anything stateful. Below 200, it’s a good home for cold storage and not much else.

Power-loss protection is the difference between...