Garage S3: LMDB Corruption Post-Mortem | Filippo BertoSearch

Garage S3: LMDB Corruption Post-Mortem<br>Complete loss of Garage metadata after a power outage on Btrfs<br>Filippo Berto<br>2026-07-22

9 min read

Yesterday I was migrating my data from an rclone-mounted bucket to a proper OpenCloud setup backed by my home Garage S3 cluster. Things were going smoothly: files copying over, users importing, the new stack finally taking shape.<br>Then the power went out.

When the server came back, Garage refused to start. The LMDB metadata database was completely corrupted, rendering the cluster inoperable. The block data (781k objects, ~57 GB) remained intact on a Btrfs RAID1, but without the LMDB index the mapping from S3 keys to block hashes was lost. The data in my personal bucket and the one used by OpenCloud as its storage backend: gone.<br>It could have been worse. My most important data lives outside S3, backed by restic to a different target, and that backup pipeline was unaffected. Still, losing two buckets stings.<br>That said, I didn&rsquo;t choose S3 on a whim. This bucket needed to be accessed from both my homelab and several VPSs hosting public services, a mix of edge and cloud. S3 gives me a single consistent API and authentication layer everywhere, and Garage&rsquo;s S3 gateway exposes it without vendor lock-in (unlike, cough, certain other open-source S3 implementations). A local filesystem wouldn&rsquo;t solve that. I like the option to add nodes for replication down the road, but a crash like this makes you question whether the complexity is worth it for a single-node deployment.<br>TL;DR : LMDB is known to be vulnerable to corruption after an unclean shutdown. Garage provides a built-in snapshot mechanism (metadata_auto_snapshot_interval) exactly for this scenario. I hadn&rsquo;t enabled it. I also had no Btrfs snapshots on the metadata partition. If you run Garage on LMDB, enable snapshots today .

System Layout<br>Metadata LMDB : single-device Btrfs partition (/dev/nvme0n1p2, 238 GiB NVMe, 98% full)<br>Block data : Btrfs RAID1 (/mnt/raid/, 5.5 TiB, HDDs)<br>Garage version : v2.x, db_engine = "lmdb"<br>COW : enabled on both partitions<br>Snapshots/backups : none for the metadata directory<br>Timeline<br>T-unknown : Power outage. Server loses power mid-operation. Garage was writing to the LMDB database.<br>T=0 : Server boots back up. Garage refuses to start. Attempting to inspect the LMDB file:<br>$ mdb_stat -e /var/lib/garage/meta/db.lmdb/data.mdb<br>mdb_stat: src/mdb.c:...: Assertion 'IS_BRANCH(mc->mc_pg[mc->mc_top])' failed.<br>Aborted

$ mdb_copy -c /var/lib/garage/meta/db.lmdb/data.mdb /tmp/recovery.mdb<br>mdb_copy: src/mdb.c:...: Assertion 'IS_BRANCH(mc->mc_pg[mc->mc_top])' failed.<br>Aborted

The LMDB b-tree is structurally gone. I sat there staring at the terminal for a good minute. No error message to decode, no partial recovery: just a dead database and 57 GB of unlabeled blocks on the RAID. Garage&rsquo;s own docs warn about this:<br>LMDB is prone to database corruption after an unclean shutdown (e.g. a process kill or a power outage).

T+1h : Garage service migrated to a fresh cluster on a different machine. Block data copied from the RAID. OpenCloud is re-targeted.<br>T+2h : Btrfs COW recovery attempted. All 10 candidate tree roots (generations 2477737–2477773) contain the same corrupt extents.<br>T+4h : LMDB raw page scan. 956k pages, all flags=0x0000, lower=1, upper≈24. No valid root page. The database was effectively erased.<br>T+24h : Recovery abandoned. Data re-seeded from scratch where possible.<br>Root Cause<br>A power outage interrupted Garage mid-write on the LMDB metadata database. LMDB uses mmap&rsquo;d I/O and, by default, operates with MDB_NOMETASYNC and MDB_NOSYNC (Garage sets metadata_fsync = false by default). Without fsync, a power loss during a transaction commit leaves the b-tree in an inconsistent state: torn pages, missing branches, zero-filled root pointers.<br>This is a well-documented limitation of LMDB, not a Garage bug. The recovery path Garage documents is:<br>Replication (requires replication_factor ≥ 2 and multiple nodes): delete the corrupt DB, restart, and run garage repair -a --yes tables to resync from peers.<br>Metadata snapshots (metadata_auto_snapshot_interval): restore from a Garage-generated snapshot, then repair.<br>Filesystem snapshots (Btrfs/ZFS): restore from an old snapshot of the metadata directory.<br>I had none of these. I didn&rsquo;t know about the snapshot feature; I hadn&rsquo;t read the docs carefully enough. If I had, this post wouldn&rsquo;t exist.<br>Recovery Attempts<br>Btrfs COW recovery<br>The LMDB file had COW enabled. Each page write creates a new extent at a different physical location; old page data remains on disk until overwritten.<br>$ btrfs-find-root /dev/nvme0n1p2<br>Well block 174354268160 (gen: 2477773)<br>Well block 174354300928 (gen: 2477772)<br>...<br>Found tree root at 174435041280 gen 2477773<br>Found tree root at 174433533952 gen 2477772<br>Found tree root at 172099076096 gen 2477739<br>...

10 candidate tree roots identified,...