🎉 Mounted — bitter-FS better with Claude
- mloduchowski.com
Tomasz Mloduchowski
Scientist by Heart, Engineer by Trade, Entrepreneur by Choice.
Blog
Contact
🎉 Mounted — bitter-FS better with Claude
June 6, 2026
A 41 TB filesystem, two kernels that didn't know about each other, and the ~320 KB of writes that brought it all back.
TL;DR: An unlocked iSCSI LUN got mounted by two OS instances at once - a condition that went unnoticed for ten months. Every standard BTRFS recovery tool failed. Claude (Opus 4.8, in a Claude Code session, as root) reconstructed what happened from first principles, found an intact-but-unreferenced second transaction history on the disk, hand-patched the superblocks to point at it, rebuilt the 19 metadata leaves that were genuinely destroyed, and remounted the filesystem:mount -o ro;no rescue flags, no errors. Data lost: zero .<br>Full disclosure - this blog post, 90% written by Claude (but 100% read by me first, I decided not to edit it much). I was just a spectator.<br>Repo with the tools that Claude wrote during this ~4h session - please use at your own risk, read them first (I did skim them) - but you might find them all useful. Claude's raw draft of this post included for comparison too.<br>https://gitlab.defensiblelogic.com/pub/rebtrfs<br>The setup<br>Somewhere in my lab there is a 41 TB BTRFS filesystem. It lives inside a LUKS2 container, on an iSCSI LUN exported by my NAS, and holds years of disk images, backups, and "I'll sort this out later" archives. 40 TiB used, zstd-compressed, about 7.5 million files. It's the backup tier — the place other machines get copied to. A backup of the backup does exist, but it's a few weeks out of date and *glacially* slow to restore from.<br>In February it died.<br>Cause of death: the LUN wasn't locked, and I assumed it was mounted in exactly one place. It was mounted in two. The dual-mount condition persisted for ten months — one instance doing the real work, the other sitting idle but alive, each kernel COW-allocating from what it believed was free space, neither knowing the other existed.<br>The standard tool parade - mount -o ro,usebackuproot,nologreplay, btrfs-find-root, btrfs rescue chunk-recover, btrfs rescue super-recover, btrfs rescue zero-log, btrfs restore — all failed. Some instantly, some after hours of grinding. Corrupted beyond the tools' ability. The standard advice at this point is "restore from backup," and that option was technically on the table; at the cost of a multi-day restore and the last few weeks of writes. Surgery first, surrender later.<br>So: re-sync a raw image of the LUN onto a local RAID-0 scratch pair (about a day over 10 GigE), open a Claude Code session as root, and state the problem more or less as: "you have the machine, you have a copy, you have ~17 TB of free scratch space. Recover it."
First moves (in which nothing is trusted)
Claude's opening sequence:<br>lsblk / mdstat / LVM recon — locate the copy: a 41 T logical volume, crypto_LUKS inside.<br>Read root's .bash_history and reconstruct every recovery step already tried on the original — including the pv /dev/dm-0 line proving the LV held a raw, block-for-block image of the damaged LUKS volume.<br>Ask the human for exactly one thing: type the LUKS passphrase. No, not in a session, directly as luksOpen /dev/mapper/RAID0-secure_repair s<br>blockdev --setro /dev/mapper/s — kernel-level write protection on the decrypted device, before anything else gets a chance to touch it.<br>Build a dm-snapshot overlay backed by a 512 G copy-on-write volume.<br>That last step is the one to steal: a dm-snapshot overlay turns a one-shot recovery into unlimited retries. All experiments hit the overlay; the underlying copy never changes; every destructive idea becomes a reversible experiment. A failed repair costs a `dmsetup remove` instead of another day of resync.<br>lvcreate -n cow0 -L 512G RAID0<br>dmsetup create sr_work --table \<br>"0 $(blockdev --getsz /dev/mapper/s) snapshot /dev/mapper/s /dev/RAID0/cow0 P 32"<br>The smoking gun<br>The superblock itself was fine — checksum valid, a healthy-looking 40 TiB filesystem at generation 24312. The rescue mount died at the very first dereference:
BTRFS error: level verify failed on logical 29245440 mirror 1 wanted 1 found 0<br>BTRFS error: level verify failed on logical 29245440 mirror 2 wanted 1 found 0<br>BTRFS error: failed to read chunk root<br>The super says: chunk root at logical 29245440, level 1, generation 24287. The node actually there (in both DUP copies) is level 0, and:<br>parent transid verify failed on 29245440 wanted 24287 found 24860<br>Generation 24860 . The superblock's world ends at generation 24312 — yet here is metadata from 548 transactions later.<br>That one line is the whole disaster. There were two divergent transaction histories interleaved on the disk. Git users: picture a repo where someone force-pushed every ref to a corrupted commit, while the real history sits intact in the object store. This isn't data recovery — it's finding the good commits and...