P[drive failure]

i want to keep my data for my entire life. modern hardware should be advanced<br>enough to support this, but the real world invariably rears its ugly head<br>whenever discussions of "forever" come up, anywhere. nothing is forever.<br>\(P[\text{the world ends tomorrow}] > 0\), after all. my entire life (and<br>possibly the lives of another generation or two) should be sufficient for my<br>purposes.

note: this is a redone version of an analysis that i did for work a while<br>back. i was operating ~200 TB of networked storage at the time for both ml use<br>(storing models) and government-inspected record-keeping, hence the desire to be<br>able to prove some measure of reliablility. unfortunately i left before i could<br>properly implement any of it. :(

background

i'm building a nas.<br>"i would never have guessed that, nausicaa!"

not just a jbod, though: as the introductory paragraph asserted, i would like to<br>be able to rely on it. i intend for it to be a repository of beloved photos,<br>memorialized text conversations, hoarded computer backups, and irreplaceable tax<br>documents, which means that it should have an acceptably low chance of<br>forgetting anything — say 1 in 104 — over its working lifespan. that working<br>lifespan is also, most likely, the rest of my lifespan, which is about half a<br>century at the time of this writing.

this necessitates redundancy: i can't rely on a single storage device because it<br>will fail (\(P \approx 1\)) in 50 years. because i'm impatient (and i've done a<br>similar analysis before), i decided to go ahead and order four 8 TB drives, then<br>write this post.<br>if you're from the future, hello! this was in 2026, when<br>ai was king and storage was extremely expensive.

all i need to do is figure out, for the most reliable architecture i can build,<br>just how reliable it really is.

architecture

my planned architecture for the nas is this, for low \(P[\text{data loss}]\):

set up the four drives in two groups of two, each btrfs RAID1,

store one group at my father's house several states away, and

automatically synchronize the local copy with the remote copy.

really important note if you're doing this yourself: use raid1 (or<br>raid1c3/4) instead of dup if you can, as the latter will<br>dutifully duplicate blocks but can put both copies on the same device, which<br>doesn't protect against drive failure. i'd only use dup if… i'm not sure,<br>actually. maybe email me if you have a use case.

the nice thing about btrfs<br>as well as zfs, if that's more your speed. next-gen filesystems<br>generally have this as a feature, because we've all started to accept that<br>storage is fallible and data on a computer matters to us.<br>for this is that it checksums each data block<br>(which appear to be 16 KiB per the docs) and will auto-repair corrupt blocks on<br>read, so long as there is a good copy of that block somewhere! this means that<br>if we have drives in raid1 — each block exists on two separate devices — we<br>protect against any single drive failure and against any single block<br>corruption.

smart monitoring & failure policy

i'd be a fool to ignore the built-in monitoring systems that the drives have if<br>i care about redundancy, so: in the event of a nas problem, either a full-on<br>drive failure or a rapid accumulation of smart errors, i will replace the<br>offending drive and resilver. btrfs replace with -r is the way to do this from<br>raid redundancy data alone, and would be what i would use if a drive fully<br>failed.

where your bits go when they die

there are two<br>well, there are at least two ways hard drives can make you<br>sad. i'm not claiming omniscience.<br>ways hard drives can make you sad:

they can die, or

they can forget what you told them.

if a drive dies, you lose everything it held. the only option is to replace it<br>and rebuild that data from redundant copies (which you hopefully have). if a<br>drive has an error on read, you lose all data in that disk sector (typically 512<br>B). this is known in the literature as a latent sector error, or lse; it's also<br>known in datasheets as an unrecoverable read error, or ure.<br>a note on ures:<br>drive datasheets will give a number for these like "1 in 1015", but this<br>number notably lacks units most of the time. claude and i agree that the most<br>likely unit for this is "sector errors per bit read" — which incidentally<br>makes drive manufacturers look as good as possible.<br>this is far from death,<br>even for that sector: drives actively remap bad sectors while in operation, so a<br>drive which experiences an lse will continue to function (but see below).

to figure out the appropriate mathematical model for \(P[data loss]\), we have<br>to understand the shape of these two failure modes. on to reading!

disk failure

when people talk about reliability engineering (read: equipment failure at large<br>enough scale that you measure via statistics), they speak of the "bathtub<br>curve:"

the classic bathtub curve. this is wrong for drive failure rates.

the idea is that most hardware failure is due to either manufacturing/qa...