FileIndexer - the simplest possible file-based backup to external USB drives
FileIndexer - the simplest possible file-based backup to external USB drives
For the TL;DR crowd:
Are you SURE you need ZFS/BTRFS/snapRAID/git-annex/etc?
python3 indexer.py -n 1 /mnt/usb1 /mnt/usb2 # index your drives, using one core only<br># (to minimize thrashing with spinning drives)
python3 indexer.py /mnt/usb3ssd # index external SSD? use all cores available
python3 indexer.py -l 2 # which files exist on fewer than 2 drives?
python3 indexer.py -v # re-hash the drives and verify every checksum<br># basically, "ZFS scrub".
MIT licensed.
Here is the repository with the standalone Python script (and tests, mypy/pylint/flake8 checks, etc)
I am old enough to remember a time when "backup" meant walking around with<br>a pile of floppy disks, or burning everything onto CD-ROMs (and discovering,<br>years later, that a fair chunk of them had rotted into unreadable coasters).<br>I even used a custom Reed-Solomon<br>to make the process more robust!
...but these days, my approach is very simple; and it grew organically.
Backup, backup, backup - and take it with you
I used to have a ZFS mirror of two USB drives. For quite some time.
And it worked fine; giving me peace of mind, snapshots, and pool scrubs.
And I had all this on an Atomic PI, no less.
So why would you move away from ZFS, the king of filesystems?
(I also know about BTRFS. And mergefs. And snapRAID. And git-annex.<br>Keep reading... Don't rush to comment just yet :-)
Basically, at some point, I realized I don't need any of the complexity involved<br>in any of these solutions. Even the simpler ones, like git-annex, still had more<br>mental overhead than I wanted (not to mention depending on, erm, some knowledge of<br>exotic things like Haskell...)
Why bother with complexity?
...and no, I do NOT need ZFS snapshots on everything .
There were lots of data that I would be fine with just this: multiple copies<br>on multiple external USB drives, with verifiable data integrity .
So I started with an old external USB hard drive, and filled it up with the files<br>I wanted to protect:
# ls -d */<br>3Blue1Brown/ 8-bit-guy/ AI.Donato.Capitella/<br>Electronics/ Mathologer/ Numberphile/<br>Think.Twice/ Thoughty2/ ...<br>...
# ls -d Electronics/*/<br>Electronics/Andreas.Spiess/<br>Electronics/Ben.Eater/<br>Electronics/BigClive/<br>Electronics/Bitluni/<br>Electronics/mitxela/<br>Electronics/MrCarlsonsLab/<br>Electronics/My.Own/<br>Electronics/NandLand/<br>Electronics/Necroware/<br>Electronics/OpenTechLab/<br>Electronics/PhilsLab/<br>...
And when that drive filled up, I used another; splitting the data<br>across the two. When that pair of drives filled up, I added a third drive. And so on.
Not rocket science. Just a growing stable of drives that I have<br>accumulated over the years, put to use. With one, simple rule, and a single number, N:
Anything I keep at all, I keep on at least N drives.
Currently, N=2; but I could easily make that 3 if I needed to, or even higher;<br>just by adding more drives; without caring about balancing ZFS pools, worrying<br>about dealing with advanced filesystems and how to recover when/if their complex<br>datastructures go haywire; with the absolute minimum of control state.
And with just one dependency: Python. Without any external libs of any kind.
Again, let me make this crystal clear: this is not about whole-drive mirroring or RAID-ing.<br>No fancy algorithms. Each drive holds its own contents, but every file<br>exists on at least N of them. If one drive dies (they will always do,<br>eventually - it's not a matter of "if" but "when"), I still have a copy<br>in at least one more drive. And I KNOW that copy is valid and correct.
How? Via checksums stored in a single-file SQLite db .
As simple as it gets.
That's what drove the creation of<br>this standalone Python script.
Danger, Will Robinson
This is a tool I built for me , not a product aimed at the masses.<br>It expects its user to be comfortable with a certain amount of discipline - case in point,<br>in a moment I'll admit to version-controlling the SQLite database itself with git.<br>If that already sounds too fiddly for your taste, it probably is.
But if you're the sort of person who enjoys building glue around your own workflow, read on.
The two serious questions
The simple rule (at least N copies) is still only as good as the ability to answer two questions:
Redundancy - is every file actually present on at least N drives?<br>When I add, edit or move a file on one drive and forget to propagate the<br>change to the other drive that's supposed to carry a copy, its count<br>silently drops from 2 to 1. Nobody notices until the day it matters.
Integrity - is the copy on drive B identical to the one on drive A?<br>Hard drives do silently corrupt bits (bitrot is real, and more common on<br>large, aging drives) - and especially external USB drives can end up with a<br>corrupted file because of transient cabling issues (a bad cable, a tired<br>connector, a glitch during a transfer). Yet the filesystem is happy...