ZFS Scrubs with no ECC (what could possibly go wrong?) - TrueNAS General - TrueNAS Community Forums

ZFS Scrubs with no ECC (what could possibly go wrong?)

TrueNAS General

ZFS,<br>Hardware

thomas-hn

January 13, 2025, 9:06am

Hello,

I know this ECC vs. Non-ECC topic it is a very sensitive one here in the forum.

But I would like to understand how a scrub exactly works on ZFS to make my own judgement if ECC is really needed or not (I had a debate with a colleague some days ago, I argueg for ECC, he argued against).

Let’s assume ZFS does a scrub over all the data on a pool and at least one of the Non-ECC RAM modules has some permanently faulty bits.

My current understanding is for a scrub:

ZFS reads a block of data (including a checksum) from the pool.

This information is stored into RAM.

ZFS calculates the checksum over the stored data in RAM.

ZFS compares the newly calculated checksum with the existing one in RAM (the one read from the pool).

If both checksums are fine, nothing to be done. Otherwise, ZFS reads the same data from another (redundant) location, copies this data into RAM, does some verification via checksum and replaces the faulty data with this redundant copy.

So, what happens if something goes wrong:

Steps 3) and 4): If ZFS calculates a wrong checksum over correct data (because of faulty bits in RAM), it assumes the data is faulty and tries to correct it. As long as we have some redundancy, it should be fine.

Step 5): If ZFS detects a problem in step 3) and 4) and tries to correct the data, it has to copy the redundant data into RAM, before it can repair the faulty blocks. What will happen if this copy goes wrong, because of faulty RAM bits. For my understanding it could happen that ZFS detects faulty blocks on the pool which cannot be repaired, because the redundant data is also faulty. In this case data could get lost (as ZFS thinks the data is faulty), even that the data on the drives might be fine and only the RAM has an issue.

Is this understanding correct? Please correct me if I’m wrong.

Thansk a lot in advance,

Thomas

Memes! TrueNAS, ZFS, and related | (Share your own!)

jro

January 13, 2025, 1:46pm

thomas-hn:

In this case data could get lost (as ZFS thinks the data is faulty), even that the data on the drives might be fine and only the RAM has an issue.

You’d get ZFS errors complaining about checksums not matching, but because the checksums don’t match, ZFS won’t touch anything on the disk. You’d presumably get a whole boatload of these checksum errors across all of your disks because the same faulty RAM is re-used during the scrub process. At that point, you’d probably conclude that you’ve got other weird hardware issues rather than all of your disks failing simultaneously.

ZFS without ECC is arguably safer than most other filesystem without ECC. Still, ECC isn’t that expensive and might save you a headache so it’s probably worth the investment.

4 Likes

Arwen

January 13, 2025, 5:16pm

The scenario is commonly referred to the ZFS Scrub Of Death . Though generally described with non-permanent RAM errors.

It is in someways a Myth of Epic proportions.

But, my own opinion is that under extremely odd and unusual cases, non-ECC RAM could corrupt a pool during a scrub. My own take is that at some point, we will have enough data stored on ZFS using non-ECC RAM that statistically The ZFS Scrub Of Death will happen. Certainly not a regular occurrence.

On the other hand, we have seen many ZFS pool corruption problems here in the TrueNAS forums. Some are clearly un-related to non-ECC RAM:

Using TrueNAS as a VM, without proper safe guards, (aka ProxMox & TN accessing the same pool, at the same time)

Using gamer system boards that default to over-clocking, (where bit flips might crash a game… but in TrueNAS corrupt a pool).

Using a striped pool

Using SMR disks

Using wide RAID-Z1 with very large disks

ZFS software bug, (and yes, their have been such)

On occasion, we have not found an obvious or even odd theoretical cause of pool corruption for some cases. So it is entirely possible that they got the pool corruption from non-ECC RAM. @jro’s comment about ZFS without ECC RAM applies.

Because of such, “RAID is not a backup”.

3 Likes

etorix

January 13, 2025, 6:38pm

We’ve had ONE actual report of a Scrub of Death:

ECC memory vs Non-ECC memory - Poll! TrueNAS General

I used to come down on the side of ECC being optional, but had a RAM stick fail in my parents server that would flip a certain bit so consistently that it not only caused scrubs to report errors, it actually caused ZFS to recalculate the parity data-- incorrectly-- and write it back to disk.<br>This caused permanent corruption of dozens of files before I realized what was happening, even with a RAIDZ2. Luckily I had backups of most of it, and only ended up losing 1 file that had not been backed up…

Mythical, certainly. Blown out of proportion with its actual probablity, most likely.

But the probability...