Benchmarking OpenZFS vs EXT4 for my NAS | Heitor's log

Benchmarking OpenZFS vs EXT4 for my NAS

Jun 08, 2026

:: tags:<br>#Linux<br>#NAS<br>#ZFS<br>#benchmark<br>#ext4<br>#homelab<br>#server<br>#sysadmin

Reading time: 8 minutes

I’m building a NAS for myself and<br>I was curious to see how OpenZFS would perform against ext4. My server will<br>have full disk encryption on RAID1 and I couldn’t find a benchmark on a similar<br>setup.

My hardware:

Intel i3-14100 CPU

2x32 GB DDR4 2666 MT/s RAM

1 TB NVMe for the OS

2x2 TB HDD in RAID1 for the main storage pool

I’m running NixOS with Linux Kernel 6.18.33.

The filesystem configuration is different for the two filesystems I compared.<br>For ext4, I had the filesystem on top of LVM on top of LUKS on top of software<br>RAID1. For ZFS, I used the native functionality for RAID and encryption, and<br>also full disk compression on.

What I want to see is how these two filesystems compare for my use case:

backups: large (~10s GiB), streaming writes and reads

thumbnail/sidecar/metadata generation: small size (~10s KiB), random I/O.<br>This is to simulate mainly Home Assistant SQLite access, Immich small file<br>generation, and Darktable sidecar file access.

photography ingestion: medium size (~10s MiB), random I/O. This box is where<br>I’ll store photos from my SLR camera, both RAW and processed.

random usage: I’m not running a single service at a given time. It is very<br>possible that an automated backup happens while I browse Immich while Home<br>Assistant updates its state.

I have a ton of RAM in this machine, and it definitely alters the results of<br>the tests. To reduce the chances of I/O being served from RAM instead of from<br>the HDDs, I allocated 50 GiB of zeros in RAM:

# My /tmp is a tmpfs<br>dd if=/dev/zero of=/tmp/crap bs=1024 count=$((50 * 1024 * 1024)) status=progress<br>For the ZFS tests, I capped ZFS ARC to 4 GiB:

echo $((4 * 1024 * 1024 * 1024)) | sudo tee /sys/module/zfs/parameters/zfs_arc_max<br>My OS is on a different ZFS pool, but there’s no way to configure ARC per pool,<br>only system-wide.

I’m not sure it makes a difference, but I also turned off swap, which lives in<br>my NVMe.

These changes are not bulletproof but help making the comparison a bit more<br>fair.

Setting up storage

This is how I prepared the disks for these benchmarks. For ext4:

# Setup RAID1<br>mdadm --create --verbose --level=1 --raid-devices=2 /dev/md0 /dev/sda1 /dev/sdb1<br># Wait until /proc/mdstat reports 100%

# Setup LUKS<br>cryptsetup luksFormat --type luks2 /dev/md0<br>cryptsetup open /dev/md0 nas_crypt

# Setup LVM<br>pvcreate /dev/mapper/nas_crypt<br>vgcreate nas_vg /dev/mapper/nas_crypt<br>lvcreate -l 100%FREE -n nas_data nas_vg<br>lvreduce -L -256M nas_vg/nas_data

# Setup ext4 fs<br>mkfs.ext4 /dev/nas_vg/nas_data

# Mount it<br>mkdir -p /mnt/nas<br>mount -o noatime /dev/nas_vg/nas_data /mnt/nas<br>chown --recursive h:users /mnt/nas<br>ZFS setup requires fewer steps, but has more knobs:

# Setup ZFS pool with encryption, compression, mirror<br>zpool create -O encryption=on -O keyformat=passphrase -O keylocation=prompt \<br>-O compression=on \<br>-O mountpoint=none \<br>-O xattr=sa -O acltype=posixacl -O atime=off -o ashift=12 \<br>main mirror /dev/disk/by-partlabel/mainTwoTB1 /dev/disk/by-partlabel/mainTwoTB2

# Create filesystem and mount it<br>zfs create -o mountpoint=legacy main/data<br>mount -t zfs main/data /mnt/nas<br>I first set up the ext4 system, then I ran the tests. After that, I formatted<br>the HDDs, setup ZFS, and repeated the same tests.

When I say “ext4” I mean the complete stack: software RAID1, LUKS encryption,<br>LVM, and ext4. Each one of these adds a layer in the VFS.

Theoretical values

Western Digital claims this HDD model’s transfer rate is up to 175 MB/s (~167<br>MiB/s). This puts a theoretical max write speed at 167 MiB/s and 334 MiB/s for<br>read (2x factor from RAID1). Let’s see if my system can get close to these<br>numbers.

Backup benchmark

For these tests, a backup workload is a task that writes or reads large files,<br>sequentially. There is only a single process at a given time. I used fio for<br>this benchmark:

# Write test: simulate creating a backup<br>fio --name=backup-write --rw=write --bs=1M --size=20G --numjobs=1 --direct=1 \<br>--filename=/mnt/nas/fio-test --ioengine=libaio

# Read test, drop caches first: simulate restoring a backup<br>echo 3 | sudo tee /proc/sys/vm/drop_caches<br>fio --name=backup-read --rw=read --bs=1M --size=20G --numjobs=1 --direct=1 \<br>--filename=/mnt/nas/fio-test --ioengine=libaio<br>I ran each test twice for each filesystem. The first one as a warm-up round and<br>the second one was the actual benchmark. There’s no need to run this test N<br>times and average them: fio already averages the results over the entire<br>test. One thing to notice is both runs gave consistent results.

As a baseline, the average CPU usage while idle is ~0.5 %.

ZFSext4<br>Sequential write172 MiB/s121 MiB/s<br>Avg latency write2.8 μs8.3 ms<br>P99 latency write7.5 μs61 ms<br>Peak CPU usage (w)14.5 %4.7 %<br>Sequential read182 MiB/s196 MiB/s<br>Avg latency read4.4 μs5.1 ms<br>P99 latency read7.9 μs5.9 ms<br>Peak...