Expand GELI Encrypted Bhyve VM ZFS Disk | 𝚟𝚎𝚛𝚖𝚊𝚍𝚎𝚗

The ZFS partition/volume expansion is generally not an ultra hard task … but needs one step at a time approach.

Today we will follow them one by one to expand GELI encrypted ZFS disk.

There is only one potential problem – and I faced that recently with one of the other VMs I manage.

If FreeBSD VM guest version is older – I have tested with 14.1-RELEASE for example – then even these will not help:

guest # camcontrol rescan all<br>guest # camcontrol reprobe ${RESIZED_DISK}

You will have to reboot.

But with FreeBSD 15.0-RELEASE VM guest the new disk size was picked up quite fast and no reboot needed.

The list of steps to be taken:

Increasing VM disk on FreeBSD Bhyve hypervisor for that VM.

Potential reboot(8) for some older FreeBSD releases.

Recover partition table with gpart(8) tool.

Resizing partition gpart(8) layer.

The GELI layer will ‘notice’ automatically.

ZFS resizing.

So lets get to work – first – resize VM Disk on FreeBSD Bhyve hypervisor host.

Its a ZFS ZVOL for the record this time.

The FreeBSD host also has vm-bhyve installed and used to manage Bhyve guests.

host # zfs list | grep vm-guest<br>zroot/vm/vm-guest 572G 358G 2.79G /vm/vm-guest<br>zroot/vm/vm-guest/disk0 569G 384G 251G -

We will resize disk for vm-guest VM now.

The guest VM can be running – resizing can be done online.

host # vm list | grep -e NAME -e vm-guest<br>NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE<br>vm-guest default bhyveload 4 8G - No Running (69638)

Lets check where is the guest VM disk.

host # vm info vm-guest | grep -A 32 virtual-disk<br>virtual-disk<br>number: 0<br>device-type: zvol<br>emulation: nvme<br>options: -<br>system-path: /dev/zvol/zroot/vm/vm-guest/disk0<br>bytes-size: 268435456000 (250.000G)<br>bytes-used: 269879066624 (251.344G)

… and that we have enough space in ZFS pool for expansion.

host # zpool list zroot<br>NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT<br>zroot 1.67T 1.15T 530G - - 62% 69% 1.00x ONLINE -

Now – below is the actual command to expand ZFS ZVOL.

host # zfs set volsize=350g zroot/vm/vm-guest/disk0

We should now see increased size.

host # vm info vm-guest | grep -A 32 virtual-disk<br>virtual-disk<br>number: 0<br>device-type: zvol<br>emulation: nvme<br>options: -<br>system-path: /dev/zvol/zroot/vm/vm-guest/disk0<br>bytes-size: 375809638400 (350.000G )<br>bytes-used: 269879066624 (251.344G)

If the disk would be a file on filesystem – then we would resize it this way instead.

same as with ZVOL we just specify the new size.

host # truncate -s 350g /vm/vm-guest/disk.img

Now next step is to recover the partition table with gpart(8) tool.

guest # gpart show<br>=> 40 734003120 nda0 GPT (350G) [CORRUPT]<br>40 1024 1 freebsd-boot (512K)<br>1064 984 - free - (492K)<br>2048 16777216 2 freebsd-swap (8.0G)<br>16779264 20971520 3 freebsd-swap (10G)<br>37750784 4194304 4 freebsd-zfs (2.0G)<br>41945088 482340864 5 freebsd-zfs (230G)

guest # gpart recover nda0<br>nda0 recovered

Now next step is to expand last partition with gpart(8) command.

We will add additional 100 GB of space.

guest # gpart show<br>=> 40 734003120 nda0 GPT (350G)<br>40 1024 1 freebsd-boot (512K)<br>1064 984 - free - (492K)<br>2048 16777216 2 freebsd-swap (8.0G)<br>16779264 20971520 3 freebsd-swap (10G)<br>37750784 4194304 4 freebsd-zfs (2.0G)<br>41945088 482340864 5 freebsd-zfs (230G)<br>524285952 209717208 - free - (100G)

Below command will use all that space for last partition on the disk.

guest # gpart resize -i 5 nda0<br>nda0p5 resized

guest # gpart show<br>=> 40 734003120 nda0 GPT (350G)<br>40 1024 1 freebsd-boot (512K)<br>1064 984 - free - (492K)<br>2048 16777216 2 freebsd-swap (8.0G)<br>16779264 20971520 3 freebsd-swap (10G)<br>37750784 4194304 4 freebsd-zfs (2.0G)<br>41945088 692058072 5 freebsd-zfs (330G)

GELI layer should resize automatically.

guest # geli status<br>Name Status Components<br>nda0p4.eli ACTIVE nda0p4<br>nda0p5.eli ACTIVE nda0p5<br>gpt/swap0.eli ACTIVE gpt/swap0

guest # dmesg | grep -E 'GEOM.*size.*changed'<br>GEOM_ELI: Device nda0p5.eli size changed from 219518685184 to 314963316736.

guest # geli list nda0p5.eli<br>Geom name: nda0p5.eli<br>State: ACTIVE<br>EncryptionAlgorithm: AES-XTS<br>KeyLength: 256<br>AuthenticationAlgorithm: HMAC/SHA256<br>Crypto: accelerated software<br>Version: 7<br>UsedKey: 1<br>Flags: BOOT, AUTH, NODELETE, AUTORESIZE<br>KeysAllocated: 660<br>KeysTotal: 660<br>Providers:<br>1. Name: nda0p5.eli<br>Mediasize: 314963316736 (293G)<br>Sectorsize: 4096<br>Mode: r1w1e1<br>Consumers:<br>1. Name: nda0p5<br>Mediasize: 354333732864 (330G)<br>Sectorsize: 512<br>Stripesize: 0<br>Stripeoffset: 21475885056<br>Mode: r1w1e1

Now we need to expand ZFS.

First we need to enable autoexpand for that ZFS pool.

guest # zpool get all zroot | grep autoexpand<br>zroot autoexpand off default

guest # zpool set autoexpand=on zroot

Next check zpool status command to get exact names of the ZFS pool and VDEV to resize.

guest # gpart show<br>=> 40 734003120 nda0 GPT (350G)<br>40 1024 1 freebsd-boot (512K)<br>1064 984 - free - (492K)<br>2048 16777216 2 freebsd-swap (8.0G)<br>16779264 20971520 3 freebsd-swap (10G)<br>37750784 4194304...