Running a Virtual Machine on a Cloud Box That Can't Run Virtual Machines — frankchiarulli.com

←all posts<br>Running a Virtual Machine on a Cloud Box That Can't Run Virtual Machines<br>June 29, 2026

I wanted to run other people’s code on my servers without trusting it, on cloud boxes cheap enough to have a lot of them. Those two goals are at odds, and getting around that pulled me deeper into how virtual machines work than I ever planned to go.

Running Code You Don’t Trust

The code I want to run isn’t mine. It might be an AI agent’s output, or a stranger’s contest submission. So I put a wall around it, and a bad program can only break its own little world.

The cheap wall is a container. Docker starts instantly and costs almost nothing, but every container shares the host’s one operating system kernel. That’s fine if you trust what’s running. If you don’t, it’s a thin wall: one bug in that shared kernel and the code is loose on the host.

A virtual machine is the real wall. It’s a whole simulated computer with its own kernel running on top of yours. Code inside can’t even see your real machine; take over its kernel and you’ve taken over a fake computer. The catch is that VMs are heavy and slow to boot.

microVMs like Firecracker and Cloud Hypervisor fix that. They boot in milliseconds and use barely any memory, with a real VM’s isolation. It’s what AWS Lambda runs your code in. That’s what I wanted: a fresh, throwaway VM for every bit of untrusted code. Then I tried to run one on a cheap cloud box.

The /dev/kvm Problem

A VM needs the CPU’s help to run at any speed. Modern chips have virtualization built into the silicon for it. On Linux, KVM is what uses it, and you reach KVM through one file, /dev/kvm. No file, no VMs, and a microVM won’t even start without it.

On hardware you own, the file is just there. The trouble with a cheap cloud box is that the box is itself a VM. Your $5 Hetzner instance isn’t a computer, it’s a VM on Hetzner’s hardware, so a microVM on it would be a VM inside a VM. That’s nested virtualization, and it only works if your provider passes the CPU’s virtualization down to you. Budget tiers almost never do. Hetzner Cloud doesn’t, on any tier. No nested virtualization, no /dev/kvm, no microVM.

The usual answer is to rent bare metal, where /dev/kvm is real. But bare metal is expensive and inflexible, and I wanted the opposite: lots of small, cheap boxes. So I kept looking.

PVM

PVM (Pagetable Virtual Machine) does in software what a VM normally offloads to those missing CPU features. It does that work itself, so it runs on a box that never had the hardware to begin with. It comes from Ant Group, who have had a lot of success running it in production, so it’s definitely battle hardened enough for my tiny project.

Unfortunately PVM requires a fork of the Linux kernel, so you run need to run custom builds. Two of them:

A host kernel for the cloud box. Boot on it and /dev/kvm appears, done in software. The box now looks like it can run VMs.

A guest kernel for the VM. PVM’s faked hardware is odd enough that a stock kernel won’t boot on it, so the VM boots this one instead.

I got it working end to end on a €4/month Hetzner cx23 with no hardware virtualization at all:

Hetzner cx23 (no hardware virtualization)<br>→ PVM host kernel makes /dev/kvm appear<br>→ Cloud Hypervisor boots a microVM with the PVM guest kernel<br>→ the guest boots all the way to a working system

Building the Kernels Once

Booting it once is easy. The annoying part is that both kernels have to be compile to add PVM, and a Linux kernel is huge: about an hour for the host, twenty minutes for the guest on my low-end CI box (Github Actions).

To avoid building the kernel each time, I build each kernel once in CI, and let every box download the result. That’s what nix-pvm is. It defines both kernels in Nix; CI rebuilds them every week and pushes them to a cache (Cachix). Each box pulls the prebuilt kernel.

Here’s what the actual config I run on my contest fleet looks like:

1. Trust the cache and pull in the module

In your flake.nix, add the cache so the kernels download instead of compiling, pull in nix-pvm, and import its NixOS module on the host:

# Pull the prebuilt PVM kernels from the cache instead of compiling them.<br># Trusted users honor this; otherwise pass --accept-flake-config or put these<br># two lines in the builder's nix.conf.<br>nixConfig = {<br>extra-substituters = [ "https://nix-pvm.cachix.org" ];<br>extra-trusted-public-keys = [<br>"nix-pvm.cachix.org-1:Nf9cU+dJIq7XpVPE9SMD4UWeXqO1u0U4m6ApnN3CtRg="<br>];<br>};

inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-25.11";<br>inputs.nix-pvm.url = "github:fcjr/nix-pvm";

outputs = { nixpkgs, nix-pvm, ... }: {<br>nixosConfigurations.my-host = nixpkgs.lib.nixosSystem {<br>system = "x86_64-linux";<br>modules = [<br>nix-pvm.nixosModules.default # PVM host kernel + pti=off + kvm-pvm + cache<br>./configuration.nix<br>];<br>};<br>};<br>That sets the host kernel, the options PVM needs (pti=off, autoloading kvm-pvm), and the...