🤏"/> Running a real Kubernetes cluster inside one microVM — smol machines

← Engineering The newest release of smol machines runs a real Kubernetes cluster — k3d, backed by k3s —<br>inside a single machine. That is a stronger claim than it reads, because Kubernetes is one of<br>the most demanding things you can ask a Linux environment to host: it wants its own kernel, a<br>cgroup hierarchy it controls, a working container runtime, and a specific set of device nodes.<br>This post covers why a microVM can host that stack where a container cannot, and the one device<br>node the kubelet refused to start without.<br>Why Kubernetes-in-a-container fights you<br>The usual way to run a cluster locally — kind, k3d — runs each Kubernetes node as a container.<br>On a shared-kernel host that hits a wall. The kubelet and the container runtime expect to own a<br>kernel: they configure cgroups, load kernel modules, mount filesystems, and program netfilter.<br>Inside a container they are doing all of that against the host’s kernel, which they do not<br>own and do not fully control. You work around it by granting the node container broad<br>privilege, sharing the host’s cgroup hierarchy, and papering over the parts the container<br>abstraction hides.<br>On a laptop the whole arrangement is usually running inside Docker Desktop’s Linux VM in<br>the first place — the VM is doing the real work and the container is a formality on top of it.<br>The isolation you actually get is thin, and the setup is easy to break.<br>A smol machine is a real VM<br>Each smol machine is its own microVM with its own Linux kernel, booted through a thin<br>hypervisor. It has a real devtmpfs, a real cgroup v2 hierarchy, real netfilter, and real<br>overlayfs — not a set of namespaces carved out of a shared host. Containers inside it run with<br>full capabilities and no seccomp filter, which is what a container runtime and a kubelet need in<br>order to set up cgroups, mount filesystems, and manage networking.<br>So a k3d node — itself a container — runs against a kernel the workload actually owns, and<br>nested Kubernetes behaves the way it does on a dedicated machine. The full-capability posture<br>inside the guest is not a cross-tenant risk here, because the boundary between workloads is the<br>hypervisor, not the container runtime: each machine is a separate microVM, so nothing a<br>container does can reach a neighbor’s kernel.<br>The device the kubelet demanded: /dev/kmsg<br>Even with a real kernel underneath, nested Kubernetes did not start. The failure was specific:<br>before the fix<br>failed to create kubelet: open /dev/kmsg: no such file or directory<br>The kubelet opens /dev/kmsg — the kernel log device, character device 1:11 — during<br>startup and refuses to run without it. A bare VM has /dev/kmsg: devtmpfs creates it<br>automatically. But a container’s /dev is not the VM’s /dev.<br>The runtime builds each container’s device tree from an explicit allow-list, and that list<br>did not include kmsg — so the k3d node container came up without it and the kubelet bailed<br>before it did anything.<br>The fix is to add /dev/kmsg to the device set the runtime assembles for every container:<br>the device added to every container's /dev<br>// /dev/kmsg - kernel log device. The kubelet (k3s/k3d, kind, most<br>// Kubernetes-in-Docker) hard-requires it and refuses to start without it.<br>// Bare VMs get it from devtmpfs, but the container /dev is built from this<br>// list, so add it here so nested Kubernetes works out of the box.<br>OciDevice {<br>device_type: "c".to_string(),<br>path: "/dev/kmsg".to_string(),<br>major: 1,<br>minor: 11,<br>file_mode: Some(0o644),<br>uid: Some(0),<br>gid: Some(0),<br>With the node container now holding the device the kubelet expects, the cluster starts. The<br>security reasoning is worth stating plainly, because exposing the kernel ring buffer to a<br>container is normally something you avoid: on a shared host the ring buffer is one buffer for<br>everything on the box, so a container reading it sees its neighbors. Here it does not — each<br>machine is its own microVM, so the kernel log the container reads is its own machine’s.<br>The change ships with a test that asserts /dev/kmsg (1:11) is present in the device<br>set, so a future refactor of the device list cannot silently drop it and break nested<br>Kubernetes again.<br>Where the container storage has to live<br>One more thing has to be right for docker-in-a-machine, and therefore for k3d. Docker’s<br>overlay2 graph driver needs a backing filesystem that supports overlay upper directories, and<br>the machine’s root filesystem is already an overlay. You cannot nest overlay2 on top of<br>another overlay — the kernel disables the index that overlay2 relies on. So the machine puts<br>docker’s storage on the ext4 storage disk instead, binding /storage/docker to /var/lib/docker. containerd and overlay2 then build image layers on a real<br>filesystem, which is what k3s and every image the cluster pulls depend on.<br>The result<br>Put together, standing up a cluster is a handful of commands against one machine:<br>a cluster in one machine<br># One machine, sized for a control...