Tangled CI runs on microVMs — Tangled's Blog
Our CI service now supports a new engine: microvm. If you're new here, our CI<br>runners — spindles — support a pluggable interface for workflow<br>execution. Until now, our flagship instance at spindle.tangled.sh only<br>supported a Nixery-based<br>engine.
Now, with the new microVM engine, each workflow gets its own little virtual<br>machine, a whole real environment you can do anything inside. It's an upgrade<br>from the Nixery engine while staying fully compatible with it, so if you<br>already have a working Nixery workflow, just change nixery to microvm and<br>it will work!
The interesting part is NixOS images: you configure the machine directly from<br>the workflow file. A few things you can do:
You can bring services up:
services:<br>postgresql:<br>enable: true<br>ensureDatabases: ["spindle-workflow"]<br>ensureUsers:<br>- name: spindle-workflow<br>ensureDBOwnership: true<br>You can build Docker containers:
virtualisation:<br>docker: true<br>steps:<br>- name: "do the thing!"<br>command: docker build ...<br>And you can use non-NixOS images too:
image: alpine<br>steps:<br>- name: install golang<br>command: apk add go<br>It's quick on the second run, too, because it caches aggressively: your<br>dependencies, your services, and any other Nix derivation built inside the<br>microVM get pushed to spindle's Nix cache, so the next workflow that needs them<br>doesn't rebuild those. More on that below.
And like everything else in Tangled, the whole thing is self-hostable, so you<br>can run your own spindle with the microVM engine on your own hardware (see the<br>self-hosting<br>guide). If you want<br>fuller examples, there are recipes in the<br>docs too.
What's in a microVM#
A microVM is just a VM with most of the boring parts removed. There's no BIOS,<br>no PCI bus to probe, no emulated graphics card, none of the slow legacy stuff a<br>normal QEMU machine drags along. You get virtio devices and not much<br>else, which means it boots very quickly and uses very little memory. Right now<br>QEMU is the only runner we support, but the engine is written so that other<br>runners (firecracker for example) can slot in later.
Inside the guest there's a small piece of software we call the agent. Spindle<br>never SSHes in or runs commands "from the outside"; instead the agent dials back<br>to spindle over vsock the moment it boots, says hello, and from then on every<br>step of your workflow is sent to it as a message. The agent runs the command as<br>an unprivileged user, streams stdout and stderr back, and reports the exit code.<br>The host side of this lives in<br>spindle<br>and the guest side is a little Rust binary called<br>shuttle.<br>(shuttle implements<br>agentproto which<br>is the protocol used by spindle. Technically speaking anyone could implement<br>this and, assuming side effects hold, you could have your own agent!)
Two kinds of images#
There are two "flavours" of image you can boot, and they're aimed at fairly<br>different people.
The first is NixOS images . These are the interesting ones: because the whole<br>guest is built with Nix, you can configure it from your workflow file directly.<br>Things like dependencies, services, virtualisation (e.g. Docker),<br>registry and caches are all written right there in the YAML, and the guest<br>agent builds and activates that config before any of your steps run. If we've<br>built that exact base plus config before, spindle can just hand the guest a<br>store path to realize (fetching from whatever cache spindle has configured)<br>instead of rebuilding it, so the second run is quick.
The second is non-NixOS images , which today just means Alpine, but can be<br>anything. You don't get the workflow-level NixOS config here (there's no NixOS<br>to configure), but if Nix happens to exist inside the image, like it does in our<br>Alpine one, it can still talk to the spindle Nix cache just fine.
An example NixOS workflow#
If you've used spindle before, this will look familiar: it's the same manifest<br>you already know, just with a few extra keys that the NixOS image understands.<br>Here's a workflow that needs Postgres to test against and Docker to build an<br>image:
# .tangled/workflows/test.yaml<br>engine: microvm
when:<br>- event: ["push", "pull_request"]<br>branch: ["master"]
image: nixos
dependencies:<br>- go<br>- github:nixos/nixpkgs#hello
registry:<br>nixpkgs: github:nixos/nixpkgs/nixos-unstable
caches:<br>https://nix-community.cachix.org: "nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs="
services:<br>postgresql:<br>enable: true<br>ensureDatabases: ["spindle-workflow"]<br>ensureUsers:<br>- name: spindle-workflow<br>ensureDBOwnership: true
virtualisation:<br>docker: true
steps:<br>- name: run tests<br>environment:<br>PGHOST: /run/postgresql<br>command: |<br>docker build -t app .<br>psql -c "select 1"<br>go test ./...<br>The new keys each do one job:
dependencies are the packages your steps get to use. They go into a<br>mkShellNoCC devshell that every step sources before it runs, so you get the<br>whole stdenv environment (setup hooks like pkg-config wiring up<br>PKG_CONFIG_PATH, etc.) and not just the bare binaries. That means you...