Building (systems) software with Nix - Javier Honduvilla Coto

Something I found incredibly frustrating when I started programming was that I could not spend more time building projects and instead had to fight with errors that seemed like a distraction. These major time sinks didn’t seem more than an inconvenience that were preventing me from making progress and focus on the fun stuff.

Even worse, pretty often what used to work would stop to and it would not be clear why. At the time, I didn’t have the knowledge to even hypothesise what might have happened.

Years later, and despite having levelled up as an engineer, I suppose I tend to feel something similar, especially when working on systems software. The project that built or run yesterday is no longer doing so in the same way. Nowadays I can come up with some possible reasons for this, test some theories, and so on, which is neat, but often this is a big time invesment, and more importantly, it’s work that I don’t find too enjoyable and that takes time from what I want to do: actually building software. The fun stuff.

There you have it, another moving target

Most modern programming languages come with package managers (bundler, cargo, npm, etc) that help fetch dependencies, and among other things they do their best to ensure that they aren’t altered without the user knowing. This is essential for security, and also helps to keep consistency across machines. This is a big improvement from how most people were building software not that long ago.

This is not bullet-proof, though, since there are other sources of variability. The compiler or interpreter version, system-provided libraries, environment variables, etc.

If this weren’t enough, when dealing with systems-level software, we might be using native libraries, and exercise the operating system in ways most other software might never do! The more surface area, the higher the chances of the software we depend changing their behaviour over time.

What about our operating system kernel? From an abstract perspective, it’s another library that code interfaces with. This leads us to some of the fastest evolving parts in the Linux kernel: the BPF subsystem, the fancy new sched_ext schedulers, or io_uring, among others.

When it comes to the whole operating system, a seemingly scheduled routine system update can update part of the toolchain such as the compiler or the linker. A system library that your code uses might change, or even the kernel itself! This can prevent your code from not building, or even worse: changing its behaviour.

One of each, please

I haven’t checked the data but I would find surprising if the BPF ecosystem is not one of the most fast evolving ones in Linux. This is great, because it means that every kernel release comes with exciting new features and fixes, but it can add spiciness to our development (and production!) environments. Besides that, for a basic libbpf based application written in Rust, we depend on:

Rust compiler (rustc) and related tools (cargo);

C compiler(s) (clang). Typically BPF code gets compiled with clang, but most Linux systems use gcc for other native code;

Native dependencies (libelf, zlib, libbpf, libc);

Rust dependencies;

Linux kernel headers (vmlinux.h), and, if generated at build time, the system’s BTF information and bpftool to produce the header file;

Kernel-side of BPF tooling (system call layer, verifier, etc);

Static linker(s);

Dynamic loader (ld.so);

The environment (env);

If I were to bet on this list being complete, I would probably lose. But hopefully it’s enough to illustrate the problem we are dealing with.

We are a single apt/dnf/$YOUR_PACKAGE_MANAGER away from large parts of our software dependencies potentially changing . By default, it’s just one part of the equation that’s fully understandable and under our control: the Rust dependencies (assuming there’s a Cargo.lock in SCM). Perhaps also libbpf, since it’s vendored by default.

If you are unlucky, the burden of hunting for differences is on you. More often that not, figuring out what the previous state of the system was is a herculean task.

Taming the dependency chaos

One of the core ideas behind modern dependency managers is the inclusion of lockfiles. They represent a serialised view of the dependencies that were “locked in” at some point. Among other data, this includes the package location, its precise version, and a checksum to ensure it’s not tampered with or corrupt.

Wouldn’t it be neat if we could apply it for all the items in the list above? Turns out there are more than one implementation of this idea, one them being Nix. Nix is:

a package manager (provides libraries, binaries, etc);

a build system (instructions on how to build code);

a programming language used in the above;

a Linux distribution, NixOS, based on the eponymous package manager and build system;

a bunch of infrastructure such as a build farm, cache, etc to support the above;

Here we...