Civboot

Build Systems Discussion

My last blog post got

some interest<br>on the permacomputing mailing list, especially as it relates to designing a<br>personal or permacomputing build system. I thought I would give a quick<br>overview of the ../lua/civ build system as well as explore those talked<br>about by the participants, as well as ideas that have been brewing in my<br>brain about a civ package manager.

For reference, here are the build systems presented. For the most part<br>I am not going to give targeted criticism but rather lay out what I believe<br>a good build system needs to accomplish. Most of my criticism will be<br>leveled at bazel and Make:<br>https://bazel.build Google's build system which can, in<br>real-time, build all software at Google incrementally or from scratch.

../lua/civ - the build system for this project.

Kartik's approach

LM Build System

clib which seems like<br>mostly a collection of small C libraries and scripts to integrate<br>them directly in your source code.

Make - which needs no introduction.

What is a build system or package manager anyway?

Before we get into specifics, it's important to go back to the ground truth:<br>what does a build system do ?

I once had a fresh-from-college engineer ask me about some problem<br>related to CAD files he was dealing with. He spent 5-10 minutes trying<br>to explain all the minutae of his particular CAD<br>tooling, how certain pieces of representation are difficult to script<br>and how he got around it, etc etc. I know almost nothing of CAD, so was<br>very little help with the technical details but I did ask him one question:

What are your inputs and what are your outputs?

With this one question he completely reframed the problem. He quickly<br>began to untangle the web of complexity he had formed and started to<br>discuss technical details of how things were cleaner, simpler, easier<br>than he had realized. Knowing nothing of CAD, I could only smile -<br>but that was enough.

So what is a build system? Civ's answer is that it is a program which<br>takes as input source code (files), binaries (compilers), and configuration<br>(flags, config files) and outputs files, binaries and configuration<br>according to a specific folder structure, which can then be executed<br>on a target platform.

So the first principle of a good build system is this:<br>all inputs and outputs of each phase of the build must somewhere be explicit.

How does a good build system turn inputs to outputs?

The second question is perhaps the more important one: how does a user<br>specify what they want to build, and how do different pieces of a build<br>(libraries, scripts, config files, etc) end up being knitted into<br>the whole?

This is where a program like Make fails miserably - Make is a effectively<br>a collection of shell scripts in an obtuse, platform-specific syntax.<br>It defines each "rule" as a collection of inputs and outputs<br>while failing to offer an overarching vision on how to stitch them together.

This is the next principle of a good build system:<br>It must be posible to programatically configure the inputs and outputs to<br>adapt to different target systems.

In essense, a good build system gives the correct layer of abstraction<br>between the source code and the program actually built - as simple as possible<br>but no simpler.

Civ's answer is this: relegate the decisions of where files go and<br>how to build them to something a user can configure centrally - effectively<br>a build library - which has a minimal and well-defined API but<br>who's internals can be swapped out for different platforms. In civ, this is called<br>the sys hub.<br>For instance cc.luk<br>(luk is lua configuration files, a sandboxed lua process with slightly different<br>globals) contains the logic of converting the user's cc{hrd='foo.h'} into<br>the object sent to<br>sys/cc/build.lua<br>the script actually executed to build the program.

This accomplishes two things simultaniously:<br>Decouples the user's specification of what to build from the<br>specific output structure needed for executing on a target<br>architecture. This gives the user freedom to structure their<br>source code however is most convinient to them while still reaping<br>the benefit of concise system-level flags like LD_LIBRARY_PATH<br>once built and installed.

Decouples the configuration of what needs to be built from the<br>process of building it - allowing the building the software to<br>scale to multiple threads or even multiple machines (if you rewrite<br>the civ protocol to scale for e.g. enterprise).

This is effectively what Bazel (Google's build system) does so well, but<br>while Bazel is a huge and complex monolith of software which decided<br>to fork python (of all things) for it's config language,<br>civ is written in a

1000 line cmd utility libraries which spawns a hermetic lua to<br>interpret configs using a

~100 line lua library.

Package Management

The final quality of a good build system is its capacity to be bolted<br>onto a simple package manager, and this is where I want to spend the<br>most time since it is in the design phase only. In civ...