The Fedora 45 Sausage Factory | supakeen's homepage<br>supakeen's homepage<br>The Fedora 45 Sausage Factory<br>This is a walkthrough of how Fedora turns source code and packages into the<br>artifacts you download and install. It follows the a package from a packager’s<br>git push to a composed release: ISOs, cloud images, container images, and OSTree<br>deployments.<br>The walkthrough describes how the Fedora ‘sausage’ is created as of Fedora 45,<br>things change all the time; I hope to have time to update this document every<br>cycle or every few cycles of Fedora releases so there’s both history and people<br>can find up to date information.<br>In fact this document is currently still living as Fedora 45 has not been released<br>yet so this post likely will get updated until the release. There’s a few change<br>proposals in flight that change some of the bits and bobs here. Specifically how<br>the boot.iso is produced which is a decently large part of this post.<br>Thus now is also the right time to let me know if I got anything wrong, please<br>e-mail me at cmdr@supakeen.com if you find any errors.<br>Starting Line: dist-git<br>Things starts with a packager pushing a commit to a package.<br>Fedora stores the source definitions of every package in individual Git<br>repositories at src.fedoraproject.org. Each<br>repo contains an RPM spec file, any downstream patches, and a sources file<br>that points to upstream tarballs stored in a separate lookaside cache. The<br>large binary files stay out of Git; the other files exist inside of Git and<br>thus have full version control.<br>Packagers usually interact with these repos through fedpkg, a CLI that wraps<br>the common operations: cloning repos, uploading source tarballs, submitting builds,<br>and creating updates. The important thing about fedpkg build is what it<br>actually does: it constructs a URL pointing at a specific commit in the Git<br>repo and hands that to Koji, the build system. The build is fully reproducible<br>from that commit hash.<br>Branches map to releases: rawhide for development, f44 for Fedora 44.<br>The hosting is split between Pagure (src.fedoraproject.org) and Forgejo<br>(forge.fedoraproject.org), with an ongoing<br>migration from the former to the latter.<br>Packagers can do work without using fedpkg, but for the purposes of this<br>post it’s easier to assume fedpkg is in use since it abstract away some of<br>the bits that are irrelevant to the reader.<br>Building Packages: Koji<br>When fedpkg build submits that Git URL, Koji takes over. Koji is Fedora’s<br>build system. It has been around since Fedora 7, and it builds essentially<br>everything.<br>Koji follows a hub-and-spoke architecture. The hub is a passive XML-RPC server<br>that sits in front of a PostgreSQL database. Builder daemons poll the hub for<br>work, create a fresh Mock<br>chroot environment for each build, run the build, and upload the results. Every<br>build starts from a clean room. You can never get a different result because<br>someone installed something on the builder last week.<br>The organizational model is built around tags . A tag is a named collection<br>of builds. Build targets map an incoming build request to two tags: a build tag<br>(which defines the buildroot, the packages available during the build) and a<br>destination tag (where the finished build lands). Tags support multiple<br>inheritance, so you can layer a Fedora 44 tag on top of a base tag without<br>duplicating everything.<br>Koji doesn’t just build RPMs. Through its plugin system and content generators,<br>it also orchestrates image builds: Kiwi images via the kiwiBuild task type,<br>Image Builder artifacts via imageBuilderBuild, and OSTree composes via<br>runroot tasks. We’ll get to those.<br>Gating Updates: Bodhi<br>A fresh RPM build sitting in Koji doesn’t automatically reach users. For<br>branched releases (anything that isn’t Rawhide), it goes through<br>Bodhi, Fedora’s update management system.<br>Bodhi gates the release of updates through a feedback and testing cycle. A<br>packager submits an update containing one or more builds. That update goes<br>through a sequence of states: pending, testing, stable. Users and automated<br>tests provide karma (+1 / -1). When an update hits +3 karma or spends enough<br>days in testing, it can be pushed to stable.<br>Behind the scenes, Bodhi manages all of this through Koji tags. When an update<br>moves from testing to stable, Bodhi moves the build from the<br>f44-updates-testing tag to the f44-updates tag. It then invokes Pungi to<br>compose the update repository, the actual yum/dnf repo that users pull from<br>when they run dnf upgrade.<br>Critical path packages, the ones your system needs to boot and function, get<br>stricter requirements: 14 days in testing instead of 7, and more karma needed.<br>Bodhi also integrates with Greenwave and ResultsDB for CI test gating, so<br>automated test failures can block an update from reaching stable.<br>For Rawhide, most of these processes are skipped. Builds go directly from Koji<br>into the Rawhide tag, which is why Rawhide occasionally breaks.<br>Composing a Release:...