Unranked, systemd, crawls |<br>marginalia.nuUnranked, systemd, crawls<br>Posted: 2026-07-22<br>There’s been some changes to Marginalia Search:<br>Migrated to systemd from docker<br>New unranked query endpoint<br>Wide domains got their own index for faster crawls<br>In brief, this has removed a lot of operational headaches,<br>reduced expensive queries leaving more computational power for the rest,<br>and cut crawl times in half.<br>Let’s tackle them in some order.<br>We have docker at home<br>The system has been migrated off docker compose, and onto bare systemd in production.<br>This has been successful, and solved a number of issues.<br>There are a few constraints that demand a non-standard deployment<br>for Marginalia.<br>NUMA (non-uniform memory architecture).<br>The production server has two CPUs, each with their own RAM bank,<br>and while they can reach over into each other’s memory, this comes at a cost.<br>In many scenarios this isn’t a big deal, but for indexes and databases,<br>generally bottlenecked by RAM bandwidth, this is far from ideal.<br>Process lifecycles.<br>A search engine is anything but stateless,<br>some processes are fairly heavy and run for weeks,<br>some services are very slow to start,<br>some services need to restart relatively often.<br>This all but demands that the search engine is cut up into different pieces.<br>Not necessarily microservices, but at least services.<br>IP addresses.<br>The crawler (and crawler-adjacent processes) run off about a dozen public IP addresses,<br>from the same host. This is done via ipvlan using network namespaces.<br>Linux has many advanced capabilities for giving processes their own virtual network stack,<br>which can be wired together with virtual switches and virtual patch cables.<br>There are to my knowledge no tools for managing this I would consider good,<br>it’s all either too low level, or abstracting away key functionality,<br>or suffering from multiple sources of truths leading to jank when they inevitably differ.<br>Docker can deal with all of these constraints, but at the expense of considerable jank.<br>Docker’s networking model in particular, while it supports ipvlan, doesn’t map onto it particularly well,<br>if you have a finite subnet you pretty much have to have spare free IPs to be able to reliably restart services.<br>It further doesn’t let you set firewall rules for the ipvlan interface, which means that anything you put there<br>better not bind on the public IP, or its ass is going to be hanging out in full view. This is in no way a limitation of ipvlans, but purely a docker problem.<br>This is made worse by the fact that there is inadequate tooling for letting the processes in the container<br>know which network interface is public, so you have to kinda guess based on dowsing rods, RFC 1918, and looking at tea leaves.<br>This is no way to live.<br>There is no magic in docker, it’s just cgroups and namespaces all the way down, and you can fully replicate what it’s doing yourself, if you really want to torment yourself using syscalls or shellscripts, or if you want a middle ground using systemd.<br>If docker’s problem is that there is a mismatch between the docker model of the system and the system itself,<br>systemd represents a more raw approach, where a lot more of the configuration is in your hands,<br>while still offering many capabilities you’d want for a more serious deployment,<br>such as health checks, automatic restarts that can be configured.<br>Setting up a million .service:s is pretty tedious, but drop-ins help. There isn’t a ton to say here, other than after some fiddling, it works very well, and feels a lot more stable than docker ever did. Cutting JIB out of builds means builds now take 2-3 seconds for the most part.<br>Deployments are faster, there’s less jank in service discovery because “containers” retain the same internal and external IP,<br>the whole operation feels considerably less floaty. Good upgrade.<br>There are many opinions about systemd, and I think for most desktop-type systems, it’s incredibly overengineered and kind of a pain to work with. Though for something like this, its design is well motivated, and it really shines.<br>Unranked Queries<br>The index now allows unranked query execution!<br>The query execution pipeline used in the search engine is fairly computationally expensive,<br>and does things that aren’t strictly necessary in many queries.<br>If you’re for example just looking for backlinks, a pure intersection of the terms ‘site:foo.com links:bar.com’ is enough,<br>there’s nothing to rank anything there, but things working the way they did, threads were allocated to ranking none the less, term positions were attempted to be retrieved, all a bunch of done that needed not to be done.<br>So I added a path for unranked queries that just do dumb term intersections and a bare minimum of anything else.<br>The bonus of this is that with some fiddling and the construction of a cursor that can map to multiple index partitions, unlike the...