Why I joined Blacksmith to work on storage again | Blacksmith

Blacksmith Sandboxes Coming soon

MENU<br>Close

SupportLog InLOG IN

Jul 24, 2026

Why I joined Blacksmith to work on storage again<br>Where off-the-shelf storage runs out

Piotr Bejda<br>Member of Technical Staff

TL;DR

Get started!<br>Try us Free

I want to tell you about the storage problems hiding inside continuous integration: why they're more interesting and more unusual than they look, and why they're the reason I joined Blacksmith. To explain that, I should start with how I ended up here.<br>What I look for in a job, more than anything, is to be challenged. I've spent my career chasing the kind of systems problems that don't come with ready answers: storage at Google, on Colossus, the largest multi-purpose storage foundation; an eBPF-based debugger at a startup; and later Datadog, most recently on production-grounded code optimization. The common thread isn't a particular layer of the stack; it's the pull of a problem nobody has a clean answer to yet.<br>So when I kept hearing that the people at Blacksmith were taking on exactly that kind of problem, it got my attention. I dug in, and found that Blacksmith (which runs continuous integration, the builds and tests that fire on every push and pull request, on its own fleet of machines) is, under the hood, a storage problem at least as much as a compute one. Standing up and tearing down enormous numbers of short-lived virtual machines, and feeding each one the data it needs (source, dependencies, build caches, the results it produces) turns out not to fit the tools you'd normally reach for: high scale, high churn, tight latency budgets. I didn't come for "managed CI." I came for that workload.<br>I've also observed that most of the industry's storage energy right now goes up the stack: new databases and data layers, increasingly AI-oriented, built on top of the same primitives we've had for years. What pulled me to Blacksmith was the rarer chance to work a layer down: to rethink the storage foundations themselves. Not as a greenfield indulgence, but because the workload here gives a concrete, well-grounded reason to question those primitives. Getting to build new foundations because the workload genuinely demands it is, to me, the most exciting kind of systems work there is.<br>So let me show you the workload. Here's what a CI job actually does to storage, at the scale we run it, and why the obvious answers don't drop in.<br>What a CI job actually does, at scale<br>Forget, for a moment, any picture of "servers with disks." The unit of work at Blacksmith is a fresh, ephemeral microVM that lives for a handful of minutes. Every CI job boots a clean VM, does its work, and is destroyed. There is no long-lived host quietly accumulating state the way a normal server does. And we run these at an enormous rate: every day we create and destroy on the order of two and a half million per-job volumes, with peaks around a thousand new VMs a minute.<br>That single fact is the root of everything below: state has to come from somewhere, fresh, every single time, fast. It starts before the job even runs: the machine has to read its own root filesystem to boot, and that same base image is demanded by huge numbers of VMs at once.

Once a VM is up, the work it does is a collection of distinct storage workloads, each with its own character:<br>Fetching the repo. Almost every job begins by pulling source. At our rate that's a wall of repetitive transfer against the same handful of repositories: the same bytes paid for over and over, often from a distant origin. What each job wants differs, too: a shallow tip here, full history there. The pressure is network throughput and locality, not bytes sitting on a disk.<br>Environment setup, repeated endlessly. Much like fetching the repo, nearly every run of a given repo redoes the same expensive setup (install the toolchain, the system packages, the dependencies) before any real work starts. The bytes that setup produces are essentially identical from run to run, and yet they're regenerated from scratch each time.<br>Integration tests that stand up dependencies. Databases, services, fixtures, seed data spun up per run: heavy, transient I/O that is almost entirely discarded when the job ends.<br>Builds that need incremental caching. Docker layer caches, compiler and dependency caches, build-tool state. These are write-heavy and only valuable if they survive the VM that produced them: the cache has to outlive the machine. And the caching semantics (what to keep, how to key it) live inside the build tool, not in any storage knob.<br>Persisting run artifacts. The flip side of all that throwaway: the things that genuinely must survive: build outputs, test reports, coverage, logs, images pushed downstream. A minority of the writes, with the opposite requirements: durable, addressable long after the VM is gone, sometimes large, governed by retention rules. The same system has to do "store almost nothing" and "never lose...