What is your CI job talking to? | Blacksmith

Blacksmith Sandboxes Coming soon

MENU<br>Close

SupportLog InLOG IN

Jul 20, 2026

What is your CI job talking to?

Andrew Werner<br>Member of Technical Staff

TL;DR<br>How we built line-rate network observability for untrusted CI jobs without touching the guest.

Get started!<br>Try us Free

What is your CI job talking to?<br>Part 1 of 2: building network observability for untrusted CI workloads.<br>What a CI job does on the network and when is a goldmine. Builds spend a shocking fraction of their wall clock waiting on the network: pulling images, resolving and downloading dependencies, hitting caches, cloning repos. When a customer asks "why is this build slow?", the answer is very often hiding in a flow nobody knew existed: the registry that isn't cached, the dependency fetched from the other side of the world, the test suite quietly downloading a browser binary on every run. Beyond performance, knowing exactly what every job talks to is the basis for something bigger: an egress policy system that can actually protect builds when a dependency turns malicious. That's Part 2.

We set out to capture a complete and queryable record of all outbound network traffic, attributed to the domain name per job and even per step. Blacksmith's whole pitch is the fastest, most broadly compatible runners around so many were off the table from day one: no man-in-the-middle proxy with an injected certificate authority (breaks certificate pinning, breaks tools that ship their own trust stores, breaks the "drop-in" promise); nothing that adds measurable latency or per-byte cost to the data path; and nothing inside the guest at all, both because the workload is untrusted and because we refuse to change what a customer's build sees.<br>What made this tractable is where we get to stand. Every Blacksmith CI job runs in its own ephemeral VM, and all of that VM's traffic crosses a single veth pair on the host. The guest can't see it, can't reconfigure it, and can't route around it  whatever we build there is invisible to the workload and immune to it. Three problems make what we built there more interesting than "run tcpdump on the veth":<br>Everything meaningful is a name, but the kernel deals in IPs. Knowing a job sent 4 MB to 140.82.113.4 is close to useless; knowing it went to github.com:443 is the whole point. We solve attribution by owning DNS with a host-side proxy.<br>The data path must cost approximately nothing. We observe at line rate with eBPF programs attached via TCX. We'll get into why that beats both a userspace proxy and iptables counters.<br>The collection mechanism needs bounded, low overhead Per-destination counters of every VM, read out every second, without a per-packet event stream. The mechanism, BPF map iterator programs, is a lesser-known corner of eBPF worth highlighting.<br>The assumption that shapes the design: trust nobody<br>One premise shapes every design decision that follows: the workload inside the VM is hostile. A CI job runs arbitrary code, and that code can become root inside the guest. Anything living in the guest is within the blast radius of the thing we're trying to observe, so nothing in the guest can be a source of truth.<br>Every mechanism in this system therefore lives outside the guest in the host kernel or in a host-side process. It's a clean boundary: the workload owns everything on its side of the virtual NIC, and we own everything on ours. Nothing the guest does can hide evade detection.<br>Here's the path a packet takes leaving a Blacksmith VM:

Every VM gets a veth pair straddling the guest's network namespace and the host root namespace, and all of the guest's traffic has to transit the host-side end of that pair to reach the internet. We attach two things there: an eBPF program at each direction of the veth (ingress and egress, so we can count bytes both ways), and an iptables rule that redirects the guest's DNS traffic to a proxy we control. Those two pieces, the eBPF observer and the DNS proxy, are the heart of the system. Let's take them in the order that makes the data meaningful: names first.<br>Names, not IPs: the DNS proxy<br>The kernel deals in IP addresses. Humans, and security policies, deal in names. If all we reported was "your job sent 4 MB to 140.82.113.4," you'd have to do the reverse lookup yourself and hope the answer was stable. Worse, that mapping is exactly the thing a hostile workload would love to muddy. So the first job is to reliably attribute every outbound flow back to the domain name the workload actually resolved.<br>We do that by owning DNS. A per-veth iptables DNAT rule rewrites the guest's port-53 traffic to a DNS proxy running in the vm-agent on the host bridge IP:<br>DNAT udp/tcp dport 53 -i → 192.168.127.1:The guest is none the wiser. Its /etc/resolv.conf still points at the conventional 192.168.127.1:53; the rewrite happens in the host kernel, below anything the guest can observe or reconfigure. Even if the workload rewrites resolv.conf to point at...