Query cycles: A compiler murder mystery - Ferrous Systems
coding
Query cycles: A compiler murder mystery
Jynn
July 2026
Article
Query cycles: A compiler murder mystery
I HAVE NO TOOLS BECAUSE I HAVE DESTROYED MY TOOLS WITH MY TOOLS
Published on July 09, 2026<br>14 min read
Rust
Ferrocene
Do you need help with coding?
Get in touch
Get updates
On the Ferrocene team, we deal with a lot of tricky problems,<br>as Aïssata Maïga describes in her recent RustWeek talk, The trials of qualifying Rust.
This post dives into a particularly thorny problem that crashed my computer, took me over a week to debug, and ended up being 3 different bugs combining into one very unhelpful error.
It would be nice to have an error message …
Ferrocene regularly merges in changes from the upstream Rust compiler. A few months ago, we encountered the following error in CI after a merge:
test [ui] tests/ui/delegation/bad-resolve.rs has been running for a long time
[ui] tests/ui/delegation/bad-resolve.rs ... F<br>test [ui] tests/ui/delegation/body-identity-glob.rs has been running for a long time
Too long with no output (exceeded 20m0s): context deadline exceeded
I took a look at this on a local build of the compiler. I do a lot of my work on "devpods": these are similar to the dev-desktops of the upstream Rust project, but hosted on Ferrocene's private infrastructure, and with custom Docker images that set up a lot of the build requirements beforehand. Unlike dev-desktops, they're Kubernetes pods, not full virtual machines. I like them because they're much faster than my local MacBook Air, and also can run x86 natively rather than through emulation.
I started the test without a problem, wandered away for lunch, and came back to this error:
ci@devpod-pr-rotation-0 ~/ferrocene (automation/pull-upstream-HEAD/mscy1uah)> Connection to bastion.devpods.infra.ferrous-systems.net closed by remote host.<br>client_loop: send disconnect: Broken pipe<br>Connection to devpod-pr-rotation-jynn.ferrocene-devpods.svc.cluster.local closed by remote host.<br>Connection to devpod-pr-rotation-jynn.ferrocene-devpods.svc.cluster.local closed.
Well, that's odd. I tried to reconnect:
$ ssh devpod-pr-rotation-jynn.ferrocene-devpods.svc.cluster.local -J bastion.devpods.infra.ferrous-systems.net<br>channel 0: open failed: connect failed: Connection refused<br>stdio forwarding failed<br>Connection closed by UNKNOWN port 65535
That's really odd! That's the error I get when DNS resolution doesn't work within the bastion container. At that point I started looking at the state of the devpod:
$ kubectl get pods<br>devpod-pr-rotation-0 0/1 OOMKilled 0 16d
… oh.
Well! Clearly something here has gone very wrong. I couldn't get the pod back; to bring it back up I had to run kubectl --namespace ferrocene-devpods delete pod devpod-pr-rotation-0, which deletes the whole container, including the data on disk. Welp. There goes an hour.
What even triggers the problem?
I got myself a cozy little dev loop:
$ git fetch origin eb655d5b3f940faf9b25749e84efed1b79944e98 && git checkout FETCH_HEAD<br># Create a build cache so we can set a reasonable timeout.<br>$ ./x test tests/ui/diagnostic-width<br># `timeout` is to make sure that we don't immediately crash the devpod again.<br># `--no-capture` is to see output from RUSTC_LOG.<br># `--force-rerun` is to rerun the test even if neither the compiler nor the test have changed,<br># since compiletest doesn't count RUSTC_LOG as invalidating the cache.<br># `--keep-stage-std` is to avoid rebuilding the standard library when the compiler changes,<br># since none of our changes affect code that doesn't use delegation.<br>$ timeout 10 ./x test tests/ui/delegations/bad-resolve.rs —-force-rerun --verbose --no-capture --keep-stage-std=1
and minimized the crash to the following Rust source code:
#![feature(fn_delegation)]<br>#![allow(incomplete_features)]
trait Trait {<br>type Type;<br>fn bar() {}
struct F;<br>impl Trait for F {<br>type Type = i32;
struct S(F);<br>impl Trait for S {<br>reuse F as Trait>::bar;
fn main() {}
The reuse syntax here is part of the unstable fn_delegation feature, which rustc notes is incomplete and buggy. On upstream rustc, this program gives the following error:
On Ferrocene, it hangs indefinitely without output.
First I tried cranking debug logging all the way up to the max:
{ \<br>RUSTC_LOG=debug,rustc_middle::ty::print=info,rustc_span=info,rustc_parse=info \<br>timeout 5 rustc +ferrocene ./bad-resolve.rs || true<br>} 2>&1 | less
and got this output towards the end:
0ms DEBUG rustc_ast_lowering return=HirId(DefId(0:14 ~ bad_resolve[627f]::{impl#1}::bar).9)<br>0ms DEBUG rustc_middle::middle::codegen_fn_attrs::ferrocene parent=DefId(0:3 ~ bad_resolve[627f]::Trait), kind=Trait
That's promising! This ferrocene module is one we just added last week; it's very possible that it still has a few bugs.
I HAVE NO TOOLS BECAUSE I'VE DESTROYED MY TOOLS WITH MY TOOLS
I suspected from the start that this was infinite recursion somehow, so I pulled out GDB. My normal trick for...