Characterizing Metastable Faults and Failures
Skip to main content
Characterizing Metastable Faults and Failures
Get link
Other Apps
July 22, 2026
Metastability has been studied in previous work as a self-sustaining degradation in goodput that persists even after the trigger is gone. The degraded state loiters on entirely due to the system's own internal feedback loops (retries, queues), and there is no simple reset button to press in distributed systems. So this is not a rare exotic problem. Since production systems would have already been hardened to handle the obvious failures, what remains is these hard-to-detect emergent failures. The "Metastable Failures in the Wild" paper (OSDI'22) reports 22 incidents across 11 organizations. Four of the 15 major AWS outages in a decade were metastable failures, with durations ranging from 1.5 to 73 hours.<br>This paper (June 2026) argues that the systems community has treated metastability phenomenologically, which led people to chase symptoms rather than causes. The paper sets out to give the first analytical causal account of these failures. This framing leads to two connections I found delightful. The first casts a metastable failure as a sin of composition among self-stabilizing systems. The second ties the healing mechanism to scheduling. Since I have worked on self-stabilizing systems for a long time (between 1998-2010), and thought hard about how they compose, these two connections really excite me.<br>I do have some reservations, though, which I will get to in my review. The paper overlooks prior work on the composition of stabilizing systems. It also pulls a bit of a sleight-of-hand in its formalization to argue that the metastable fault tolerance (MFT) design can be done via local pairwise reasoning between components. I don't buy that as I explain below.<br>These reservations do not dampen how much I enjoyed this paper. This is an idea paper, and it has been a while since I saw one of these in distributed systems. Moreover, the author list includes two of my all-time favorite distributed systems researchers: Robbert Van Renesse and Lorenzo Alvisi.<br>So let's dive in.
Sins of Composition and Self-Stabilization<br>The authors give formal definitions of a metastable fault and a metastable failure , and they draw a distinction between the two.<br>A metastable fault is what they beautifully call a sin of composition . (I am guessing this is Lorenzo being poetic.) Loosely speaking, the metastable fault appears when two or more components that are perfectly stable on their own get wired together into a cyclic interference/destruction loop. When a shock (say overload or loss of cache) then triggers the system, each component runs its local corrective action to stabilize itself, and in doing so it destabilizes its neighbor.<br>Let me back up and explain self-stabilization. A self-stabilizing system can start in any state and, with no outside intervention, converge on its own to a legitimate state and stay there. It lays out an elegant unified framework to tolerate any transient fault: A transient fault just leaves the system in some arbitrary state, and stabilization gradually heals it from there.<br>The paper leans hard on this stabilization theory going back to the 1980s. It defines a potential function (also called a variant or metric function) $f$ for each component, measuring how far the component is from a good state. A component is stabilizing if it eventually drives $f$ to zero, provided it runs inside a well-behaved "environment" $E$.<br>To formulate composition of stabilizing components, they add a compatibility check: if component A can stabilize when B is stable, and B can stabilize when A is stable, the two are compatible. But compatibility is not enough for guaranteeing composition of stabilization. Right after a shock, neither component is stable, so they hit a bootstrapping problem. Each waits for the other to recover first, and they get stuck in mutual destabilization, as their actions during recovery interfere with each other's healing progress.<br>This "sin of composition" is defined as the metastable fault. The fault turns into a failure only when the shock lands and the system's scheduler keeps favoring the locally stabilizing but globally destabilizing interactions over the stabilizing ones.<br>None of this surprises a stabilization researcher. Stabilization is famously hard to compose, precisely because the recovery strategies of the components interfere with each other. The challenge is to keep the components from corrupting each other during recovery. One clean way is layered composition: let the lower layer stabilize first (the higher layer can read it but not write to it), and let recovery flow upward. In general, though, when you compose systems you have to design the correction actions deliberately, check that they don't interfere, and prove they stabilize together.
Overreaching for Local Reasoning<br>The paper's theoretical...