Map the Failure Boundary — Poisson Labs Map the Failure Boundary<br>August 19, 2026
Traditional software relies on unit tests. Non-deterministic AI, from LLM agents to RL policies, relies on static benchmarks and single-trajectory evaluations. But knowing a system succeeded once doesn’t tell you how reliably it will succeed again.
We built Monte to map the failure boundaries of decision-making systems. As a first test of the instrument, we froze a trained Unitree Go1 locomotion policy and tested it against 6,400 combinations of floor friction and lateral push. The result is a map of where the policy recovers and where it falls.
For each condition, Monte spins up the physics, runs repeated rollouts, and estimates the probability of surviving a five-second episode. The same system lets us probe individual worlds, inspect the resulting trajectories, and re-run the experiment after retraining.
Survival probability across the grid. Pale cells survive the full 5-second episode; dark cells fall. Push magnitude runs left to right (10–200% of bodyweight); floor friction runs bottom to top (μ 0.05–1.00). ×
6,400 rollouts · 20×20 conditions · 16 trials per cell · 126 s of GPU time
At a glance
the failure boundary is broad and variable: a band, not a threshold
below μ ≈ 0.15, the policy usually falls before the push matters
retraining on pushes + low friction moved the boundary outward in 60 cells and worsened none
the full experiment, including retraining and the repeat sweep, cost $42
Monte runs the sweep, the statistical analysis, the live probe, and the surface comparison.
Nine worlds, one push, synchronized at t=2s. Friction increases bottom-to-top, push magnitude left-to-right.<br>The result
The boundary is diagonal: grippier floors tolerate harder pushes. On medium friction the policy survives pushes up to roughly its own bodyweight (about 125 N for this 12.7 kg robot) and fails beyond that. An earlier back-of-the-envelope estimate had put the boundary near 15 N; the measured value is several times higher, which is why this study reports only measured numbers.
The surprising failure mode. Below μ ≈ 0.15 the map goes dark at the left edge: the robot falls at a 10% bodyweight nudge, or with no push at all. On ice the policy can barely walk. That is a different failure mode from being knocked over (the gait itself collapses), and it lives entirely outside the friction range the policy trained on.
The boundary is also not a sharp threshold. Bootstrapping the 16 trials in each cell (2,000 resamples) gives a 95% interval on each row’s crossing point: about ±4% of bodyweight on high friction, ±33% on ice, a 7× spread. A single curve would hide the width, and the width is a finding.
The S=0.5 crossing per friction row with its 95% bootstrap interval. Median width is about 16% of bodyweight; the boundary is sharpest where the policy is most comfortable. ×
How we measured it
A demonstration video is a single draw from a stochastic system; it says little about the conditions next door. The grid exists to measure the distribution: which conditions the policy survives, and how often, cheaply enough to re-measure after every training change. Monte turns each grid cell into a small experiment: launch the simulator, run repeated trials, record time-to-failure, and aggregate the outcomes into a survival estimate.
Policy. Go1 joystick locomotion (MuJoCo Playground’s recipe, Brax PPO), trained once for 217.9M steps (736 seconds on one A100) to a nominal reward of 30.30 ± 0.54, then frozen.
Training distribution. Friction randomized over U(0.4, 1.0), with mass, center-of-mass, and joint jitter, and no pushes. Every push in this study is out-of-distribution by construction, and the bottom third of the friction axis is below anything the policy saw.
Perturbation. A constant lateral force on the torso for 0.5s, starting at t=2s, applied in physics only; the observation the policy sees is untouched. 100% of bodyweight = 125.0 N.
Failure. Torso below 0.15 m or tilt past 60°, whichever comes first, with thresholds frozen before the sweep. Rollouts that survive the 5-second window are censored, not counted as late failures. We use survival analysis (Kaplan–Meier per cell) precisely because some trials never fail.
One cell of the map, three camera angles: μ=0.35, an 80% bodyweight push at t=2s, time-to-failure 2.72s. The clip freezes at the failure moment: the training simulator models foot-ground contact only, so a fallen body has nothing to rest on.<br>How much should we trust the map?
We tested the measurement itself, three ways.
Harness fidelity. The evaluation harness reproduces the training environment: we verified that replaying the nominal trajectory through the injection path yields identical states over 500 steps, and that the push enters through physics only.
Numerical sensitivity. Halving the timestep moved the boundary within bootstrap noise; doubling solver iterations moved it by about 17% of...