GitHub - telemetry-sh/retry-storm-lab: Simulate how retry policies amplify traffic during partial outages. · GitHub

/" data-turbo-transient="true" />

Skip to content

Search or jump to...

Search code, repositories, users, issues, pull requests...

-->

Search

Clear

Search syntax tips

Provide feedback

--><br>We read every piece of feedback, and take your input very seriously.

Include my email address so I can be contacted

Cancel

Submit feedback

Saved searches

Use saved searches to filter your results more quickly

-->

Name

Query

To see all available qualifiers, see our documentation.

Cancel

Create saved search

Sign in

/;ref_cta:Sign up;ref_loc:header logged out"}"<br>Sign up

Appearance settings

Resetting focus

You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.

Dismiss alert

{{ message }}

Uh oh!

There was an error while loading. Please reload this page.

telemetry-sh

retry-storm-lab

Public

Notifications<br>You must be signed in to change notification settings

Fork

Star

main

BranchesTags

Go to file

CodeOpen more actions menu

Folders and files<br>NameNameLast commit message<br>Last commit date<br>Latest commit

History<br>2 Commits<br>2 Commits

.github/workflows

.github/workflows

.streamlit

.streamlit

docs

docs

retry_storm_lab

retry_storm_lab

tests

tests

.env.example

.env.example

.gitignore

.gitignore

LICENSE

LICENSE

README.md

README.md

app.py

app.py

pyproject.toml

pyproject.toml

View all files

Repository files navigation

Retry Storm Lab

A retry is not free. See when your recovery policy becomes the outage.

Retry Storm Lab is a deterministic Python simulator for partial dependency failures. It models fresh requests and retries competing for the same capacity, then compares immediate, fixed, exponential, and full-jitter backoff policies.

Why this exists

Retries improve reliability when failures are brief and spare capacity exists. During a sustained partial outage, synchronized clients can instead amplify demand, starve fresh requests, and keep a recovering service overloaded.

The simulator makes that feedback loop visible:

Generate a steady stream of fresh requests.

Reduce successful capacity during a configurable failure window.

Reschedule failed and over-capacity attempts using the selected policy.

Measure amplification, peak demand, abandoned requests, and recovery time.

Compare all four policies against the exact same scenario and random seed.

This is an intentionally small queueing model, not a production capacity planner. It omits network latency distributions, client timeouts, autoscaling, circuit breakers, and correlated downstream failures. Its job is to make retry dynamics easy to inspect and discuss.

Run the interactive lab

Python 3.11 or newer is required.

python -m venv .venv<br>source .venv/bin/activate<br>pip install -e .<br>streamlit run app.py

Open http://localhost:8501, choose a preset, and adjust the workload from the sidebar.

Use the CLI

retry-storm-lab --preset "Rate-limit spiral" --compare<br>retry-storm-lab --preset "Slow recovery" --policy full_jitter --json

Example comparison:

immediate amplification=1.11x peak=2720 rps success=97.2% recovery=1s<br>fixed amplification=1.08x peak=1600 rps success=99.9% recovery=3s<br>exponential amplification=1.07x peak=1600 rps success=100.0% recovery=3s<br>full_jitter amplification=1.10x peak=1651 rps success=99.9% recovery=3s

Exact values depend on the selected preset and seed.

Use the simulation engine

from retry_storm_lab import BackoffPolicy, SimulationConfig, simulate

result = simulate(<br>SimulationConfig(<br>baseline_rps=800,<br>server_capacity_rps=1_000,<br>outage_failure_percent=80,<br>policy=BackoffPolicy.FULL_JITTER,

print(result.amplification_factor)<br>print(result.peak_rps)<br>print(result.recovery_seconds)

The engine uses aggregate 100 ms buckets. Full jitter distributes retries individually with a seeded pseudo-random generator, so the same configuration always produces the same result.

Optional aggregate telemetry

The lab works entirely offline. To record aggregate usage bands with Telemetry, install the optional dependency and provide a project API key:

pip install -e '.[telemetry]'<br>export TELEMETRY_API_KEY=your_project_api_key<br>streamlit run app.py

Only the selected policy, broad capacity/amplification bands, and whether the run recovered are sent. Raw workload values and timeline data are not logged.

Development

python -m unittest discover -s tests -v<br>python -m compileall app.py retry_storm_lab

License

MIT

About

Simulate how retry policies amplify traffic during partial outages.

Topics

python

distributed-systems

simulation

reliability

backoff

sre

retries

observability

chaos-engineering

streamlit

Resources

Readme

License

MIT license

Uh oh!

There was an error while loading. Please reload this...