GitHub - telemetry-sh/tail-latency-lab: Simulate when hedged requests improve tail latency and when they cause overload. · GitHub
/" data-turbo-transient="true" />
Skip to content
Search or jump to...
Search code, repositories, users, issues, pull requests...
-->
Search
Clear
Search syntax tips
Provide feedback
--><br>We read every piece of feedback, and take your input very seriously.
Include my email address so I can be contacted
Cancel
Submit feedback
Saved searches
Use saved searches to filter your results more quickly
-->
Name
Query
To see all available qualifiers, see our documentation.
Cancel
Create saved search
Sign in
/;ref_cta:Sign up;ref_loc:header logged out"}"<br>Sign up
Appearance settings
Resetting focus
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
Uh oh!
There was an error while loading. Please reload this page.
telemetry-sh
tail-latency-lab
Public
Notifications<br>You must be signed in to change notification settings
Fork
Star
main
BranchesTags
Go to file
CodeOpen more actions menu
Folders and files<br>NameNameLast commit message<br>Last commit date<br>Latest commit
History<br>1 Commit<br>1 Commit
.github/workflows
.github/workflows
docs
docs
src
src
web
web
.env.example
.env.example
.gitignore
.gitignore
Cargo.lock
Cargo.lock
Cargo.toml
Cargo.toml
LICENSE
LICENSE
README.md
README.md
View all files
Repository files navigation
Tail Latency Lab
A duplicate request can rescue your p99, or become the load that breaks it.
Tail Latency Lab is a deterministic Rust simulator for hedged requests. It races duplicate attempts against long-tail responses, places every attempt behind a finite worker pool, and shows the point where latency insurance turns into queue collapse.
The interesting part
A hedge sent after 100 ms looks cheap when only 3% of requests are slow. It is not cheap when the worker pool is nearly saturated: duplicates wait behind originals, uncancelled losers keep consuming service time, and more requests live long enough to launch another attempt.
On the default workload, hedging has a clear win:
hedge p99 timeout traffic queue<br>disabled 727ms 0.00% 1.00x 0.0ms<br>50ms 89ms 0.00% 1.03x 0.0ms<br>100ms 139ms 0.00% 1.03x 0.0ms<br>200ms 239ms 0.00% 1.03x 0.0ms
Move the same policy closer to the capacity boundary and the feedback loop reverses. The included Near saturation preset demonstrates that transition.
Run the interactive lab
Rust 1.85 or newer is required.
cargo run --release
Open http://127.0.0.1:3000. If that port is occupied:
PORT=3001 cargo run --release
The browser interface is served by Axum and calls the same simulation engine as the CLI. It has no frontend build step or CDN dependency.
Use the CLI
cargo run --release -- compare --preset "Quiet heavy tail"<br>cargo run --release -- compare --preset "Near saturation" --json<br>cargo run --release -- compare --preset "Pathological tail" --hedge-ms 150
Available presets:
Quiet heavy tail
Near saturation
Pathological tail
Use the engine
(())">use tail_latency_lab::{simulate, SimulationConfig};
let result = simulate(SimulationConfig {<br>arrival_rps: 1_000,<br>workers: 64,<br>hedge_delay_ms: 75,<br>..SimulationConfig::default()<br>})?;
println!("p99={}ms traffic={:.2}x", result.p99_latency_ms, result.amplification_factor);<br># Ok::(), String>(())
Model boundaries
The engine is a seeded discrete-event simulation, not a production capacity planner.
Logical requests arrive at a uniform rate.
Service time comes from a jittered fast/slow mixture.
Attempts queue against a fixed-size worker pool.
A hedge is sent only if the logical request is still pending.
The first successful attempt wins, but losing work is not cancelled.
Queued attempts that cannot start before their deadline are discarded.
Autoscaling, connection pools, network loss, retries beyond one hedge, and correlated failures are intentionally omitted.
Stable per-request samples make policy comparisons fair: changing the hedge delay does not silently redraw the original workload.
Optional aggregate telemetry
The lab runs offline by default. To record aggregate usage bands with the Telemetry Rust SDK:
export TELEMETRY_API_KEY=your_project_api_key<br>cargo run --release --features telemetry
Only the hedge-delay band, timeout band, and whether traffic exceeded 1.1x are sent. Request values, simulation timelines, and IP-derived data are not logged by the integration.
Development
cargo fmt --all -- --check<br>cargo clippy --all-targets --all-features -- -D warnings<br>cargo test --all-features
License
MIT
About
Simulate when hedged requests improve tail latency and when they cause...