Designing Market Data for Deterministic Replay - Electronic Trading Hub

Skip to content

Search for:

Electronic TradinginfrastructureMain News

Ariel Silahian<br>Ariel Silahian is a senior technology executive in institutional electronic trading, with 30+ years across the buy and sell side (New York, Miami, London, Hong Kong). He is the author of "C++ High Performance for Financial Systems" (Packt) and the creator of VisualHFT, the open-source microstructure analytics stack. He writes on exchange architecture, market microstructure, and execution quality, and advises a select number of trading firms on infrastructure decisions that move P&L. Talk architecture: https://hftadvisory.com

You are designing the market-data path for a trading system. Somewhere in the requirements, written down or not, is one that quietly shapes every layer beneath it: any moment the system traded through has to be reconstructable later, exactly. Feed the system the same input it consumed at 09:31:07.442, in the same order and with the same relative timing, and it should produce the same output down to the bit.

That property is deterministic replay. Most designs treat it as something to add when an incident finally forces the question, and by then it is already gone. Replay is a property of the market-data path you design in from the start or lose. This is a walk through the design decisions that keep it, in roughly the order you make them.

Table of Contents

The Requirement Most Designs Skip

Know What Breaks It: The Six Determinism Leaks

Decision 1: Capture at the Boundary

Decision 2: Event Order Belongs to the Data, Not the Scheduler

Decision 3: Replay Is a Runtime Mode

Decision 4: Verify Fidelity, Then Size the Storage

Porting the Design to Other Venue Classes

When the Design Is Done

The Requirement Most Designs Skip

Write the requirement down before you draw the first box, because it constrains all of them: for any past interval, you can produce the exact input the system consumed, and re-running the production code over that input reproduces the production output.

Two halves, both load-bearing. The input has to be the real input, captured as it arrived, rather than a later approximation rebuilt from something downstream. And the code path over it has to be the real production code path, not a separate model of it. Miss either half and replay becomes a plausible story you cannot verify, which is worse than no replay at all because it looks like a control while behaving like a guess.

Everything that follows is in service of those two halves. Capture the real input at the earliest point you own. Fix the order of that input in the data itself. Run the same binary over it. Then prove the output matches.

Know What Breaks It: The Six Determinism Leaks

Determinism breaks at specific, findable points, and your design has to close each one, so name them first. Each leaves a distinct signature you can look for in a system that has the leak.

Thank you for reading this post, don't forget to subscribe!

Subscribe by Email

Wall-clock reads in the hot path. Any branch that reads the system clock as part of a decision, rather than only for logging, injects a value that differs every run. Signature: feed the same recorded packets through the strategy twice and get two different decisions. The design rule is to separate clock reads used for timestamping, which are necessary, from clock reads that feed logic, which are a leak.

Non-deterministic thread scheduling in a multi-threaded merge. When several threads each consume a channel and hand events to a shared merge point, the order the OS schedules them is not guaranteed run to run. Signature: a nightly regression replay produces a slightly different book state each night on identical input files, with no code change.

Unordered UDP multicast arrival. Exchange market data typically arrives over UDP multicast with no transport-layer ordering guarantee, often over redundant A and B lines delayed independently by network path. Signature: two packets carrying the same event arrive in a different relative order depending on switch load or NIC queue, and merge logic that trusts arrival order instead of sequence number builds a different book on different days.

Gap-fill and retransmission races. When a gap is detected and a retransmission or snapshot-recovery request goes out, the recovered data arrives interleaved with live traffic. If the splice point is decided by arrival time rather than sequence number, the resulting book depends on network timing that varies run to run.

Floating-point non-associativity. IEEE 754 arithmetic is not associative: summing the same values in a different order can produce a different result. If a book aggregate, a VWAP, or any derived quantity is computed over an event order that is not itself fixed, the value is not reproducible even when the underlying facts are identical.

Any I/O not captured at the boundary. A config read, a reference-data...