Scenario Tests | Srinidhi Nagendra

Scenario Tests

2026-07-12

Next in my list of trying to use AI to further my PhD research is Scenario Tests. A unit testing<br>framework for distributed protocols. For a given protocol implementation, I use AI to write Scenario<br>Tests automatically from the protocol specification.

The first work in my PhD was Netrix, a domain specific language for testing distributed protocol<br>implementations. Essentially, I was trying to answer the question - What is the equivalent of a unit<br>test for distributed protocols? The answer is Scenario tests.

Distributed protocols are notoriously hard to implement correctly, and even harder to find bugs in.<br>However, it is relatively easy to present new protocols and theoretically prove their correctness. The<br>proofs typically follow some form of scenario reasoning. If this invariant is violated, then we get<br>into a scenario which can never happen.

Understanding the reasoning and believing in the correctness of the proof is a first step in implementing<br>the protocol. Therefore, it is also very natural for the developer implementing the protocol to write<br>the tests as scenarios.

To write a Scenario Test, a developer writes a sequence of filter rules. They filter events to track a<br>runtime monitor, and also guide the execution of the distributed protocol. Let's look at a simple<br>example for Raft.

// MsgApp from a leader would mean election succeeded — failure condition.<br>sm := netrix.NewStateMachine()<br>sm.Builder().On(netrix.IsMessageType(pb.MsgApp), netrix.FailureState)

filters := netrix.NewFilterSet()<br>filters.AddFilter(netrix.If(netrix.IsMessageType(pb.MsgVote)).Then(netrix.DropMessage()))<br>filters.AddFilter(netrix.If(netrix.IsMessageType(pb.MsgVoteResp)).Then(netrix.DropMessage()))

...

result := runNetrixTest(t, tc, envFunc)<br>require.NoError(t, result.Err)<br>require.False(t, result.IsFailure(), "no leader should be elected when all votes are dropped")

The test states a simple property and enforces a specific scenario in the execution of the distributed<br>system - one where votes and their responses are never delivered and hence, no leader is ever elected.<br>The test has two components, (1) The state machine, and (2) the filters. The state machine tracks the<br>state of the test, derived either from messages sent or from the local state of the nodes in the<br>distributed system. The filters decide which messages are delivered, dropped, duplicated, manipulated,<br>or just observed. Together, they create a scenario and describe a property that needs to be satisfied.

Underneath, the test is driven by a stream of events. Events are created when messages are sent,<br>received, or other internal steps are taken by the nodes (e.g. becoming a leader). The events are passed<br>through the filters one by one, and the corresponding action is taken if any of the filters match. If<br>none match, the default action is to deliver a message.

Lots of interesting questions come up with scenario tests. How does the test decide the order of messages<br>observed (since there is no single true order in distributed systems)? In the original paper, we allow<br>users to plug in their own strategy for exploration, but the default is random order. The second<br>question which I use the help of AI to answer is - Who writes the test? How do you know which tests are<br>needed or sufficient?

For the original paper, I manually wrote all the tests and we come up with a methodology to derive tests<br>from the protocol specification. Now with the advent of AI, I give the LLM the spec of a distributed<br>protocol, the DSL syntax and semantics, and instruct it to write as many tests as it can with the goal<br>of being exhaustive. As expected, we got a lot of meaningful and interesting tests that I'd missed when<br>I manually wrote them. But since etcd/raft is such a robust implementation, I did not find any bugs.

All the tests including the framework can be accessed here - zeu5/raft

You can find the paper (published in NETYS 2024) and all the detailed experiments here - arXiv (2303.05893)

#tech