Property-based testing | Liam DeVoe

Liam DeVoe

A curated list of property-based testing resources.

This list is intended as a "seminal overview". If one were to read every linked resource, they would be a well-rounded property-based testing expert, with knowledge of all the important ideas.

Libraries<br>Posts<br>Generation<br>Shrinking<br>Test synthesis<br>Tools<br>Empirical results<br>Other lists

Libraries

Python

Hypothesis

Rust

proptest

JavaScript / TypeScript

fast-check

Go

Rapid

gopter

Haskell

QuickChecknoteThe first recognizable PBT library. See also the paper QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs.

SmallChecknoteExhaustively enumerates its search space, incrementally expanding in size. See also the paper Smallcheck and lazy smallcheck: automatic exhaustive testing for small values.

Hedgehog

Java

jqwik

C++

RapidCheck

.NET

FsCheck

Erlang

PropEr

Scala

ScalaCheck

Clojure

test.check

Rocq

QuickChick

Elm

elm-test

Elixir

PropChecknoteBased on PropEr, an Erlang PBT library.

-->

Posts

Discussions of recurring themes in property-based testing.

Property-Based Testing Is Fuzzing argues that PBT and fuzzing are two different names for the same problem.

Two kinds of testing and its predecessor post Property Testing like AFL describe some desirable UX of property-based testing tools.

Reducers are Fuzzers argues that during the course of shrinking, one is also inadvertently fuzzing.

Fuzzers Need Taming describes the fuzzer taming problem: how to identify when two failing test cases are caused by the same bug?

What is Property Based Testing? gives one opinion on where property-based testing sits relative to fuzzing and other techniques.

Tutorials

Introductory material and property-based testing tutorials.

Zac Hatfield-Dodds and Ryan Soklaski's An Introduction to Property-Based Testing.

F# for Fun and Profit's Property Based Testing series introduces PBT; Choosing properties for property-based testing is commonly referenced.

Generation

Can we do better than sampling test cases uniformly at random? Yes; this section discusses various techniques for doing so. Many are adjacent to, or framed as part of, the fuzzing literature.

Swarm Testing and its corresponding blog post Better Random Testing by Leaving Features Out describes how randomly disabling a subset of the features in a program actually increases behavioral diversity and bug-finding power.

Inputs from Hell: Generating Uncommon Inputs from Common Samples observes that if you have a distribution of inputs over some grammar, you can invert the observed probabilities to generate inputs along rare grammar paths.

Targeted Property-Based Testing suggests that an ergonomic form of feedback to the search is to hill-climb toward increasing a developer-specified metric.

Shrinking

Shrinking (also known as minimization or test-case reduction) takes a complex failure and shrinks it to a simpler one. This eases debugging, among other things. Shrinking is an important and sometimes overlooked part of property-based testing.

Everything You Ever Wanted To Know About Test-Case Reduction, But Didn’t Know to Ask gives an overview of shrinking and the shrinking literature.

Test-Case Reduction via Test-Case Generation: Insights from the Hypothesis Reducer introduces internal shrinking: instead of shrinking the test case, shrink the choices made during the generation of that test case. This internal view avoids several classical challenges of shrinking.

falsify: Internal Shrinking Reimagined for Haskell describes a Hypothesis-inspired internal shrinking algorithm that represents test cases as structured trees. See also the blog post and the talk by the same name.

C-Reduce is a shrinker for C and C++ programs. See also the paper Test-Case Reduction for C Compiler Bugs and the blog post Design and Evolution of C-Reduce.

Shrinkray is a general-purpose shrinker that works over any file format.

The Shrinking Challenge compares the shrinking performance of property-based testing frameworks on a common benchmark suite.

Notes on Test-Case Reduction is an experience report from the implementation of shrinking. Best read after already familiar with the details of shrinking.

Test synthesis

Not satisfied with automating the generation of test cases through property-based testing, this section explores automating the process of writing property-based tests themselves.

The Hypothesis Ghostwriter generates a starting point for property-based tests using good old fashed heuristics. See also this in-browser demo.

A Claude Code command for Hypothesis and the related paper Finding bugs across the Python ecosystem with Claude and property-based testing automatically infers property-based tests and generators with LLMs.

Tools

Hegel is a family of PBT libraries built on a shared Hypothesis backend.

Bombadil is a property-based testing library for web UIs and TUIs.

Tyche is an observability tool for property-based testing. See...