Hypothesis
Hypothesis
The property-based testing library for Python
What is Property Based Testing?
May 14, 2016
David R. MacIver
I get asked this a lot, and I write property based testing tools for a living, so you'd think<br>I have a good answer to this, but historically I haven't. This is my attempt to fix that.
Historically the definition of property based testing has been "The thing that<br>QuickCheck does". As<br>a working definition this has served pretty well, but the problem is that it makes us unable<br>to distinguish what the essential features of property-based testing are and what are just<br>accidental features that appeared in the implementations that we're used to.
As the author of a property based testing system which diverges quite a lot from QuickCheck,<br>this troubles me more than it would most people, so I thought I'd set out some of my thoughts on<br>what property based testing is and isn't.
This isn't intended to be definitive, and it will evolve over time as my thoughts do, but<br>it should provide a useful grounding for further discussion.
There are essentially two ways we can draw a boundary for something like this: We can go<br>narrow or we can go wide. i.e. we can restrict our definitions to things that look exactly<br>like QuickCheck, or things that are in the same general family of behaviour. My inclination<br>is always to go wide, but I'm going to try to rein that in for the purpose of this piece.
But I'm still going to start by planting a flag. The following are not essential features<br>of property based testing:
Referential Transparency.
Types
Randomization
The use of any particular tool or library
As evidence I present the following:
Almost every property based testing library, including but not limited to Hypothesis and<br>QuickCheck (both Erlang and Haskell).
The many successful property based testing systems for dynamic languages. e.g. Erlang<br>QuickCheck, test.check, Hypothesis.
SmallCheck. I have mixed feelings about<br>its effectiveness, but it's unambiguously property-based testing.
It's very easy to hand-roll your own testing protocols for property-based testing of a<br>particular result. For example, I've previously done this for testing a code formatter: Run it over a corpus (more on<br>whether running over a corpus "really" counts in a second) of Python files, check whether<br>the resulting formatted code satisfies PEP8. It's classic property-based testing with an<br>oracle.
So that provides us with a useful starting point of things that are definitely property based<br>testing. But you're never going to find a good definition by looking at only positive examples,<br>so let's look at some cases where it's more arguable.
First off, lets revisit that parenthetical question: Does just testing against a large corpus count?
I'm going to go with "probably". I think if we're counting SmallCheck we need to count testing<br>against a large corpus: If you take the first 20k outputs that would be generated by SmallCheck<br>and just replay the test using those the first N of those each time, you're doing exactly the<br>same sort of testing. Similarly if you draw 20k outputs using Hypothesis and then just randomly<br>sample from them each time.
I think drawing from a small, fixed, corpus probably doesn't count. If you could feasibly<br>write a property based test as 10 example based tests in line in your source code, it's<br>probably really just example based testing. This boundary is a bit, um, fuzzy though.
On which note, what about fuzzing?
I have previously argued that fuzzing is just a form of property-based testing - you're testing<br>the property "it doesn't crash". I think I've reversed my opinion on this. In particular, I think<br>the style of testing I advocate for getting started with Hypothesis, probably doesn't<br>count as property based testing.
I'm unsure about this boundary. The main reason I'm drawing it here is that they do feel like<br>they have a different character - property based testing requires you to reason about how your<br>program should behave, while fuzzing can just be applied to arbitrary programs with minimal<br>understanding of their behaviour - and also that fuzzing feels somehow more fundamental.
But you can certainly do property based testing using fuzzing tools, in the same way that you<br>can do it with hand-rolled property based testing systems - I could have taken my Python formatting<br>test above, added python-afl to the mix, and<br>that would still be property based testing.
Conversely, you can do fuzzing with property-based testing tools: If fuzzing is not property<br>based testing then not all tests using Hypothesis, QuickCheck, etc. are property based tests.<br>I'm actually OK with that. There's a long tradition of testing tools being used outside their<br>domain - e.g. most test frameworks originally designed as unit testing tools end up getting<br>being used for the whole gamut of testing.
So with that in mind, lets provide a definition of fuzzing that I'd like to use:
Fuzzing is feeding a piece of code...