Kuiper Q-Q plot: are these the same?

When we tried to get an intuition for the differences in distribution functions,<br>we learned that children subjected to a treatment had greater variation in their<br>score than the reference group.

However, we couldn&rsquo;t tell if this difference was large enough to be a useful<br>guide for making decisions. It could just be an artifact of the small sample<br>size.

Summary

In case you know the drill already, just paste your data into the tool and get<br>on with your life. I wish I had made this long ago! That tool shows a q-q plot<br>for visual comparison, it computes Kuiper&rsquo;s test statistic, and has a handy<br>reference table for significance thresholds.

It lets you choose between multiple theoretical reference distributions. Or you<br>can input your own reference distribution, either as an empirical distribution<br>function known exactly, or a reference sample with embedded uncertainty. The<br>theoretical reference distributions that support it come with aic values for<br>model selection.

If none of the above made any sense to you, read on.

Tests of distribution similarity

Many quantitative questions in life take the same general shape:

We have some maybe-special data.

We know such data commonly follows some distribution.

We want to know if the maybe-special data belongs to the common case, or if<br>it is indeed special in some way.

We tend to have a good idea of what happens generally – this is a reference<br>distribution we know with certainty (in our example, what the scores are for the<br>population in general). We then have a smaller set of data points – a sample –<br>which we think might be special in some way (in our example, the scores of a<br>treatment group), but we&rsquo;re not sure. To get more sure, we want to perform some<br>sort of test that can tell us if our sample is actually special. If the test<br>fails, we&rsquo;ll assume our sample is a small manifestation of the reference<br>distribution.

Such a test would be like a black box with two slots and a light bulb. Into the<br>big slot, we put a reference distribution. In the smaller slot, we slide in our<br>sample. Then the box hums for a brief moment, and when it goes silent, one of<br>two things can happen. Either

the light bulb lights up! In which case the box has determined, with high<br>certainty, that the sample did not come from the reference<br>distribution11 Okay, this is a frequentist test, so technically what the box<br>has determined is that if repeated, similar-sized draws were made from the<br>reference distribution, the test statistic would be less extreme in more than<br>95 % of them. But frequentist confidence intervals commonly approach Bayesian<br>posterior credible intervals, so I&rsquo;m going to gloss over this difference.<br>Albers, Kiers, and Ravenwaaij&rsquo;s 2018 argument is the commonly cited defense.; or

the light bulb stays off, in which case the box cannot tell the sample and the<br>reference distribution apart.

It&rsquo;s important to note that in the second case, the box does not claim the<br>sample came from the reference distribution. That&rsquo;s impossible to prove, because<br>it may come from something that&rsquo;s so subtly different it isn&rsquo;t detectable with<br>limited data. The lamp staying dark only means the sample wasn&rsquo;t different<br>enough from the reference distribution to tell them apart. This can happen<br>because they are the same, or because they are very-similar-but-not-the-same,<br>but also because the sample is too small to achieve certainty in the difference,<br>even if the difference is quite large.

False positives and false negatives

Thanks to the random nature of … things, such a test will sometimes show the<br>wrong result. It can indicate our sample is special even when it is not (a false<br>positive; a type I error). It may also fail to indicate that our sample is<br>special even when it is (a false negative; a type II error).

Since type I and and type II errors trade off against each other, we cannot set<br>targets for both types of errors to be low. Instead, we set a target for a low<br>type I error (i.e. low false positive rate), and then we look for a test that<br>has as low a false negative as possible within that constraint. The reason we<br>focus on false positives is because we want to be fairly confident before we<br>claim a sample is special, so we prioritise rarity of false positives.

It is very common in science to pick a test that gives a false positive once<br>every 20 uses of the test, but the appropriate choice threshold depends on what<br>decisions are powered by the conclusion. For the sake of argument, let&rsquo;s say we<br>want a type I error rate of 10 % – once in every ten applications gives a false<br>positive.

Then of the tests with that false positive rate, we are looking for a test with<br>as low type II error rate as possible, or, in statistical terminology, a test<br>with high power. The statistical power of a test is its ability to<br>successfully detect that a sample is special, even if it&rsquo;s only very little<br>special...