Confidence Sets, Confidence Intervals
Notebooks
Confidence Sets, Confidence Intervals
Last update: 28 May 2026 09:13
First version: 31 August 2022
This is, to my mind, one of the more beautiful and useful ideas<br>in statistics, but also one of the more tricky.<br>(I might admire the idea more because of the trickiness.)
We have some parameter of a stochastic model we want to learn about,<br>proverbially \( \theta \), which lives in the parameter space \( \Theta \). We<br>observe random data, say \( X \). The distribution of \( X \) changes with \(<br>\theta \), so the probability law is \( P_{\theta} \). Our game is one of<br>"statistical inference", i.e., we look at \( X \) and make a guess about \(<br>\theta \) on that basis. One type of guess would be an exact value for \(<br>\theta \), a point estimate. But we'd basically never expect any<br>point estimate to be exactly right, and we'd like to be able to say something<br>about the uncertainty. A level \( \alpha \) confidence set is<br>a random set of parameter values \( C_{\alpha} \subseteq \Theta \)<br>which contains the true parameter value, whatever it might happen to<br>be, with probability \( \alpha \) (at least):
\[<br>\min_{\theta \in \Theta}{P_{\theta}(\theta \in C_{\alpha})} \geq \alpha<br>\]
We say that \( C_{\alpha} \) has coverage level \( \alpha \).
Quibbles:
It's (pragmatically) implied that the coverage probability is \( =\alpha \) for at least some \( \theta \); if the probability is \( > \alpha \) for all \( \theta \), we say the confidence set is "conservative".<br>If you know enough to quibble about "min" vs. "inf", you also know what I<br>meant.<br>\( C_{\alpha} \) is really \( C_{\alpha}(X) \), a (measurable) function of the data, but I am trying to keep the notation under control.<br>In many situations there will be other ("nuisance") parameters we don't care about, canonically \( \psi \), and then we have to consider the worst case over both \( \theta \) and \( \psi \) simultaneously, even if really only want to draw inference about \( \theta \).
Either the confidence set contains the truth, or we were really unlucky
Now, confidence sets are notoriously hard for learners to wrap their minds<br>around, but I have a way of explaining them which seems to work when I<br>teach, and so I might as well share.
When I construct a confidence set from our data, I am offering you, the<br>reader, a dilemma: Either
the true parameter value is in the confidence set \( C_{\alpha} \), or<br>we were very unlucky, and we got data that was very improbable (\( P \leq<br>1-\alpha \) and unrepresentative under all values of the parameter.
The second fork of the dilemma obtains because the event \( \theta \not\in<br>C_{\alpha} \) clearly has probability at most \( 1-\alpha \), regardless of \( \theta \).
(More strictly there is really a tri-lemma here:
the true parameter value is in the confidence set \( C_{\alpha} \), or<br>we were very unlucky, and we got data that was very improbable (\( P \leq<br>1-\alpha \) and unrepresentative under all values of the parameter, or<br>the model we're using to calculate probabilities is wrong.
But even interpreting parameters in mis-specified models is hard, and I don't<br>want to pursue the third fork [tine?] of the trilemma here.)
The confidence set is every parameter value we can't reject
At this point a very reasonable question is to ask how on Earth we're<br>supposed to find such a set. Here is one very general procedure. Suppose that<br>we can statistically test whether \( \theta = \theta_0 \). That is, we have<br>some function \( T(X;\theta_0) \) which returns 0 if \( X \) looks like it<br>could have come from \( \theta=\theta_0 \), and returns 1 otherwise. More<br>concretely, \( P_{\theta_0}{(T(X;\theta_0) = 1)} \leq 1-\alpha \), so the "false<br>positive" rate or "false rejection" rate is at most \( 1-\alpha \). (That is,<br>the "size" of the test is at most \( 1-\alpha \), over all parameter values.)<br>Now building \( C_{\alpha} \) is very easy:<br>\[<br>C_{\alpha}(X) = \left\{ \theta \in \Theta ~ : ~ T(X;\theta) = 0 \right\}<br>\]<br>(Here I am being explicit that \( C_{\alpha} \) is a function of the data \( X \), which I otherwise suppress in the notation.)
In words: the confidence set consists of all the parameter values we<br>compatible with the data, i.e., all the parameter values we can't reject (at<br>any acceptably low error rate \( 1-\alpha \) ).
This construction is called "inverting the hypothesis test". Clearly, any<br>hypothesis test gives us a confidence set, by inversion. Equally clearly, any<br>confidence set can be used to give a hypothesis test: to test whether \( \theta<br>= \theta_0 \), see whether \( \theta_0 \in C_{\alpha} \); the false-rejection<br>rate of this test is, by construction, \( \leq 1-\alpha \).
It is a little less clear that every confidence set can be<br>constructed by inverting some test, but it's nonetheless true, and<br>a textbook result (see, e.g., Casella and Berger, or Schervish). This is called the "duality between hypothesis tests and confidence sets".
Consistency and...