Exploring Kendall's \(\tau\) - iwhalen.com

Skip to content

Categories

Exploring Kendall's \(\tau\)

For one reason or another, I've been working with Kendall's \(\tau\) lately. This post explores a little bit of my excitement with the statistic and some possible extensions. Disclaimer: I am not a statistician.

The code used to created the plots in this blog can be found here.

Kendall who?

Wikipedia tells me Sir Maurice George Kendall was a statistician in the 20th century. Among other things, he was interested in rank correlations between to variables in the context of psychology. He wanted to quantify the similarity of two observers' rankings of quality of some events or individual.

For example, imagine we want to compare the rankings of movies between two different review sites. Maybe site A uses a 1 through 5 rating and site B uses a 1 through 10 rating. Let's say we compare 100 movies' ratings on both sites. Kendall's \(\tau\) quantifies the "agreement" of the ranking of those movies from 1 to 100 based on both site's scores.

Toy example

I'm sure this has been beaten into the ground elsewhere, but I like the math for this statistic. So I'm going to show a toy example.

Let's continue our movie comparison example. Let's say we just have three movies. Their rankings from each site are plotted below.

In this plot we have, approximately,

\[<br>\begin{align}<br>x_1 &= 2.30 &~ y_1 &= 8.21 \\<br>x_2 &= 2.93 &~ y_2 &= 7.92 \\<br>x_3 &= 4.65 &~ y_3 &= 9.53 \\<br>\end{align}<br>\]

Looking at a single point doesn't tell us much about the ranking agreement. However, looking at pairs of points would tell us if the ranking of two movies is in agreement. Since we have three movies, there are \(\binom{3}{2} = 3\) total pairs of points to compare.

First we can look at the ranking for movie 1 and 2.

Site A has and average rating of \(2.30\) and \(2.93\) for movies 1 and 2, respectively.

Site B has and average rating of \(8.21\) and \(7.92\) for movies 1 and 2, respectively.

We can see that the ordering doesn't match here. \(2.30 but \(8.21 > 7.92\). The sites disagree on the ordering of these two pairs based on average rating. In other words, this pair is discordant.

On the other hand, if we look at the ranking for movie 1 and 3, we get a concordant pair for site A and B. Namely,

If we look at all the other pairs of points, we get a total of 1 discordant and 2 concordant pairs. Explicitly, all three comparisons are:

Pair<br>X comparison<br>Y comparison<br>Type

\(\left[ (4.65, 9.53), (2.93, 7.92) \right]\)<br>\(4.65 > 2.93\)<br>\(9.53 > 7.92\)<br>Concordant

\(\left[ (4.65, 9.53), (2.30, 8.21) \right]\)<br>\(4.65 > 2.30\)<br>\(9.53 > 8.21\)<br>Concordant

\(\left[ (2.93, 7.92), (2.30, 8.21) \right]\)<br>\(2.93 > 2.30\)<br>\(7.92<br>Discordant

So now we have these counts. Of course, the raw values don't tell us that much and don't let us compare across methods that might have different absolute counts.

Some measures we might care about could be concordance ratio (\(0.67\)) or discordance ratio (\(0.33\)). Kendall's \(\tau\) combines these into a single number calculated by taking

\[<br>\frac{\text{# concordant pairs} - \text{# discordant pairs}}{\text{# pairs}}<br>\]

This value ranges from -1 (complete disagreement) to 1 (complete agreement). For our toy problem, \(\tau = 0.33\).

Kendall's \(\tau\) measures exactly what we would want in this kind of problem. This toy problem essentially walked through some first principal ideas behind the statistic. Namely, there's a bunch of pairs of points and we want to know how often they are properly ordered.

Below are some examples of what different plots might look like for various values of \(\tau\)1.

Anyone familiar with regular old correlation shouldn't be surprised by these plots.

Math time

Now let's be annoying and fire up the ol' MathJax.

There are a few ways to define Kendall's \(\tau\). The easiest one we just looked at is the original definition, sometimes called Kendall's \(\tau_a\). The set counting definition is very understandable. But we'll make it more explicit here.

First, we define the set of all combinations of indices as

\[<br>P = \{(i, j) \mid 1 \leq i If we define our concordant pair set

\[<br>C = \{ (i, j) \in P \mid (x_i - x_j)(y_i - y_j) > 0 \}<br>\]

and our discordant pair set

\[<br>D = \{ (i, j) \in P \mid (x_i - x_j)(y_i - y_j) then

\[<br>\tau_a = \frac{|C| - |D|}{|P|}<br>\]

Note our simplification of the concordance and discordance logic. Instead of writing out individual inequalities, we simply write out \((x_i - x_j)(y_i - y_j)\), which will be positive if both \(x_i > x_j\) and \(y_i > y_j\) or both \(x_i and \(y_i . This is the definition of concordance. The same idea holds for discordance.

However, we've clearly glossed over the case where \((x_i - x_j)(y_i - y_j) = 0\).

Let's call it a draw

In our movie example, its unlikely any two movies would have the same rating in each system. The two site's systems use different scales and are floating point values after averaging. However, in...