The Metaculus Democracy Threat Index - by Scott Alexander
Astral Codex Ten
SubscribeSign in
The Metaculus Democracy Threat Index<br>...
Scott Alexander<br>Jun 25, 2026
132
228
Share
In recent posts on Trump and dictatorship, people have asked me - how do you know you’re not suffering from Trump Derangement Syndrome?<br>I take this seriously; we’ve all lost loved ones to this condition. The best check on my reasoning would be an objective measure of the health of American democracy. There are several “democracy indices” that purport to do this, but they have a mixed reputation. My impression is that most current accusations of bias are relatively weak - I agree with Claude’s analysis here - but they rely enough on “expert” opinion that I don’t expect them to convince a skeptic.<br>The newest entrant in this space - Metaculus Democracy Threat Index - works differently, and deserves a closer look.<br>Metaculus is a prediction site - like a prediction market, except that no money changes hands. People can record their guesses for how future events will turn out, which get aggregated by an algorithm (currently just a recency-weighted median, although they’ve done fanicer things in the past).<br>Their Democracy Threat Index is a collection of 153 questions relevant to US democracy. For example:
This question says there’s a 3.5% chance that a political party will keep an opponent off the ballot in a state election in 2026. You can see that 86 people have made forecasts about this. When probabilities like this go up on the 153 questions, the index gets higher.<br>This has some advantages over traditional democracy index design. It’s transparent: you can see all the questions and how they fit together. It’s crowdsourced, so there’s limited opportunity for ideologues or biased experts to put their fingers on the scale. It does a good job limiting itself to things which naturally seem democracy-related, resisting the pressure to add “and they support my preferred policy” to the definition of democracy.<br>But the biggest change is that Metaculus leverages modern forecasting science (popularly called “superforecasting” - although I think the people with that trademark would be unhappy to hear me use it this way). Does this help?<br>We care most about questions like “Did they cancel elections?” or “Did they murder protesters?” But these are coarse binary outcomes - when someone wants to know if democracy is “under threat”, they want to know when there’s increased chance of these things happening, even though they haven’t happened yet. As a forecasting platform, Metaculus can distinguish not just between “keeps elections” vs. “cancels elections”, but between 10% chance of cancelling the next elections vs. 50% chance. This both lets them focus on the big questions (rather than the small questions that are most likely to differ between one mildly-concerning regime and another) and give finer-grained estimates (a 10% vs. 50% chance of elections getting cancelled, rather than just ‘not cancelled yet’).<br>What are the remaining risks/biases?<br>Susceptibility to crowd attack: If a hundred Democrats join Metaculus and give maximally pessimistic answers to all democracy-related questions, would that make Trump look worse? Currently Metaculus has near-perfect security through obscurity - nobody cares about this index enough to attack it. I asked them what happened if that changed, and they said they had good security, and that medians are naturally more secure than means to this kind of threat. I’m still disappointed to have to rely on the security of one centralized site.
These problems could be solved by transitioning from Metaculus to a true prediction market. Prediction markets’ main advantage over Metaculus-style forecasting engines is resilience to attack (because new bettors are incentivized to come take the attacker’s money). Unfortunately right now Metaculus is focusing on being a responsible academic institution, and prediction markets are focusing on attracting sports-gambling-obsessed degens, so we’ll have to settle for the former.<br>That still leaves one possible source of risk:<br>Susceptibility to question selector bias: Who decides which questions get added to the Index? Right now, it’s a “nonpartisan group” called Bright Line Watch. Are they really nonpartisan? You can see some discussion here, but even if we trust them, it’s disappointing that the question of trust has to come up at all, given the otherwise-trustless design.
I thought about this most when seeing the question about whether government departments will change policy in response to bribes, which mentions as a possible bribe vector an “expenditure to a cryptocurrency” - for example, some high official creates a crypto token, and the would-be-bribe-giver buys into it. Democrats and Republicans are corrupt in different ways, and privileging memecoin buyers is so far a uniquely Republican form of corruption. If they’d missed that vector, the bribery question would have been...