Beware Change-Blocking Priors

Beware Change-Blocking Priors - by Robin Hanson

Overcoming Bias

SubscribeSign in

Robin Hanson Jun 28, 2026

“Faced with the choice between changing one’s mind and proving that there is no need to do so, almost everyone gets busy on the proof.” John Kenneth Galbraith

When trying to resist evidence offered for contrarian claims, many invoke the mantra: “extraordinary claims require extraordinary evidence”. Sometimes they just mean to point to Bayes’ Rule, where posterior beliefs are renormalized products of prior beliefs and evidence likelihoods, so that unusually low priors must be countered by unusually strong evidence. But at other times, those who invoke this mantra point instead to how many big implications a claim would have if true, for personal and social practices, and for other practical beliefs. For example, this has been invoked by skeptics of cold fusion, room temperature superconductors, COVID lab leak, high-intensity interval training, low marginal medical effectiveness, big battery breakthroughs, cryonics, life on Mars, and UFOs as aliens. In the past, such arguments have been offered against believing in meteorites, rogue waves, the platypus, dirty-hand docs infecting via germs, atomic nucleus energies tapped, ball lightning, quasi-crystals, prions, continental drift, bacteria causing ulcers, endosymbiosis, and human level AI. Why might people do this? Consider the task of securing a large facility against theft. In that case you’d want to make the costs of stealing any particular thing larger than how much a thief could gain by stealing it. Extraordinarily valuable items would need extraordinarily strong security. Now consider an alliance of groups who get government, philanthropic, or customer funding. For example, an alliance of academic groups who get research funding. Each group’s funding is based on some supporting beliefs, and so might be threatened if those beliefs were undermined. Such an alliance might want push for priors that ensured that the larger was a group whose funding might be cut if a particular belief was adopted, the stronger their prior against that belief, to prevent such adoption. I’d call these “change-blocking priors”, as their function is to prevent social change. And others who hold related beliefs might also want to supports such priors, even if they don’t share in this funding, if they’d dislike the extra mental work to rethink many beliefs, if contrarian claims were to win out. Change-blocking priors protect both status quo social relations, and also status quo mental habits. The problem of course is that a claim having big implications for our lives or practical beliefs is just not the same as that claim having a low prior, at least in terms of priors that reasonably follow from all else we know. (Yes, claims that undermine our well-established deep theories are different.) In fact, contrarian claims which would have big implications for our lives, yet have only modestly low priors, seem actually to be especially promising as targets of study, if our goal is to learn consequential things about our world. (Much of my research has been like this.) Making it especially a problem when alliances push for change-blocking priors. The deeper problem is that such alliances have many tools to enforce their change-blocking priors, such as ways to exclude and diss those who explore non-conforming alternatives. Prediction markets offer a reward for contrarians to pursue alternatives in the face of such conformity pressures, but only when they are legal, and when there’s a wider norm of believing their prices, and “putting your money where your mouth is” when you claim that prices are wrong. Note that folks eager to cause social and cognitive change do have stronger incentives to fake evidence supporting claims with bigger social and cognitive implications. This is a reason to want especially honest evidence there. But is honestly really that extraordinary?

Discussion about this post CommentsRestacks

Anon User Jun 29Edited

In a purely intellectual world, this might be reasonable. But I real world - you need to also be asking whether the evidence is real and honest vs fake or selectevely reported. Everything else being equal, in cases where accepting the evidence would result in large wins for somebody, the likelihood of the evidence being compromised is higher, so it is still reasonable to require a higher burden of proof.

1 reply by Robin Hanson

Leon Voß Jun 29

How do you even apply prediction markets to important questions academics lie about, such as HBD? Which markets right now can an HBD person make money on? Seems like none.

Beware Change-Blocking Priors

Related Articles

(no title)

Scientists reverse brain aging, with a nasal spray

AI has torched the market for junior programmers

Al Vigier: Canada's AI strategy shouldn't include secret Palantir bills

The Anatomy of an AI-Native Org