When the best decision is no decision: the rise of randomization in grant funding | Nature
Skip to main content
Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain<br>the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in<br>Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles<br>and JavaScript.
Advertisement
Bluesky
Find a new job
Save article
View saved research
Randomizing parts of grant decision-making increases diversity and reduces costs, argue some researchers.Credit: Andrew Angelov/Alamy<br>Rachel Heyard’s job was to find a signal in the noise, but the trouble was how often there wasn’t one. As a biostatistician in the data team at the Swiss National Science Foundation (SNSF) in Bern, Switzerland’s main public research funder, Heyard faced a tricky problem. Reviewers and panel members scored each grant application against criteria such as scientific relevance, feasibility and the applicant’s track record, grading the proposal on a scale from A to D. Her team’s task was to test how much those scores could be trusted: to determine which applications truly outranked the others in the competition for limited funds.<br>The strongest and the weakest proposals were sorted readily enough. But the rest piled up in the messy middle — what she calls “a cloud of B proposals” — clustered so tightly in score that panels struggled to differentiate them. To force a ranking, they sharpened the grades with pluses and minuses, splitting hairs that might not have existed.<br>Grant proposals drafted with AI help more likely to win NIH funding
The stakes were high at the SNSF, which in 2021 was handing out the equivalent of about US$1 billion annually to projects across every discipline. Each year, the cut-off between the financed proposals and the rejected ones ran straight through the crowded middle. Where a proposal fell could shape a laboratory’s future or a young researcher’s career.<br>Heyard helped to build a statistical framework to measure what the reviewers’ scores were actually worth. It treated each proposal’s standing not as a fixed rank, but as a range that reflected the uncertainty in a panel’s judgement. Instead of pretending that one proposal was exactly 23rd and another precisely 27th, the model asked whether those apparent ranks could really be distinguished.<br>Near the funding line, they often could not. When a proposal’s possible ranks crossed the threshold between funded and rejected, it fell into the grey zone, good enough to stay in contention but too close to its neighbours to justify a precise ordering.<br>Fortune favours the fundable<br>Starting in 2022, the SNSF stopped treating these rankings as exact, and implemented a new approach. A mathematical model identified the proposals that were too close to separate and sent them into a lottery — a tie-breaker the foundation had previously trialled. Peer review still set the bar to qualify, but among the proposals near the funding threshold, a financed project and a rejected one would now be divided by chance. “Instead of saying, ‘should we introduce randomness into science processes?’, we’re reframing this question into ‘are we confident that our decisions are precise enough?’” says Heyard, now a researcher at the University of Zurich’s Center for Reproducible Science and Research Synthesis in Switzerland. “And the answer to that question is often no.”<br>The research backs up Heyard’s opinion. When 43 reviewers judged the same 25 applications in a 2018 study1, they barely agreed. “We already know that the awards are a lottery,” says Tom Stafford, a cognitive scientist at the University of Sheffield, UK, who is on secondment at the Research on Research Institute in London, which studies how science is funded. “We should just make it a formal lottery, rather than an unofficial one.”<br>After judgement has reached its limit, handing a decision to chance might sound like surrender, but it is one of the oldest tools of fair government. Ancient Athens filled many public offices and juries by drawing lots, pulling out the names of eligible citizens using a stone slab called a kleroterion. What these funders propose might be less a break with tradition than a return to it.<br>From splitting hairs to drawing straws<br>The Research on Research Institute keeps a catalogue of such trials and has logged more than a dozen funders that have used or tested partial lotteries, with 11 catalogue entries since 2022. The practice remains uncommon in global terms but is no longer hypothetical, spanning national agencies, charities such as Wellcome in London and the Novo Nordisk Foundation in Hellerup, Denmark, as well as universities. Individual awards range from the equivalent of a few thousand US dollars to more than $1 million. Stafford and others in the field have no usable global data on how...