Breaking down the winner’s curse: Lessons from brain-wide association studies | The Transmitter: Neuroscience News and Perspectives
Perspectives
fMRI
Add us as a Preferred Source on Google
Set us as a Preferred Source to see The Transmitter more prominently in your Google Search results.
Breaking down the winner’s curse: Lessons from brain-wide association studies
We found an issue with a specific type of brain imaging study and tried to share it with the field. Then the backlash began.
By
Nico Dosenbach, Scott Marek
25 March 2024 | 6 min read
comments
https://doi.org/10.53053/SCFX4454
Cite this article
https://doi.org/10.53053/SCFX4454 https://doi.org/10.53053/SCFX4454 - opens a new tab
Cite this article
Study size: some types of fMRI studies benefit from very large sample sizes.
Illustration by<br>Mari Fouz
Nico Dosenbach
Associate professor of neurology<br>Washington University School of Medicine
Scott Marek
Assistant professor of radiology<br>Washington University School of Medicine in St. Louis
Share this article:
Facebook - opens a new tab
Linkedin - opens a new tab
Mail - opens a new tab
Share more - opens a new tab
Tags:
fMRI,
Audio research news,
GWAS,
Reproducibility
Listen to this story:
Your browser doesn't support the audio element
Back 15 seconds
Play
Pause
Forward 15 seconds
0:00
Spotify<br>Apple<br>Apple - opens a new tab
Spotify<br>Spotify<br>Spotify - opens a new tab
YouTube
Youtube<br>Youtube - opens a new tab
Download
Share on social media
x twitter
Copy
In 2022, we caused a stir when, together with Brenden Tervo-Clemmens and Damien Fair, we published an article in Nature titled “Reproducible brain-wide association studies require thousands of participants.” The study garnered a lot of attention—press coverage, including in Spectrum, as well as editorials and commentary in journals. In hindsight, the consternation we caused in calling for larger sample sizes makes sense; up to that point, most brain imaging studies of this type were based on samples with fewer than 100 participants, so our findings called for a major change.
But it was an eye-opening experience that taught us how difficult it is to convey a nuanced scientific message and to guard against oversimplifications and misunderstandings, even among experts. Being scientific is hard for human brains, but as an adversarial collaboration on a massive scale, science is our only method for collectively separating how we want things to be from how they are.
The paper emerged from an analysis of the Adolescent Brain Cognitive Development (ABCD) Study, a large longitudinal brain-imaging project. Starting with data from 2,000 children, Scott showed that an average brain connectivity map he made using half of the large sample replicated almost perfectly in the other half. But when he mapped the association between resting-state activity—a measure of the brain during rest—and intelligence in two matched sets of 1,000 children, he found large differences in the patterns. Even with a sample size of 2,000—large in the human brain imaging world—the brain-behavior maps showed poor reproducibility.
For card-carrying statisticians, the result was not surprising. It reflected a pattern known as the winner’s curse, namely that large cross-sectional correlations can occur by chance in small samples. Paradoxically, the largest correlations will be “statistically significant” and therefore most likely to be published, even though they are the most likely to be wrong.
Smaller-than-expected effect sizes were old news in genome-wide association studies (GWAS), which had been forced more than a decade ago to push their sample sizes higher, eventually into the millions. But the neuroimaging field lagged behind—despite warnings about sampling variability in 2009 from neuroimaging statistician Tal Yarkoni.
Our finding offered a clear example of the problem, and we felt it was important to share with the field. But it was a difficult message for the community to hear. Indeed, when we decided to publish the finding, prominent collaborators asked to be removed from the manuscript, because they did not want to be associated with its message. Bracing for more fallout, we started to call it the “Manhattan Project.”
e knew functional MRI (fMRI) researchers sensitized to bad press in the past might project their worst fears onto our article. Several high-profile papers in the field, including one on a dead salmon, have identified statistical issues with analyses of fMRI data, calling into question some published findings, sometimes overly broadly. So we took great pains with the abstract, praising the transformational power of fMRI research, for example. Our findings applied to a specific type of fMRI study, which we coined brain-wide association studies (BWAS), rather than to classical brain-mapping studies, and we tried to clearly outline the difference. Classical fMRI activation studies look for an association...