Three Things You're Getting Wrong About Religion Data
SubscribeSign in
Three Things You're Getting Wrong About Religion Data<br>A rant from a guy who's been making graphs about it for ten years
Ryan Burge<br>Jul 27, 2026
129
25<br>15
Share
Article voiceover<br>0:00
-10:14
Audio playback is not supported on your browser. Please upgrade.Today’s post is going to be a bit different than the usual stuff you get on my newsletter. Instead of just focusing on a single question and making a bunch of graphs that try to figure out that causal puzzle, I am going to share a few things that I’ve learned after doing this kind of work for the last ten years.<br>I see people make assumptions about data and statistical analysis that just make me scratch my head. So, this is a rant of sorts. A short list of things that come up again and again when I post any type of graph on social media.<br>You better believe that I’m going to be dropping this link into my replies on dozens of occasions in the future.<br>The average rate of public opinion change is much slower than people anticipate.
I can’t tell you how many times I will post a graph that has data collected in 2024 and shows no major changes in public opinion and someone will reply, “If you had data from 2026, I’m sure it would show a lot of movement. ” That kind of response is just so divorced from how public opinion actually works that I feel like I need to take some time to really lay out how wrongheaded it is.<br>For those aspiring data analysts out there who are just learning how to code and make graphs, let me give you a piece of advice — if you see a survey question that has been asked with regularity over a long time move more than 3-4 points in a short window of time (less than two years), then you need to check your code again. It is highly unusual for stuff to shift that much that quickly.<br>Take, for instance, the General Social Survey’s question about allowing a woman to have access to an abortion if she wants one for any reason.
If you track the individual points over time, there’s one really weird aberration between 1978 and 1980 when support for abortion jumped 8 points. In terms of actual percentages, it was 33% to 41%. But here’s the other part of the story — in 1982 it was already down to 37% and then it declined to 33% in 1983. So a complete reversal in a span of five years. You see this same thing happen between 1991 and 1998 — a temporary rise, then all those gains were given back .<br>You know what the average change is year to year in this graph? It’s 2.6 points. Which is inside the margin of error for most samples. But as you can tell in how I annotated the graph — something happened between 2016 and 2018. I think we all know what I’m talking about right now. In that two-year window, support for access rose from 44.8% to 49.9%. A full five percentage points. Now, that could have just been polling noise, right? We would be right to be cautious about writing a story with a headline about the huge surge in pro-choice views.<br>But here’s how we know that’s the real deal — the trend has continued. In 2021 it rose again to 53.1%, up nearly four more points from the prior survey. Then in the last two years it was 57% and 58% respectively. Which is more plausible: that every year of the GSS from 2018 through 2024 has just “missed” on this question in the same way? Or did an external event (or series of events) lead to a macro-level change in opinion?<br>Also, don’t let me forget to point out the fact that the last couple of surveys have “calmed down” on the movement after a whole bunch of rapid change. There was a total of ten points of change in four decades of polling . Then, we see a clear directional shift that can be easily traced to an external set of political events.<br>The average person thinks that THEIR world is THE world.
I get a version of this email about once a week. A reader, usually a thoughtful one, writes in with a question they’re sure is right in my wheelhouse. Here’s a recent example: “Have you worked on the question of whether the error of Christian Zionism and/or Dispensational theology is waxing or waning in American or global Christianity? Seems right up your alley.”<br>I understand why people think this is a survey question. It’s about religion. It’s about change over time. Those are the two things I write about every single week. The person that emailed is probably an evangelical pastor and went to seminary. They can read Hebrew and they listen to podcasts about minute details of Koine Greek. A lot of their friends are pastors and they stay up late at night debating pre-trib vs. post-trib.<br>In other words, they come from a very specific subset of the American population. Which makes a question like this basically unanswerable with survey data, and the reasons why tell you a lot about what surveys can and cannot do.<br>Think about what happens if I field a general population survey of 1,000 Americans (which would cost thousands of dollars) and try to say...