The Art of Asking Questions—Asterisk

Everyone seems to agree that self-report questions are fraught with lies, biases, errors, and other inaccuracies. We all use them anyway. How can we ask them better?

It was 2016, and Professor Tony Greenwald was peering into the future. It was a heady time for election prognostication: FiveThirtyEight was in its prime, the New York Times had begun revving up its “election needle,” and Princeton’s neuroscientist-cum-forecaster Sam Wang, after cramming every poll into a mathematical model, claimed he would eat a bug if his predictions were wrong.<br>But Greenwald wasn’t satisfied with these traditional prediction methods. Pollsters collect data about voter preferences by asking people a series of questions. First, they try to measure how likely the respondent is to vote: “How much thought have you given to the upcoming election?,”  “Do you happen to know where people in your neighborhood go to vote?,” “Did you vote in the last election?,” etc. Then they ask about which candidate the respondent plans to vote for.<br>This approach relies on what social scientists call “self-report”: a person’s verbal description of what’s going on inside their head.1<br>Self-reports are quick and easy to get — you simply ask someone a question. But they can also mislead you. What if people lie? What if they’re ashamed to admit which candidate they support (as many worried might happen with Trump supporters)? What if they don’t really know whom they’re going to vote for, and they make something up?<br>Enter Greenwald and his Implicit Association Test (IAT), a psychological tool that uses reaction times to measure how tightly certain concepts are tied together in a person’s mind. If you’re slower to put Black faces and positive words together than you are to put white faces and positive words together, the thinking goes, then “Black” and “good” are less closely associated in your mind. The IAT is one of many tools that social scientists have designed to bypass self-report. Rather than ask people to speak, these tests observe people’s bodies and behaviors — how quickly they push a button, how much they sweat, how quickly they breathe, or where the blood flows in their brains. These measures cost more time and money than self-reports, but they promise a different angle into people’s minds, granting access to answers that are difficult — if not impossible — to fake.<br>Greenwald’s election IAT was returning some surprising results. According to the test, those who had favored a Republican other than Donald Trump in the primary appeared to have negative associations with Trump, even if they claimed they planned to vote for him — that is, it took them longer to associate Trump with a positive word than it took for other candidates.. But people who had supported Bernie Sanders in the Democratic primary showed equally positive associations with Hillary Clinton. “When we used both polling-type questions and the IAT, it became instantly clear that spoken and unspoken measures are no longer in sync,” Greenwald said in a November 2016 interview. “The message of these findings is that Clinton has unspoken support that is likely being missed by the polls. I’m going out on a limb to predict that Clinton’s vote margin on November 8 will exceed the prediction of the final preelection polls.”<br>A short time later, of course, that limb broke. Trump exceeded predictions while Clinton underperformed them, and the rest is history.<br>Why did Greenwald’s prophecy fail to come true? The whole reason to bypass self-report was to avoid the booby traps that lie between people’s minds and mouths. Methods like the IAT are  supposed to offer a glimpse into people’s heads, to uncover the attitudes and beliefs that people wouldn't explicitly admit — perhaps that they didn't even know they had. Instead of getting us closer to the truth, that glimpse led us even farther away. What gives?

Jordan Awan

If you wanna know if he loves you, it's in his pupil dilation

Social scientists have a complicated relationship with self-reports. On the one hand, we use self-reports all the time. Whenever a pharmaceutical company wants to prove that their drug decreases depression, eases anxiety, makes people feel less fatigued, or has any other psychoactive effect, they have to use self-report scales like the Beck Depression Inventory or the PHQ-9. The "longest study on human happiness," featured in a TED talk with over 45 million views, is based on people’s answers to questions like “How happy or unhappy did you feel in the past 24 hours? (1 = very unhappy, 7 = very happy).” The Big Five, the most widely used personality assessment, is a long list of self-report questions.<br>And yet researchers often criticize self-reports and apologize for using them. Self-reports are supposedly riddled with error and biases: people lie, they tell you what they want to hear or what they want to be true, they can’t remember, they make things up, they give very different...