Confusion, Indifference, and Compromise: How Do We Answer Surveys When We Don't Know the Answer?

Cards, showing all four suitsImage by Enoch Lau, via Wikimedia Commons under a CC-BY-SA license. Take a card deck (no jokers). Pull out a card. What’s the probability that you’ll see a spade?

25%, right?

What would you do if I don’t show you the card, but just give you a piece of paper asking “Is the card a spade, heart, club, or diamond?”

You might reasonably want to circle all four, since they’re equally likely. On one hand, that’s an absurd outcome: it cannot possibly be all four suits. But it’s equally likely to be any of them, and circling all four may seem like the clearest way to invoke what statisticians call the principle of indifference (or sometimes, the principle of insufficient reason). If you don’t have any reason to think otherwise, you expect a randomly selected card from a deck has equal chances of coming from each of the four suits, and signal that belief by selecting all of them.

What if, instead, the paper I gave you asked “Is the card a spade or not a spade?”

In this case, I think it would be a misapplication of the principle of indifference to circle both options. There’s a 25% probability of getting a spade, and 75% probability that it isn’t. Circling both treats both outcomes as equally likely. That would be the right way to approach guessing whether a dropped card will be face up or face down, but the wrong way to think about whether a randomly-chosen card is a spade or not.

It occurred to me recently that comparable misapplications of the principle of indifference might be playing a role in how people respond to surveys.

Consider the survey of science teachers NCSE and Penn State recently conducted. We asked 1500 science teachers in public middle schools and high schools around the country about their personal views on the reality and causes of recent climate change. Two thirds (65%) agreed that climate change is real and mostly caused by humans, one sixth agreed that it is mostly caused by natural changes in the environment, and only 2% said it isn’t happening at all (5% skipped the question). Roughly another one in six either wrote in “both” or circled both the “mostly natural” and “mostly human” options. It can’t literally be “mostly” both, so those responses are a bit weird on their face, but they make sense if treated as an expression of indifference.

But doing so stretches the principle past the breaking point.

Essentially, the approach of those who agreed with both the “mostly natural” and the “mostly human” is to treat human and natural causes as two sides of a tossed coin. If we aren’t sure about the relative importance of human and natural causes, we ascribe each outcome equal probability or equal impact.

But of course human and natural causes aren’t two sides of a coin, and interpreting indifference to make them equally likely misinterprets our baseline assumptions about each kind of process.

Here’s a better application of the principle. Knowing nothing else about the issue, the principle of indifference should compel us to say that natural forces are equally likely to warm or cool at any moment, averaging out to no effect in the long run. If that weren’t the case, we would either observe a consistent warming or cooling throughout the 4.55 billion years of earth history, and we don’t. In general, natural forcings on climate reach some equilibrium, with periods of cooling influence alternating with periods of stasis and periods of warming. Without additional information, the principle of indifference suggests that the contribution of natural forces ought to be near zero and whether natural forces are warming or cooling at any given moment is a coin toss.

Indeed, if you look at IPCC reports, you find that the error bars around the role of natural forcings in recent times do include zero. Depending on the timeframe one examines, the net effect of natural forcings alone may even be negative in the modern era, with human activities contributing more than 100% of the warming.

By contrast, even a relatively simple model of the role of humans on the environment should lead to a default assumption that humans have warmed the global climate. We know humans have been burning fossil fuels in vast quantities, and that atmospheric concentrations of carbon dioxide have risen as a result (Arrhenius knew that over a century ago). We know, and have since the mid-19th century, that carbon dioxide tends to trap heat in the atmosphere. So the principle of indifference should not lead us to treat the role of humans the same way we treat the role of natural processes; it should lead to a baseline assumption that humans have warmed the global climate, with the coin toss being whether the warming was more or less than what that simple analysis would predict. Again, IPCC reports and other scientific assessments are uniform in concluding that humans have caused most of the modern warming.

Indeed, I would be surprised if a fair number of the teachers who wrote in a “both equally” response wouldn’t have answered differently if we posed the question in a form that didn’t let the principle of indifference guide them toward the right responses. For instance, I wonder what would have happened if we had asked:

“What role do you think natural forces have had on global temperatures over the last X years: substantial warming, substantial cooling, close to none”

and

“What role do you think human activities have had on global temperatures over the last X years: substantial warming, substantial cooling, close to none”

Here, the principle of indifference should lead uncertain teachers to correctly peg natural contributions near zero, but gives them a chance to then recognize that humans are having a noteworthy effect.

There’s another possible explanation. Marketers and cognitive psychologists have long observed a “compromise effect,” whereby people tend to shy away from extreme options, especially when they have limited information. A classic example comes from restaurant menus, where an expensive bottle of wine might be added to the wine list not in hopes of selling that $100 bottle, but to make the $30 bottle look like a reasonable middle position.

Something like this may be at work in the responses to Gallup’s long-running survey question on evolution, in which respondents are offered three choices: creationism (defined in terms of God’s creating human beings within the last ten thousand years or so), evolution with no divine involvement, and a middle option in which a deity guides evolution. Generally speaking, around 36–40% of respondents pick the middle option. Some are old-earth creationists, others are “intelligent design” creationists, and others are theistic evolutionists. But others seem to have picked the middle option just because they are uncertain and regard the middle option as safest. Researchers have found that if you offer additional options—“none of these comes closest to my views” and “not at all sure which is true”—support for the middle ground option shrinks notably, from 36% when offered three options to 26% when offered five. In other words, people may not strongly support the middle ground position, but it seems like a safe compromise.

It’s possible to interpret the teachers’ responses about climate change through that same framework, where they created their own compromise position by circling two options or writing in “both.” And it’s possible to interpret the Gallup results through a framework of the principle of indifference: if respondents felt they were being offered a coin with evolution on one side and creationism on the other, they may have signaled their indifference by selecting a choice that seems to encompass both outcomes.

I suspect we may be able to use future surveys to test these hypotheses and get a clearer sense of how people who haven’t studied the issue before work through questions about what causes climate change.