Questions that give away the answer

[Continuing our series on deception in politics and public policy.]

It’s considered unethical to publish the results of a public opinion poll without publishing the exact questions, in the order they were asked, because the wording of a given question and of the preceding questions has a significant effect on people’s answers.

I encountered this effect a few years ago when I was working in the campaign to defeat a tax increase in Northern Virginia (the DC suburbs in Virginia).  The tax hike was to be decided in a referendum, and the Washington Post reported that, according to a poll, most voters supported the measure. That didn’t seem right to me, so I asked to see the questions, pointing out, to the Post editor I reached on the phone, that there is an ethical requirement that the questions be published.  To their credit, the folks at the Post posted the questions.

It turned out that the poll had pushed respondents toward a Yes answer by minimizing the impact of the increase and by  listing, one after another, all the favorable results that supposedly would come from a tax increase. Better roads, shorter commutes, and other positives were listed, in the context of questions like “Would you vote for a small increase in the gasoline tax if the effect was to reduce the commuting time in Northern Virginia?” (I’m working from memory, but the question was along those lines.) Only after a respondent heard this list of tax hike benefits was that respondent asked how he or she would vote.

Despite that poll “showing” support for the increase, it was defeated – although, as you might expect, corrupt pols in Virginia eventually overruled the voters and pushed a tax hike through, anyway.

This biasing effect in polling is related to Cold Reading, a common deception technique.

Cold Reading is a set of techniques used by magicians, intelligence operatives, and con men (including many politicians) to elicit information from a targeted individual, often in order to create the impression that the reader has special knowledge about the target. That psychic who passed along messages from your dead uncle was probably using techniques of Cold Reading.

If the con man spies on you using traditional techniques in order to make himself appear knowledgeable about you, that’s Hot Reading. For example, a psychic might have an associate pick your pocket and examine your wallet photos or use the information on your driver’s license to Google you or find your Facebook page. Perhaps you showed your ID as you entered the studio for the taping of a psychic’s TV show, and the psychic’s confederates pulled the information at that point.  Thus: “I see you with two children at Disney World. One of the children is wearing Mickey Mouse ears,” and “You live on a street with… wait, it’s coming to me… it’s a plant. The name has a plant in it. And it starts with an M. Is it… Mulberry Street??”

In Cold Reading, the con man figures things out about you without any information except your facial expressions and  body language. Typically, his tricks consist of shotgunning (making a broad range of statements, some of which are likely true, such as the psychic asking people in an audience if anyone has a brother named John who’s had cancer, then narrowing his attention to the person in the audience who suddenly becomes attentive); the rainbow ruse (making a statement that covers the gamut of possible situations, such as “You are usually a happy person, but there are times in your life when you’ve been really sad”); and Barnum statements, which apply to pretty much everyone (“You had a pet as a child, or wanted one very much”).

The effect demonstrated in Barnum statements is often described as the Forer effect, named after a psychologist named Bertram Forer. In 1948, Forer gave his students personalized profiles that included 13 statements such as “You have a great need for other people to like and admire you” and “You have a tendency to be critical of yourself.” Asked to gauge the accuracy of the personalized statements on a scale of 0 to 5, they gave the statements an average score of 4.26. As you might have guessed, the statements weren’t really created for individual students; each student got the identical “personalized” profile, one compiled from statements in an astrology book.

In polling, there are many types of deception, such as presenting members of the general public as having an opinion on a complex or obscure issue or on a term-of-art whose meaning is known to relatively few people. For example, pollsters have asked people whether they’re for a “nuclear freeze,” whether they’re for a particular “chemical weapons treaty,” whether they’re for “a moratorium on fracking,” and whether they’re for “a path to citizenship” for illegal aliens (or “undocumented immigrants,” as they’re falsely called by people trying to rig polls). None of those results can be taken seriously, except, perhaps, in a comparison to people’s response to an identical question in an earlier poll.

Some of the deception in polls is related to Cold Reading, in that people are pushed toward a particular response, as in the Washington Post tax hike poll noted above. Often, people’s ignorance on complex issues (see last week’s column at capitalresearch.org/2014/07/progressives-and-the-dumb-factor) is used as a cover for the “push” in the poll. The language that pushes people toward a particular answer is explained as necessary to explain the issue so that people can express their opinions.

There’s the rub, expressed as one of Allen’s Rules: No joke is funny that must be explained, and no poll question is valid that the pollster must “explain” to the respondent. If the issue is “explained,” the respondent is not responding to the issue itself but to the pollster’s explanation.

Correspondingly, Allen’s Limitation is this: If a poll question is longer than roughly 12-18 words, it’s not a valid question.

(…unless, of course, the stated purpose of the poll is to test out different arguments for or against a particular position.)

The limit of 12-18 words is a rule of thumb, based on the fact that, the longer the question, the stronger the effect of the question’s inherent bias and the weaker the effect of the underlying opinion of the respondent.

Examples, from a Pew poll published June 20:

► “Right now, which ONE of the following do you think should be the more important priority for addressing America’s energy supply? Developing alternative sources such as wind, solar, and hydrogen technology, or expanding exploration and production of oil, coal, and natural gas?” That’s 42 words, in which the pollster implies that “wind, solar, and hydrogen technology” are legitimate “alternative sources” (solar and wind being ancient technologies to which oil and gas were the alternative, not the other way around, and “hydrogen technology” being something unfamiliar to 99 percent of Americans).

► “Should the U.S. Supreme Court base its rulings on its understanding of what the U.S. Constitution meant as it was originally written, or should the court base its rulings on its understanding of what the U.S. Constitution means in current times?” That’s 41 words, and completely misstates the positions of the two sides in the great debate over the interpretation of the Constitution. Originalists believe in following the Constitution’s words based on the meaning of those words at the time they were written into the Constitution, not “as it [the Constitution] was originally written.” (The Constitution was “originally written” at a time when slavery was considered legal. Originalists reject the idea that slavery is legal, and interpret the 13^th Amendment, which abolished slavery, based on the meaning of the words at the time the 13^th Amendment was written.) And proponents of the “Living Constitution” interpretation believe that the meaning of the Constitution’s words change due to the fads and fashions of later times, not based on their  understanding “of what the U.S. Constitution means in current times.” For example, Living Constitutionalists such as former Justice David Souter have argued, in effect, that school segregation was properly judged as legal until society changed and that practice became illegal.

►”In general, do you think affirmative action programs designed to increase the number of black and minority students on college campuses are a good thing or a bad thing?” That’s 29 words, which includes loaded terminology (“affirmative action,” which sounds a lot better than an accurate term such as “racial discrimination”) and a blatant falsehood (“designed to increase the number of . . . minority students,” which sounds a lot better than an accurate statement such as “designed to reduce the number of Asian-American students”).

Finally, here’s an exercise: The next time you see a news story based mainly on the results of a public opinion poll, call the reporter, producer, or editor who prepared the story and demand to see the poll questions. If he or she doesn’t produce them or point you to them, and refuses to retract the report, he or she is acting unethically. Call ’em on it. I’ve done it, and it’s loads of fun.