A Survey Research Interlude

Presumably because I’m beginning to unconsciously prep for my annual empirical research methods course, methodological questions have been capturing my attention with great frequency lately. So here’s a survey research methods question that caught my attention. It comes from a recent Gallup Poll asking Americans to rank the top priorities for the lame-duck Congress (surprisingly, accepting Christmas gifts from friendly lobbyists wasn’t one of the answers). The most highly ranked issue, with 56% identifying it as “very important” was:

Passing legislation that would keep the estate tax from increasing significantly next year

Fred Clark, the Slacktivist (best blog name ever) objects that the question is misleading, and should have been written:

“passing legislation that would abolish the estate tax, preventing it from returning to normal levels as promised”

Clark is, I think, obviously right in saying that this “would get a somewhat different response.” But is what he has suggested really a “more accurate question”? There are two separate issues that need to be explored in answering that question, clarity and ideology.

As to clarity, I think Clark’s suggestion fails. I, at least, did a *blink* as I read it, and had to go back and read it again. That’s a bad sign in a research question. I’ve long since lost the source, but I once read a book on writing that emphasized the problem of what I call “stumble over” questions. When you’re walking and you stumble, you tend to look back to see what you stumbled over. The same conceptual thing happens with badly written sentences (but this should not be confused with questions that you have to re-read so that you can “chew them over”), and so of course it happens with survey questions as well. Perhaps the most famous example came from a poll about the holocaust, which contained the incomprehensible question:

Does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened?

We all know how much confusion double negatives can cause, so including one in a poll question, particularly with the added complexity of a compound sentence structure, represents a deplorable lack of judgment.

But while I want to note that Clark’s suggestion lacks clarity, I don’t mean to criticize him too harshly for that. He’s just blogging, after all, and the question wording isn’t his main point, so it would be preposterous to condemn him for not writing an off-the-cuff comment with all the clarity required for doing professional research.

More interesting here is the question of ideology. Ideology biases researchers at least to the extent of directing them toward researching particular questions and not others. For example, left-leaning scholars are not likely to study how regulations distort market incentives, and right-leaning scholars are not likely to study how market outcomes affect social justice. There’s absolutely nothing wrong with that–we all study what interests us. But how does ideology affect the way we shape our research methods? It’s not supposed to, but in some cases it can be hard to avoid.

I think the point Clark is trying to make with his rephrased question is clear. He’s objecting that the question doesn’t provide enough context, and he proposes the context that should be provided–that repeal of the estate tax will prevent it from returning to previous levels. The question is, is his favored context so flavored by ideology that he’s actually creating a push poll? It’s easy to devise ways of asking the question that would clearly be ideologically biased. E.g.,;

Passing legislation to end the estate tax, which is a socialist wealth-redistribution scheme that robs families of hard-earned money.

Passing legislation to completely eliminate the currently very modest estate tax, which would give a giant tax break only to the wealthiest one percent of Americans and do nothing for the middle class.

Obviously I’ve written very obviously slanted questions here, the type that you would find only in push polls. A reputable firm like Gallup would never use such blatantly leading questions. But is their question actually well-written, or does it just have a more subtle (presumably accidental) push? Look at it again:

Passing legislation that would keep the estate tax from increasing significantly next year

I think the question pushes respondents through the information they chose to include and exclude. They mention a “significant increase in taxes,” without mentioning that this tax rate has been decreasing for a decade and the “significant increase” will be a return to the status quo ante, before we embarked on a policy of reducing it.

But Clark suggests we replace that subtle bias with another subtle bias, emphasizing only that it would simply return to prior levels, without consideration of which former level (in fact, the level of 2001, adjusted for inflation, but why could not the unsophisticated respondent think maybe a return to the already lowered 2008 levels, which would decrease support for new legislation). I imagine Clark is sincere in his belief that his preferred wording would be “more accurate,” because he strikes me as a scrupulously honest person. But he is a scrupulously honest person with a distinct ideological bias that could misdirect his interpretation of what is factually correct. That is, although he is sincere, he is mistaken.

Here is the mistake Gallup has made. They have included precisely the wrong amount of information in their question. They could have legitimately included less information, in which case the question could have read something like:*

Passing legislation that would make permanent the elimination of the estate tax.

This approach includes no emotive language such as “increasing significantly.” It simply asks respondents whether they favor a permanent repeal of the estate tax. But while that is an improvement on what Gallup actually asked, I think a better approach is to include more information, in which case the question could have read something like:

In 2001, Congress passed legislation reducing the estate tax by stages, until in 2010 it would be eliminated entirely, and then in 2011 would return to its 2001 levels. How important is passing legislation that would make permanent the complete elimination of the estate tax?

The intent in the latter approach is to present the context in Dragnet fashion (“Just the facts, ma’am”**), so that context is provided without being leading.

The difficulty for Gallup lies in the nature of their survey structure. By listing a set of issues together and asking respondents to rate each one, they constrain how much verbiage they can give to each one. It’s not generally considered to be a really good survey structure, but it has the benefit of being quick and easy to administer, and when you’re a professional polling organization that does lots of polls with samples of over 1,000 people, quick and easy is important. But it comes at a cost.

Your thoughts?

_____________________________________________________
*I emphasize “something like,” because like Fred Clark, I am only blogging. While I am putting more thought into the question than he is, due to the nature of the post and to my sense of professional responsibility, I am fully aware that any first draft of a survey question should be viewed with cautious skepticism. Critiques of the way I phrase my proposed alternatives is perfectly legitimate–just try to diminish any remaining bias, rather than add to or replace it.
**This is an analogy that utterly fails in a classroom of 19 and 20 year old students! Heck, even I don’t actually remember Dragnet.