A Survey Research Interlude


James Hanley

James Hanley is a two-bit college professor who'd rather be canoeing.

Related Post Roulette

9 Responses

  1. Avatar Pat Cahalan says:

    If you can’t ask a decently well grounded question inside the frame, don’t use the frame to ask that question.

    I respect Gallup for what they do, but there are going to be questions you can’t effectively ask using that methodology. *Asking* those questions with that methodology is going to give you bad results, unless you ask several slightly different versions of the questions to let the audience provide *you* with a context. Of course, that slows down your results and gives you the, “I already answered this question!” problem.

    Since Gallup is (for better or worse) regarded as something of an authoritative source of public opinion, they probably should take care not to ask questions that can produce bad results.Report

  2. Avatar tom van dyke says:

    Ideology biases researchers at least to the extent of directing them toward researching particular questions and not others. For example, left-leaning scholars are not likely to study how regulations distort market incentives, and right-leaning scholars are not likely to study how market outcomes affect social justice. There’s absolutely nothing wrong with that–we all study what interests us.


    And in the end, the “scholarly consensus” becomes a democratization of truth, depending on the composition of the academy, which “side” gathers more data.

    There is also the question of asking the wrong questions, and if an academic who does so gets the wrong [right?] answers, the researcher becomes a pariah. I have found careerism—call it “office politics”—to be the rule, not the exception—among men. Go along, get along. unpopular findings can make a person unpopular.

    I found your questioning of the phrasing of the question from the Gallup poll spot-on, James. I was going to register an objection that perhaps the poll respondents were already aware of the issue, and so were not affected by the phrasing.

    A viable objection, I think, but another question in the same poll asked whether ratifying the START Treaty was a priority for this lame-duck Congress [the Senate, in this case, which ratifies our treaties].

    73% said it was at least “somewhat important.”

    I dunno about you, but I have no idea what’s in it. Nor do I think many of those 73% do either. I doubt many could identify what the START Treaty is, or even that we have a nuclear treaty with Russkies under consideration, if you asked them out of the blue.

    These polls are very crude tools indeed, and although your objection to the wording is well-observed, this sort of poll-taking is at best about first impressions. When the pedal starts getting near the metal, then The People start paying attention. And this START treaty is DOA in the Senate anyway, and even a political junkie like me [and perhaps you] just can’t spare the time to opinionate on it.

    And that’s why we pay these clowns to represent us in the first place, so we can get on with our lives. The Senate rejected the shit out of the Clinton/Gore Kyoto Treaty, didn’t even debate it. Fortunately, we citizens were spared the pain of having to read it for ourselves.

    I’m all for SALTs and STARTs because I hate the idea of nuclear war, which is why I would have voted against Goldwater in 1964 if I could vote because he would have blown up the world.

    Anyway, to return to your thoughts on the Gallup poll and methodology and shit, I think I agree.

    [I do think that mebbe Gallup is right about the Death Tax, though. My own position is Burkean conservative, that we’ve had a Death Tax and life goes on. I’ve never been one for “tax fairness” arguments. But looking at the data for a long time now, not just this poll, it seems to me the American people don’t like it, either because of practical succession issues in family businesses where tax avoidance freezes growth, or out of “fairness” arguments, that tax was already paid on the rich fuck’s money and it’s unfair to tax it twice.]Report

  3. Avatar James Hanley says:

    Mr. Van Dyke,

    Let’s not pretend to play nice until you ‘fess up about making a false accusation about me preventing you from replying. Hell, I haven’t even asked for an apology, just an acknowledgment of error, but even that apparently requires more integrity than is currently available. And I’ll just add that your first complete sentence is little more than word salad tossed, while your second repeats trite criticisms of an academy that you continue to badly misunderstand. I had a feeling as I was writing this post that you’d immediately extrapolate from my methodological critique about this particular poll question to a much broader and unsubstantiated critique of polling and even the academy, thereby demonstrating your inability to distinguish between the micro and the macro level of analysis.Report

    • James, your post was Charlie Brown’s Christmas tree, sitting unloved and threadbare. I felt sorry for it.

      Yes, it was an erroneous accusation. You would never try to censor me in commenting on one of your posts!


      Yes, you opened the door—explicitly—and I walked through it. The ideologies of the academy will tend to make it look for what it wants to find. Since the academy is biased in one direction and not the other, it will tend to find more data confirming its biases.

      I did agree with your criticism of the wording of the poll question, but I do wonder if poll answers are always given strictly based on the wording of the question or, if the issue is familiar, based on the underlying concepts and not merely the words. This of course is on the nature of ideas, not the techniques of sophistry, and therefore may be inapplicable to your writing.Report

      • Avatar James Hanley in reply to tom van dyke says:

        Thank you.

        As to the door, my perspective is that I opened the door into the social science research lab, and you mistakenly walked through the door into the offices of the National Review.Report

      • > The ideologies of the academy will tend to make it look for what
        > it wants to find.

        Have you ever actually worked in a research group, with a large collection of academics? I’m curious.

        Because in my experience, your characterization of how academic research is done is just way off base. This, for example:

        “And in the end, the “scholarly consensus” becomes a democratization of truth, depending on the composition of the academy, which “side” gathers more data.”

        That’s just wrong. I’ve been to more than a few academic conferences, in a few different fields, and the audience is pretty much uniformly attracted to ripping up anybody who is using bad methodology. Academic researchers are cold blooded bastards when it comes to illustrating the flaws in studies. The results are far less important than the design. There are exceptions, but they aren’t particularly common… in fact, I’d go so far as to call them insignificant in at least six different fields.

        I suppose the “Canon Of The Established” vs. “Reality” can be a viable problem in some fields or at some universities, but I simply don’t see much evidence that it’s either common or pervasive. If you have an alternate experience, I’d like to know more detail.Report

  4. Avatar D. C. Sessions says:

    What would be wrong with simply asking “How important is passing legislation that prevent the estate tax from reverting to %x of estates greater than $y from the 2010 value of zero?”

    Short, accurate, etc. None of the others actually tell people what the actual effect would be.Report

    • Avatar James Hanley in reply to D. C. Sessions says:


      The statement refers to two values (% taxation on estates) and (estates>$y), but conclude with a single value reference (zero). Although the statement is logically sound, it takes some thought to realize that. And for my part, I think referring only to the 2010 value provides insufficient context–but how much context is required is more art than science, so who knows.Report

  5. Avatar Michael Heath says:

    If we assume, I think safely, that the vast majority of Americans are uninformed regarding the history of the estate tax, I’d argue the poll potentially reveals the American people admiringly showing empathy for the estates of those who are financially successful from being punitively taxed. This assumption requires the public to not understand the number of people impacted, the dollar level where they begin to be impacted, and the reason why even a modest proposal is “significant” relative to this year’s rates and not significant relative to previous years.

    The question’s answer reveals nothing about the public’s position on the estate tax, it’s a fatally defective question.

    What frustrates me is my perception of this empathy for those who’ve done well doesn’t extend to our gay service members. I can’t think of a group more disrespected in proportion to their service to all of us.Report