How Not to Do A Survey

So one of the hot topics on Memeorandum today is a study by a Tea-Party affiliated group called the Doctor and Patient Medical Association claiming that, as the Daily Caller’s headline proclaims, “83% of doctors have considered quitting over Obamacare.”

I would think that even amongst most people who are opposed to Obamacare, such a statistic sounds unbelievable and and should raise all sorts of red flags warranting further inquiry before accepting it as true.  Yet every single site coming from the Right who has picked up on the study seems to have accepted it as true without any further questioning.

I started writing a post showing why this survey is complete and utter bunk, but then Media Matters – an outlet I ordinarily dislike immensely – went and did the job for me, discussing exactly the same points I was going to mention, and accurately describing the survey’s methodology as “comically awful.”

The long and short of that debunking is that the relevant question in the survey does not specifically ask about Obamacare, and any suggestion that the polling sample is “random” is beyond preposterous: the poll was in the form of a fax sent to 36,000 fax numbers for doctors culled from an unknown list, of which less than half  went through.    Of the faxes that went through, a whopping 4.3%, representing about 2% of the “randomly selected” faxes, actually chose to respond either online or by fax themselves.  In other words, this is a survey with a massive self-selection problem, particularly given that the group conducting the survey is explicitly and vehemently opposed to the legislation.

To add to Media Matter’s analysis, though, it also seems worth noting that a doctor who was familiar with the group (and thus likely sympathetic to it) doing the survey would be many times more likely to respond to what amounts to fax-spamming than a doctor unfamiliar with it.

Adding to the problems here, there do not appear to have been any safeguards put in place to ensure that only the doctors to whom the fax was directed were respondents, with the only requirement for a valid response being that the respondent proved he or she was a doctor.  So a doctor who was aware of this group and began to fill out a survey would be able to get a colleague known to share that doctor’s views to do likewise.  This is about as far from a “random sample” as you can get.

Underscoring the problems due to the complete lack of randomness involved in this survey is the fact that over 70% of the respondents have been practicing for more than 20 years.  Indeed, there were more respondents, constituting 9.7% of the sample, with more than 40 years in practice than there were respondents with less than 10 years in practice (8.3% of the sample).  Although it’s true that in general the physician population is increasingly aging over the last decade or so, the notion that there are fewer doctors with less than 10 years experience than there are doctors with more than 40 years’ experience is outright preposterous – even despite the “graying” of our physician population, close to 20% are under the age of 35 (and thus clearly have less than 10 years’ experience, though there are surely many doctors over 35 with less than 10 years’ experience).  This portion of the sampling problem has clear and direct effects on the survey’s headline question, which specifically asks whether the respondent has recently considered quitting the profession – since older people are, definitionally, closer to retirement age, any survey that oversamples from that demographic is going to be skewed heavily towards an answer of “yes, I’ve considered quitting.”

The final piece of evidence that this survey is horribly skewed is that 16.2% of respondents are in family practice, while only 7% claim to specialize in internal medicine.  This despite the fact that there are slightly more doctors in internal medicine, comprising about 15% of the overall physician population, than there are doctors in family practice.  This discrepancy serves to demonstrate the degree of self-selection involved with this survey, as internists are, generally speaking, the specialty most likely to support Obamacare, while primary care physicians are the least likely to support it.  Similarly, pediatricians are the third-largest specialty group, yet represent less than 3% of the respondents to this survey; by contrast, general surgeons, the second-largest group of respondents, are the 9th largest group of physicians, comprising only around 3% of the total physician population.   I’m going to guess that pediatricians, on the whole, are comparatively likely to support Obamacare, while general surgeons, on the whole, are not.

In total, this survey is a case study in how not to do a poll.  That is unfortunate, because the opinion of doctors about Obamacare and about how it will affect them is an important piece of the discussion, and useful polling on this issue would be most welcome.  This is not that.