Question Order May Bias Fox News Health Care Polling

There's a Vietnamese proverb, con sâu làm sầu nồi canh. This loosely translates to: a drop of poison spoils the whole glass of wine. Here is an analogous proverb in polling: a drop of bias can spoil your whole poll. Let me explain what I mean.

Fox News yesterday came out with a poll that suggested that just 33 percent of registered voters favor the Democrats' health care reform package, versus 55 percent opposed. These are not good numbers for Democrats, as they represent a backtrack from the improvement that other pollsters had shown in their health care polling recently.

The Fox News numbers on health care, however, have consistently been worse for Democrats than those shown by other pollsters. Since the health care debate began, the average non-Fox poll has shown 43 percent of the population supporting health care and 45 percent opposed -- producing a net score of -2. By contrast, the average of four Fox polls on health care has shown 35 percent in support of health care and 49 percent opposed -- an ugly -14. The differences in the net numbers statistically significant at the 99 percent threshold.

The first instinct that most of the liberals in the audience will have simply this: well, it's a Fox poll, so of course it's biased. The reality is a little bit more complicated, however. Fox News's pollster, Opinion Dynamics, generally hasn't shown much evidence of a Republican-leaning "house effect". Take a look, for example, at their Obama approval numbers. Since the beginning of Obama's term, they have shown, on average, 58 percent of registered voters approving the President versus 32 percent disapproval. This is, if anything, generous to Obama, as the average non-Fox polls has shown 57 percent approval and 37 percent disapproval over this interval.

The next suspect would ordinarily be the question wording. But Fox News's question is perfectly fine:

Based on what you know about the health care reform legislation being considered right now, do you favor or oppose the plan?

No bias that I can detect there. The question is slightly unusual in that it mentions neither "Congress" nor "Obama" nor "the Democrats", but it's not unusual in a biased way. If anything, the question is particularly unbiased -- it looks to me that support for health care improves slightly if Obama's name is mentioned, but goes down slightly if Congress is mentioned instead. Fox escapes this problem by simply phrasing things in the passive voice and not mentioning either institution.

So how can Fox News ask a seemingly unbiased question of a seemingly unbiased sample and come up with what seems to be a biased result?

The answer may have to do with the questions Fox asks before the question on health care. This weekend, for example, Fox News put out a separate release with their health care questions -- but the health care questions weren't asked separately. Instead, they were questions #27-35 of their larger, national poll, which you can find here. And what were some of those questions? Here are a few:

3. Do you think Barack Obama's travel and speaking schedule makes him look more like he is a candidate on the campaign trail or more like he is the president of the United States?

4. Do you think President Obama apologizes too much to the rest of the world for past U.S. policies?

5. Do you think the Obama administration is proposing more government spending than American taxpayers can afford, or not?

6. Do you think the size of the national debt is so large it is hurting the future of the country?

7. Would you rather: [ROTATE OPTIONS 1 and 2]
Cut spending now so future generations don't have to pay
Keep spending at current levels and let future generations pay

20. When Barack Obama was a candidate campaigning for the presidency, he spoke of the urgent need to finish the fight in Afghanistan, which he called the central front on the war on terrorism. Do you think that, as president, Obama is doing what it takes to win in Afghanistan?

These questions run the gamut slightly leading to full-frontal Republican talking points. Some of them, such as question #3, are almost literally rhetorical questions, which are never good things to have on a poll. And no, you can rest assured that Fox News was not asking questions formed from comparably biased Democratic talking points.

A respondent who hears these questions, particularly the series of questions on the national debt, is going to be primed to react somewhat unfavorably to the mention of another big Democratic spending program like health care. And evidently, an unusually high number of them do.

The reason Fox's Presidential approval numbers are not affected is because Obama approval is question #1 -- asked before the leading series of questions on spending and foreign policy or anything else. Likewise, when Fox was conducting its Presidential polling (which also did not have any discernible house effect), the horse race questions were asked before any of the policy ones.

But when you ask biased questions first, they are infectious, potentially poisoning everything that comes below. I don't particularly care if Fox News wants to ask leading or even outrightly biased questions -- but they have to ask them after any questions they expect the policymaking community to take seriously.

Another problem is that Fox tends to ask different sets of questions before they get to the health care ones -- sometimes, questions about Afghanistan, sometimes questions about deficits, etc. What seems, then, to be "movement" in the numbers may simply be an artifact of what sort of mood they've worked the respondent into before they get to the health care stuff.

To be clear, these question order effects can arise even when pollsters have the best of intentions, and even when they are asking unbiased questions. If, for instance, back during the Presidential campaign, you had asked a series of perfectly neutrally-worded questions on the economy before asking about the horse race, they could easily have tipped the numbers slightly in Obama's direction, since the economy was perceived to be the Democrats' strength. (The LA Times poll had this exact problem at various points). Pollsters have to grapple with these sorts of considerations all the time, and some do a better job of handling them than others.

But when you ask a series of biased questions before taking the voters' temperature on health care or the horse race, you have much less excuse. Going forward, Fox News should put its health care questions closer to the top of their survey or break them out into a separate poll; take their numbers with a grain of salt until they do.