If Frank Newport at Gallup is right, than that LA Times/Bloomberg poll that showed Barack Obama with a 12-point lead may have had a flaw after all.
As is always the case, there are some slight differences in the way the polls are conducted. The Los Angeles Times/Bloomberg poll asks a "right direction/wrong direction" question before the ballot. Our Gallup Poll Daily tracking asks a registered voter screen before the ballot. The Los Angeles Times/Bloomberg poll includes the phrase "or would you vote for a candidate from some other party?" Our Gallup poll does not include this phrase. It is unclear how the order of these questions may affect the polling results.That's a pretty big no-no in my book. Question order definitely matters -- the later you ask a question, the more it's going to be influenced by the implicit messaging triggered by previous questions. In this case, an overwhelming majority of Americans think the country is on the wrong track, and most of them also associate that with the incumbent Republican administration. So it isn't surprising that a Republican presidential candidate performs worse if you ask about presidential preference immediately afterward, or that Republican party ID was lower in this survey.
In fact, there happens to be some direct evidence of this. In its new New Jersey poll, Fairleigh Dickinson split its panel into halves. The first half received a battery of national questions -- including a "wrong track" question -- before being asked their presidential preference. The second half got the Presidential questions first.
Among those respondents who got the presidential question first, Obama had a 47-34 lead. But among who got the wrong track question and then were asked for their presidential pick, Obama's lead expanded to 51-33. The difference was particularly large among independent voters, who split 24-24 with huge numbers of undecideds when asked the presidential question first, but went 41-14 for Obama if they had been prompted by the national mood questions.
One shouldn't read that much into this -- when we're looking at subsamples, and particularly subsamples of subsamples -- the margins of error are very high. Nevertheless, those splits would fall into line with generally accepted theory. I understand that folks like LA Times/Bloomberg take a lot of pride in their national mood questions, which in some cases they have asked for years on end. But it's the horse race questions that generate the earned media for your polling organization and convey prestige upon it. That's why most of the media-savvy pollsters that we tend to reference frequently understand that you need to give horse race questions the top billing.
18 comments
Interesting... thanks.
Thank you for your analysis...Finally a reasonable explanation for why the outlier polls are so far out of their margins of error. (Though there is always a chance for this, of course, as the margin of error isn't definitive.)
I'm not sure if they still do it, but for a while the Rasmussen polls were all asking those surveyed their opinion of Bush as question 1, and their McCain/Obama preference as question 2.
This seems like it would have the same problem.
Well said Nate. But now that I think about it, some "non-partisans" should all go hold signs outside of polling locations on Election Day that say:
"Do you think the country is headed in the right direction, or the wrong direction? Be sure to vote today!"
That'd improve Obama numbers without looking partisan.
For those of you with less of a statistical background, may I suggest the following episode from the BBC series Yes Prime Minister:
http://www.youtube.com/watch?v=2yhN1IDLQjo
It’s a great clip I’ve used to make Nate’s exact point.
Nate, how about an open thread on occasion?
This is revealing... So, it indicates that the real stakes of this election are not about experience or issues, but rather about whether McCain can paint a rosy picture of the shithole we live in today, or whether Obama can hammer home just how bad things are...
When people go to vote thinking of how shitty things are, they vote Obama much more; when they eat the force fed McCain Bullshit, they'll vote like ignoramouses...
Explains how things got so bad in the first place if you ask me.
arguably, though, actual voters will get a fair amount of "national mood prompting" before the election just based on the fact that they, y'know, live in the nation.
A more general point, other than that context and question order matter, is that a source of error in horse race polls is that the surveys in which they are embedded often have multiple purposes and multiple clients.
Readers here often ask "when's the next poll from [NAME OF STATE] coming out from [NATE OF POLLSTER]. Often the answer can only be, "When that pollster has a client or set of clients to put a survey into the field."
But it's up to the survey designer to make sure that the question order doesn't corrupt the results either within or across blocks of questions. Looks like the LA Times poll was done by an amateur.
In this case, an overwhelming majority of Americans think the country is on the wrong track, and most of them also associate that with the incumbent Republican administration.
I agree with you that the question biases the poll and shouldn't have been in the original questionnaire.
But isn't this a fundamental problem that McCain is going to have as we get closer and closer to the election? He's ultimately going to be tied to the incumbent and the "right track/wrong track" numbers aren't likely to turn around dramatically.
Right now many people aren't thinking about the presidential race and the result is large numbers of undecided voters. By September people will be taking more notice. It seems like this will weigh more on McCain as the months go by.
I'm stunned that you find the flaw in that and not the other polls:
Who in their right mind thinks that in November 2008 voters will not have the change/anti-republican mood in mind when they are voting?
Sorry Nate but I see Frank Newport's comments as yet another example of him trying to defend Gallup tracking polls' skewed results. Why does he fee he needs to defend it? Because his is the ONLY poll showing a tie! EVERY other poll shows an Obama lead (big or small, but at least a lead). Please disregard Frank Newport's comments as little more than a defense of his own polls messed up results.
Jinchi wrote: "But isn't this a fundamental problem that McCain is going to have as we get closer and closer to the election? He's ultimately going to be tied to the incumbent and the "right track/wrong track" numbers aren't likely to turn around dramatically.
Whether that's "a fundamental problem that McCain is going to have" shouldn't (as you say) be assumed in the survey. Indeed, there are many other "fundamental problems" that both Obama and McCain will have before election day, and the campaign is all about trying to frame the debate in a way that favors one's own candidate.
IOW, who's to say that a simple "right track, wrong track" assessment is going to dominate such other frames as "experience vs. change" or "victory vs. defeat" or "radical liberal vs. middle of the road"?
Just as Quinnipiac did in that NJ poll that Nate analyzed, the survey researchers ought to be seeing how different frames work, but they can't themselves determine which frame will be most salient to the voters on election day. That's going to come from "life itself" and from the media and from the campaigns. It shouldn't come from the pollsters.
At the very least we know what Obama should concentrate on. If he convinces the electorate that McCain is tied to "wrong-track" then he wins easily.
these must be the "low info" voters that brian schweitzer talked about on his speech you can find on youtube.
that's why to me, he would probably make a great vp pick.
its sad to think that although 85% of us are pretty well informed, its that unpredictable know-nothing 15% that really breaks the tie on these things.
Though Nate's comments that question order matters are undoubtedly true, I have a somewhat different take on the point.
At this (still relatively early) stage of the campaign, questions dealing with issues and candidate images provide a shorthand "simulation" of the campaign itself.
I'd argue, in fact, that the polls that precede a candidate preference question (or a partisan ID question) with a series of issue questions (or right track/wrong track) provide a context that the voter may well experience in the days leading up to the election and may well provide a better gauge of voter intention than a bare voter preference question as found in the daily tracking polls.
jsh1120 - I agree, to better simulate the actual lead up to the election, the horse race question should be preceded by "Do you approve of Senator Obama's joining the mosque of Hamas' leader?" and "Have you heard that John McCain beat his wife in a cocaine-induced rage?". That way they'll be sure to frame their decision in the context of this historic election.
Post a Comment