Are Pollsters Oversampling Republicans?

I have received a myriad of requests asking me to respond to this Seth Coulter Wells article at the Huffington Post on changes in the party ID composition of recent polls. Although the article itself is a good and thoughtful piece, some of the interpretations of it are not. So let's get a few things straight:

1. Polls that show changes in party ID composition are not "cooked", "rigged", or "biased". Changes in party ID composition may occur for a variety of reasons, including random chance, response bias, temporary changes in party affiliation, and longer-term changes in party affiliation. It is possible that, if the partisan ID composition of a poll shifts rapidly from week to week (more so than is likely from random chance alone, and more so than is precipitated from external events), this may be an indictment of a pollster's methodology -- that the pollster is having difficulty getting a good, random sample. But the notion that credible pollsters like Gallup or SurveyUSA are deliberately rigging their samples is patently ridiculous. Polling is a very competitive industry, with relatively low barriers to entry; they would go out of business in a hurry if they did this.

2. The decision a pollster faces is whether or not to weight its sample by party ID. In fact, the whole point is that pollsters like Gallup and SurveyUSA do not weight their samples by party ID -- they just tally the results, and let the chips fall where they may. So in some sense accusing them of "cooking" their samples has it backward; what you're really arguing is that they should weight their samples, presumably in a way that is more favorable to your preferred candidate.

However, there is no one right answer as to whether to weight by party ID is the superior choice. A couple of pollsters, notably Rasmussen, do weight their samples by party ID. These pollsters tend to get more stable results from week to week. However, more stable does not necessarily mean more accurate over the long run, as these pollsters may miss real changes in party ID that occur over time.

Rasmussen, which targets its party ID figures based on a rolling, three-month sample of interviews with thousands and thousands of voters, has what I consider to be a eminently reasonable approach. Even so, the true party ID composition of the electorate is difficult and perhaps impossible to ascertain. You can take a survey of party ID, but it is still a survey, and therefore itself may be subject to issues like poor methodology and response bias.

There might be some promise in targeting party ID figures based on actual changes in voter registration -- although many states have open (non-partisan) voter registration, and some of those that do require registration by party do not report these figures. Some state-level pollsters seem to flirt with this approach, although I do not believe that any of the more prolific, national pollsters do.

3. Changes in party affiliation are tied up in the notion of the "convention bounce". Successful party conventions usually produce some shift in the self-reported partisan composition of surveys taken in the days and weeks that follow them. This may be a form of response bias, if voters of a particular partisan persuasion are fired up by their convention and more likely to take a pollster's phone call. (It may particularly be a form of response bias if the poll is taken during the convention itself, as voters of a particular party may be more likely to be at home watching their party's convention, and therefore more available to take a phone call).

However, remember what the conventions are. They are the Republican National Convention and the Democratic National Convention -- not the McCain National Convention and the Obama National Convention. They are big advertisements for the parties themselves. And so, it is not unlikely that, if voters respond favorably to a convention, they will respond favorably both to the candidate and to the candidate's party.

In other words, while some of these polls may have implausibly high numbers of self-reported Republicans in their samples, this is more a symptom of the convention bounce rather than a separate and distinct phenomenon. Moreover, conventions sometimes do trigger longer-term shifts in party identification. In 2004, for instance, the Republican convention appeared to cause a semi-permanent shift in party ID toward the Republicans, amounting to a couple of points. As I also argued recently, it is also plausible that unenthusiastic Republicans or Republican-leaning independents (which I call "shy Tories") were being undersampled prior to the Republican convention, until the convention renewed their enthusiasm.

The fact that the Bush administration is in its last days, that much of the dissatisfaction with the Republican Party had to do with George W. Bush, and that this is an election featuring two non-incumbents, is a salient factor here. It may be that, rather than trying to tie John McCain to George W. Bush, the Democrats should be making more of an effort to indict the Republican Party as a whole, of which John McCain is indisputably a member.

4. Responsible pollsters should always disclose the partisan ID composition of their samples, and should notate in their press releases when material shifts occur. Regardless of the above, there is little doubt that the partisan ID composition of a poll is an extremely important element in the proper interpretation of a poll. Pollsters should always report the partisan ID composition of their sample, and should qualify in their write-ups when substantial changes in party ID have occurred since their last survey.