Anatomy of a Polling Disaster

Carl Bialik of the Wall Street Journal has the scoop on the bizarre internals in that IBD/TIPP poll, which as we noted last week, found John McCain as having a substantial lead among young voters. I speculated that this result could only be possible if IBD/TIPP were radically undersampling young voters, and indeed that seems to have been the case:

Raghavan Mayur, president of TechnoMetrica, told me he was equally surprised by the results, saying the widespread perception that Obama is leading by a large margin in that group “is my perception, too.” He blamed the result on a small sample size. Each daily tracking poll includes about 1,000 interviews spread over the prior five days; each day a new set of survey respondents is added and the oldest set is discarded.

Ideally, Mayur would like to have 75 of all those respondents fall into the youngest age range. Some pollsters would have preferred more; this age group makes up 13% of the adult population, though its voting rate historically has been lower than average. His sample fell far short even of his lower goal, typically including just 25 to 30 respondents from age 18 to 24 — meaning just five or six new interviews with these young voters were being conducted each day. “We are not able to get to speak to as many as we would like to in that group,” he said.

He blamed that on several factors. For one thing, nearly one-third of adults in that age range lack landline phones, and Mayur’s pollsters don’t dial cellphones. (He points out that when calling cellphones, the chance that the person who picks up lacks a landline and is in the relevant age range is quite low.) Furthermore, among those who do live in households with landlines, young people may be away at school or in the military, Mayur said.

This small sample size at first didn’t trouble Mayur, as Obama led among these voters in the first three tracking polls. But when the results started to break McCain’s way as suddenly and dramatically as they did, Mayur began to question his own methodology. On the day McCain’s lead widened in this group to 52 points, Mayur added a footnote to the 18-to-24-year-old group: “Age 18-24 has much fluctuation due to small sample size.” He says he didn’t add a similar one to the Jewish subgroup, with just half the sample size as the young voters, because the Jews in his sample consistently stated a preference for Obama, as he expected.

Now, read those paragraphs carefully, because there are several problems with this pollster's process:

1. 75 young voters out of a 1,000-person panel is an awfully low target. In 2004, about 93 out of every 1,000 voters were age 18-24, according to statistics compiled by the Census Bureau. Now, we can debate about whether that number is going to go up this year (youth turnout increased by about 50 percent as a share of the Democratic primary electorate), but it's certainly not going to go down.

2. That notwithstanding, their target may be a moot point, it doesn't appear that the pollster felt any compulsion to weight for age-based demographics in the first place. "This small sample size at first didn’t trouble Mayur" ... well, it ought to have troubled him, because if only 3 percent of your sample consists of 18-24 year-olds, when that fraction should be closer to 9 percent even assuming no increase in youth voter support, you're going to significantly understate Barack Obama's margins. A superior pollster would have flagged this problem long before it became manifest to the entire world.

3. Lastly, Mayur appears to have "resolved" the problem by relying on non-random sampling techniques. Now, I don't want to be too critical of this decision, because Mayur has been kind enough to disclose his process. I'm sure that he isn't the first pollster to take a few shortcuts in the process of creating his sausage, and I'm sure that he won't be the last. Still, this would seem to violate one of the most basic premises of survey research, which is that of the random sample ... in "resolving" his young votes problem, Mayur could very easily have introduced a whole host of others, the effects of which may be harder to detect.

The point of all of this is that just because some pollster puts some numbers together in a PDF doesn't mean that they have any particular idea what they're doing. This pollster apparently made its name for itself because they had forecasted the results of the 2004 popular vote accurately. Notwithstanding that one result isn't anywhere near enough information to conclude that a pollster is strong, the 2004 election was perhaps the easiest one in history to forecast. The electorate was highly partisanized with few undecideds, and both bases turned out in roughly equal numbers; it wasn't just IBD/TIPP that got it right -- almost everyone did. This election is considerably more difficult to poll, and it's exposing the weaker pollsters.