What's Wrong With the Battleground Poll?

I have gotten an increasing number of questions about the GWU/Battleground Poll, which presently gives John McCain a 2-point national lead, even as essentially every other current national poll shows Barack Obama with a lead of at least 5 points.

Just because a poll is an outlier doesn't necessarily mean that it's doing something wrong. Pollsters may have legitimate reasons for having a different perspective on the election, and they may also occasionally produce odd results due to chance alone.

In this case, however, the poll seems to be making a relatively fundamental mistake: it is not weighting by age.

Take a look for yourself at the "weighted tables" that Battleground released a couple of days ago (PDF). These crosstabs provide a ton of detail -- kudos to Battleground for doing so -- but unfortunately there is one red flag. This is the age makeup of their weighted sample:

18-34    17%
35-44    12%
45-64    40%
65+      31%

Intuitively, this probably looks fairly wrong to you -- almost twice as many age 65+ voters as age18-34 voters? And in fact, it almost certainly is wrong. By comparison, here is the approximate age composition of the electorate in 2004, as according to the US Census Bureau**:

18-34    26%
35-44    17%
45-64    38%
65+      19%

Battleground's numbers are not even close. About 19 percent of voters were aged 65 and older in 2004, as compared to the 31 percent in the Battleground sample. On the other hand, 43 percent of voters were aged 18-44, as opposed to Battleground's 29 percent. These differences are much, much too large to be attributable to chance alone. (And all of this is assuming that turnout in 2008 will match that in 2004, even though youth turnout increased markedly in the primaries and is at least somewhat probable to do so in the general election.0

It is highly unlikely that Battleground has deliberately decided to set their weightings this way. Instead, very probably they simply aren't weighting by age groups at all. Since getting young voters to respond to polls is difficult because of the cellphone problem and other reasons, unweighted data will skew substantially older.

Battleground actually has a fairly good track record; they had a slight GOP-leaning house effect in 2004, but not a dramatic one, and before that had a couple of strong elections in a row.

Until fairly recently, however, it was probably not absolutely necessary to weight by age groupings. This year it is, because two factors have coalesced at once:

(1) Voter preferences are much more strongly correlated with age than they have been in recent elections. In 2000, Al Gore won young (age 18-29) voters by just 2 points, barely different than his overall margin against George Bush. In 2004, John Kerry won young voters by 9 points. But this year, young voters are going for Barack Obama by anywhere from 15 to perhaps as many as 35 points, depending on which poll you look at.

(2) In addition, principally because of cellphones, young voters are becoming much harder to reach than they used to be, and so the age skew of unweighted samples will be significantly greater.

Indeed, it is fairly predictable that a poll that had otherwise been fairly accurate but which did not weight by age groupings would encounter slight a Republican skew in 2004, and then a much more dramatic Republican skew in 2008.

Battleground has a rich history and should be given the benefit of the doubt -- if and when they correct this problem, or provide a cogent explanation of why they believe it doesn't need fixing (which I would be happy to publish here). Until then, this should be presumed to be a bug rather than a feature, and their tracking poll should not be taken especially seriously.
_______
** Note: the Census Bureau uses slightly different age groupings from Battleground -- in particular, they group 18-24 year-old voters together, and 25-44 year-old voters together. To match the results to Battleground's numbers, I have assumed that half the Census Bureau's age 25-44 voters are between ages 35-44, and the other half are between ages 25-34, which are then lumped together with the 18-24 voters.