Three footnotes that weren't quite worth a post unto themselves:
1. Around 337,000 North Carolinians have already voted in their state's primary, a fraction that is likely to represent approximately one-quarter of the eventual turnout. According to some data mining done by the Center for Congressional and Presidential Studies, around 38 percent of those voters have been black. As you'll find from playing around with our prediction tool it is very hard for Obama to lose the state if black turnout winds up at 38 percent. Even if he "only" won the black vote 85-15, and lost the white vote 70-30 -- he would still win the state by 2 points if black turnout was 38 percent. If we instead use the numbers from Survey USA's most recent poll: Obama loses whites 61-30, but wins blacks 87-11 -- he would win the state by 10 points given those turnout demographics. EDIT -- the percentage of black voters is apparently 39.3% based on the very latest numbers.
2. North Carolina uses a modified primary, rather than an open primary. What that means is that while independents may vote in the Democratic primary, Republicans cannot. Also, it is not possible to change one's party affiliation at the last minute -- the state must receive any changes by the registration deadline, 35 days ahead of the election. Moreover, there is a highly competitive Republican primary in the state's gubernatorial race. Long story short: while independent voters are something of a factor in North Carolina -- they make up about 16% of the early voting turnout -- Republican voters really aren't. An Operation Chaos voter would have had to change his registration weeks ahead of time -- and sacrificed the chance to vote in the gubernatorial race.
3. While those first two items probably bear well for Obama, here's something that I think might be worth a point or so for Clinton. A couple of pollsters, like Zogby and SurveyUSA, have been picking up a fairly large "other" vote in their North Carolina surveys, particularly among white voters. I would guess that some of these people are folks that intend to vote for John Edwards. However, I would surmise that those people who were still planning to vote for Edwards at this stage would tend to be low-information voters that will probably default to Clinton.
5.03.2008
Today's Polls: Obama and Clinton chasing 50% against McCain
@ 2:47 PM
Barack Obama has been under the 50% threshold in his match-up against John McCain in each update since March 19th. He is now within percentage points of that number, checking in at 49.8% in today's update, his best standing since that March 19 date. Hillary Clinton, meanwhile, has a 46.8% win percentage, her highest-ever number since we began tracking these results in early March.
The first bit of new polling data comes from Rasmussen, which shows a huge, 29-point lead for Hillary Clinton in New York, as well as a relatively impressive 17-point margin for Barack Obama. This is part of a recent pattern in which Democrats have begun to consolidate their numbers in traditional Democratic-leaning states, as we've also seen reflected in recent polling of Minnesota and New Jersey.
We also have a Downs Center / SurveyUSA poll in Indiana that shows both Democrats with a tiny advantage over John McCain: Clinton by 3 points and Obama by 1 point. This is now the second set of polling -- last week's Sezler polls being the first -- that shows the Democrats tied or ahead of John McCain in Indiana, and it may be time to begin to take Indiana a little more seriously as a swing state.
Finally, in North Carolina, Research 2000 shows John McCain with a solid lead over both Democrats: he leads Barack Obama by 9 points and Hillary Clinton by 12.
Also, a very quick methodological note: for the time being, I am forcing Obama's regression model to include the Mormon variable, even though it is just slightly below the statistical significance threshold that we usually require. If we were not making this adjustment, the model would show Obama being highly competitive in Utah, a result which seems implausible on its surface. Another alternative would be to combine Mormons and white evangelicals into one variable -- this would very much be theologically incorrect, but among other things it would clear room for the inclusion of another regression variable (such as unemployment rates) into our model. Thoughts?
The first bit of new polling data comes from Rasmussen, which shows a huge, 29-point lead for Hillary Clinton in New York, as well as a relatively impressive 17-point margin for Barack Obama. This is part of a recent pattern in which Democrats have begun to consolidate their numbers in traditional Democratic-leaning states, as we've also seen reflected in recent polling of Minnesota and New Jersey.
We also have a Downs Center / SurveyUSA poll in Indiana that shows both Democrats with a tiny advantage over John McCain: Clinton by 3 points and Obama by 1 point. This is now the second set of polling -- last week's Sezler polls being the first -- that shows the Democrats tied or ahead of John McCain in Indiana, and it may be time to begin to take Indiana a little more seriously as a swing state.
Finally, in North Carolina, Research 2000 shows John McCain with a solid lead over both Democrats: he leads Barack Obama by 9 points and Hillary Clinton by 12.
Also, a very quick methodological note: for the time being, I am forcing Obama's regression model to include the Mormon variable, even though it is just slightly below the statistical significance threshold that we usually require. If we were not making this adjustment, the model would show Obama being highly competitive in Utah, a result which seems implausible on its surface. Another alternative would be to combine Mormons and white evangelicals into one variable -- this would very much be theologically incorrect, but among other things it would clear room for the inclusion of another regression variable (such as unemployment rates) into our model. Thoughts?
5.02.2008
ARG Attacks!
@ 7:01 AM
In the strange-but-true department, American Research Group polling head Dick Bennett has directed some criticism at this website over, of all things, our prediction on the Pennsylvania primary.
Need I remind him of how our results compared against his?
Need I remind him of how our results compared against his?
It's not all about Obama
@ 3:25 AM
Whenever there is a shift of some kind or another in the polling numbers, there are fundamentally two headlines that can be written about it:
"Candidate X loses support!"
or
"Candidate Y gains support!"
At some point in late February or early March -- when it became apparent that he was the favorite for the Democratic nomination -- Barack Obama became the central figure in the psychodrama of the 2008 campaign. Thus, when the polling numbers show a shift toward Obama, the interpretation is usually "Obama gaining support!". And when the opposite is true -- as it has been in recent days -- the narrative is invariably "Obama support collapsing!".
If there were only two candidates in the known universe, it would be hard to distinguish an intrinsic increase in the support for one candidate versus an intrinsic decrease in the support for another. Fortunately, we have another benchmark we can look at: the performance of each Democrat against John McCain, whose campaign has now been largely newsless for several months.
And the present numbers do not show a collapse in Obama's support against John McCain. If anything, in fact, Obama's numbers have been improving. The Clinton campaign was understandably giddy about the results of the new Quinnipiac polls -- but those polls also contained good news for Obama. He gained 5 points in Pennsylvania versus Quinnipiac's previous poll of that state, and 8 points in Florida. True, he lost 2 points in Ohio. But the balance of polling data has been pretty good for Obama. He's at his highest point in our win percentage tracker since March. We don't look much at the national polling data here, but Pollster.com does, and they show Obama's numbers against McCain trending upward rather than downward:
Now, what is also true is that Clinton's numbers against McCain are moving upward. And in fact, they're moving upward faster than Obama's are moving upward:
And why shouldn't Clinton's numbers be moving upward? She has had a pretty good month:
The irony, of course, is that it's very much in Clinton's interest to spin the "Obama is melting!" storyline rather than the "Clinton is surging!" storyline. Because, for all of these things -- Obama remains roughly tied or slightly ahead of Clinton in national polls of Democrats, and it's going to be very, very difficult for her campaign to argue that the superdelegates should overturn the pledged delegate count so long as that is the case.
I tend to be guilty, as most analysts do, of speaking of the superdelegates as some sort of high council that make their decisions in unison, but instead they are hundreds of individual decision-makers. By far the most likely outcome is that the primary season ends not with a bang but a whimper, and that at some point in early June Obama receives enough superdelegate endorsements to clinch the nomination. Obama's pledged delegate advantage is worth something -- a lead of about 125 pledged delegates (which is where he's likely to end up after all states have voted) means that he would only need to win the support of 42% of all superdelegates to win the nomination. Even more importantly, it means that he'll only need to win the support of 35% of all superdelegates who have not already committed to a candidate -- and the threshold is even lower when you consider that many of the remaining uncommitted superdelegates are add-ons, whose support for one candidate or another is essentially already in the bag.
For Clinton to win the nomination -- what she needs is for panic to set in. She needs a run on the Obama bank. That would probably require at least a couple of high-profile Obama superdelegates to switch their support from Obama to Clinton -- at which point the floodgates might open.
The trigger for that would be a truly shocking result in one of the remaining primaries -- such as a Clinton win in North Carolina or maybe Oregon or maybe a very close result in North Carolina coupled with a double-digit win in Indiana. Obama does benefit a tiny bit here from lowered expectations: I would guess that most pundits deep down expect Clinton to win Indiana by a Pennsylvania-type margin, and that Obama will have a fairly easy time of spinning away his loss as the same old story, provided that he has a solid victory to show for himself in North Carolina. Or, a new gaffe/scandal/inconvenience for Obama of at least Wright/bittergate magnitude (but not merely of Ayers/flag-pin magnitude). Those things remain unlikely, although the possibility of the former has somewhat increased. But there certainly hasn't been a run on Bank Obama yet, nor really even the beginnings of one -- instead, there has merely been some refreshed enthusiasm for Hillary Clinton.
"Candidate X loses support!"
or
"Candidate Y gains support!"
At some point in late February or early March -- when it became apparent that he was the favorite for the Democratic nomination -- Barack Obama became the central figure in the psychodrama of the 2008 campaign. Thus, when the polling numbers show a shift toward Obama, the interpretation is usually "Obama gaining support!". And when the opposite is true -- as it has been in recent days -- the narrative is invariably "Obama support collapsing!".
If there were only two candidates in the known universe, it would be hard to distinguish an intrinsic increase in the support for one candidate versus an intrinsic decrease in the support for another. Fortunately, we have another benchmark we can look at: the performance of each Democrat against John McCain, whose campaign has now been largely newsless for several months.
And the present numbers do not show a collapse in Obama's support against John McCain. If anything, in fact, Obama's numbers have been improving. The Clinton campaign was understandably giddy about the results of the new Quinnipiac polls -- but those polls also contained good news for Obama. He gained 5 points in Pennsylvania versus Quinnipiac's previous poll of that state, and 8 points in Florida. True, he lost 2 points in Ohio. But the balance of polling data has been pretty good for Obama. He's at his highest point in our win percentage tracker since March. We don't look much at the national polling data here, but Pollster.com does, and they show Obama's numbers against McCain trending upward rather than downward:
Now, what is also true is that Clinton's numbers against McCain are moving upward. And in fact, they're moving upward faster than Obama's are moving upward:
And why shouldn't Clinton's numbers be moving upward? She has had a pretty good month:
1. Clinton, under Geoff Garin's leadership, has finally focused on one message -- Hillary as the champion of the working class -- and that message seems to be resonating.Some of these things would be real advantages in a general election matchup against John McCain, while others are more circumstantial and transient. But the point is -- the Obama campaign has generally received too much blame, and the Clinton campaign too little credit.
2. Clinton won a major victory in the Pennsylvania primary.
3. Clinton received a major surge in fundraising following her victory in the Pennsylvania primary.
4. Clinton has avoided major gaffes since the Tuzla incident.
5. Clinton's campaign has belatedly recognized that an obsession with the electoral process does not play flatteringly for her among ordinary voters, and there is no longer the sense that the Clinton Spin Machine is constantly working in overdrive.
6. Clinton routinely nails her national media appearances.
7. Clinton seems to have a bit more stamina than Obama -- an advantage over a campaign season as prolonged as this one.
8. Clinton's morale -- and the morale of her staff and supporters -- has improved as a result of all the things I've just finished describing.
9. Yes, Clinton has tended to benefit from the fact that the media has focused mostly on Obama.
10. Yes, Clinton has tended to benefit from the fact that McCain has focused mostly on Obama.
11. Yes, Clinton has tended to benefit from the fact that David Axelrod has run a fairly risk-averse campaign, and that the Obama campaign has generally (though not universally) demurred on making attacks that might undermine Clinton.
The irony, of course, is that it's very much in Clinton's interest to spin the "Obama is melting!" storyline rather than the "Clinton is surging!" storyline. Because, for all of these things -- Obama remains roughly tied or slightly ahead of Clinton in national polls of Democrats, and it's going to be very, very difficult for her campaign to argue that the superdelegates should overturn the pledged delegate count so long as that is the case.
I tend to be guilty, as most analysts do, of speaking of the superdelegates as some sort of high council that make their decisions in unison, but instead they are hundreds of individual decision-makers. By far the most likely outcome is that the primary season ends not with a bang but a whimper, and that at some point in early June Obama receives enough superdelegate endorsements to clinch the nomination. Obama's pledged delegate advantage is worth something -- a lead of about 125 pledged delegates (which is where he's likely to end up after all states have voted) means that he would only need to win the support of 42% of all superdelegates to win the nomination. Even more importantly, it means that he'll only need to win the support of 35% of all superdelegates who have not already committed to a candidate -- and the threshold is even lower when you consider that many of the remaining uncommitted superdelegates are add-ons, whose support for one candidate or another is essentially already in the bag.
For Clinton to win the nomination -- what she needs is for panic to set in. She needs a run on the Obama bank. That would probably require at least a couple of high-profile Obama superdelegates to switch their support from Obama to Clinton -- at which point the floodgates might open.
The trigger for that would be a truly shocking result in one of the remaining primaries -- such as a Clinton win in North Carolina or maybe Oregon or maybe a very close result in North Carolina coupled with a double-digit win in Indiana. Obama does benefit a tiny bit here from lowered expectations: I would guess that most pundits deep down expect Clinton to win Indiana by a Pennsylvania-type margin, and that Obama will have a fairly easy time of spinning away his loss as the same old story, provided that he has a solid victory to show for himself in North Carolina. Or, a new gaffe/scandal/inconvenience for Obama of at least Wright/bittergate magnitude (but not merely of Ayers/flag-pin magnitude). Those things remain unlikely, although the possibility of the former has somewhat increased. But there certainly hasn't been a run on Bank Obama yet, nor really even the beginnings of one -- instead, there has merely been some refreshed enthusiasm for Hillary Clinton.
5.01.2008
Today's Polls, 5/1
@ 1:23 PM
I'm running behind today and so the analysis will have to wait until later, but Quinnipiac has polling out in three major swing states, which shows very favorable results for Hillary Clinton. She leads John McCain by 14 points in Pennsylvania, 10 points in Ohio, and 8 points in Florida. Obama, meanwhile, leads McCain by 9 in Pennsylvania, but trails McCain by 1 point in the other two states. The very brief takeaway is that these polls are actually good news for both Democrats -- for Clinton more so than for Obama, obviously -- but Obama is at his highest win percentage in weeks, while Clinton's is at her highest since our tracking began.
And in New Hampshire, Rasmussen shows Clinton trailing McCain by 3 points and Obama trailing him by 10. These represent a big drop for Obama from Rasmussen's previous polling of New Hampshire, and McCain is now the solid favorite in the state against both Democrats.
And in New Hampshire, Rasmussen shows Clinton trailing McCain by 3 points and Obama trailing him by 10. These represent a big drop for Obama from Rasmussen's previous polling of New Hampshire, and McCain is now the solid favorite in the state against both Democrats.
Indiana Prediction: Toss-Up, but tilts Clinton; possible 36-36 delegate split
@ 7:30 AM
Overview
As we found out when we were trying to determine which region to place it in, Indiana is a little bit of a demographic chameleon. It shares certain demographics with Rust Belt states like Ohio, and for instance has the highest percentage of manufacturing jobs of any state in the country. It has other commonalities with the Highland/Appalachian states, particularly along its Southern boundaries. And in other ways still, such as in its tendency to vote Republican -- Indiana is one of just a handful of states to have voted more Republican than the electorate as a whole in every election since 1948 -- it behaves almost like a displaced Prairie state. Clinton has done well in the Rust Belt and Highland states, whereas Obama has done well in the Prairies, so the initial scorecard would read Clinton 2, Obama 1. But, Obama has a few small advantages which could help to patch up that gap: Indiana's border with Illinois, particularly in CD-1, and its several large colleges and universities.
Indiana's closest demographic parallel is probably Missouri, which produced the second-closest result in the primary cycle to date. As in Missouri, the intrinsic balance of demographic factors weights slightly to Clinton, but it is certainly within range for an Obama upset.
Methodology
As in Pennsylvania, these projections are based entirely on a careful balance of demographic factors. I do not look at polling numbers in any way, nor do I make subjective adjustments to the model, although I will tell you when I think one might be required. What these projections should be thought of is as a demographic baseline. At this stage, we know an awful lot about how Obama and Clinton tend to split up the Democratic vote. If one candidate winds up outperforming their numbers by more than about 5-6 points, that means that something went down differently in Indiana than it had been before.
Our regression model has changed slightly from our previous version, so let me describe its parameters.
1. Partisan Voting Index. One adjustment from the previous model is that we know use separate PVI factors for open and closed primary states. Although Clinton tends to perform better in Democratic-leaning districts, and Obama in Republican-leaning ones, these effects are stronger in closed-primary states than in open primaries like Indiana.
2. Percentage of Adults with Bachelors' Degrees or higher. Same as before.
3. Percentage of Seniors (Age 65+ Adults), out of all Adults. Same as before.
4. Percentage of Young Voters (18-29), out of all Adults. Same as before.
5. Percentage of African-American voters. Same as before.
6. Percentage of Males. Because sex ratios do not vary all that significantly from region to region, this variable struggles to achieve statistical significance, but we do find a slight advantage to Obama in districts that are more male, in alignment with what we have seen in exit polls throughout the primary cycle.
7. Percentage of "WASP"s. It would be nice to be able to look at data on religious affiliation, since Obama tends to perform well among mainline Protestants, but less well with evangelicals and Catholics. However, such data is not available on a Congressional District level. We can, however, pull some useful proxies out of the Census Bureau's data on ancestry. Specifically, while we can't guarantee that the "WASPs" in our study are actually Protestants, we can get an reasonable estimate of the number of White Anglo-Saxons. This category includes people who describe their ancestry as: British (or English), German, Danish, Dutch, Norwegian, Swedish, Scottish, Scots-Irish, Welsh, Austrian, or Swiss. And yes, this variable is highly statistically significant. (Henceforth, we will drop the quotation marks when referring to "WASP"s, but be aware of the precise way in which we define the term.)
8a/8b. Fundraising Ratios for Obama and Clinton. These replace the home state dummy variables that we'd included in the model before, as they appear to do a better job of accounting for home-state effects and other regional factors. Specifically, this is the ratio of funds raised by each of the candidates as reported to the FEC, against the total funds raised by John Kerry in 2004. So far, Obama has taken in 128% of John Kerry's haul from Indiana voters, while Clinton has taken in 96%. This compares to their national averages of 95% and 80% respectively. Note that this variable is applied at the statewide level, as it is not available by Congressional District.
This is, I think, a somewhat more intuitive set of variables than we'd had before, and it also has slightly more explanatory power. Naturally, I have incorporated the results of Pennsylvania into the model, and also caught up with Mississippi and Tennessee, which were missing before. However, we're still out data on Texas (which uses state senate districts rather than CDs to allocate its vote) and New Jersey (which seems to have taken a cue from Tony Soprano and not fully disclosed its activities to the general public).
Congressional District Demographics and Projections
CD-1 (NW/Gary): This district, in the Northwest part of the state, is one that falls within the sphere of influence of greater Chicago. Does that matter? Our evidence, which we'll get into below, suggests a very qualified 'yes', such that you might append a point or two to Obama's numbers. It's also the second blackest district in the state, although there are some electoral negatives for Obama too, as the whites in the district tend to be working class. Because of the regional factors, you should probably consider the district as leaning Obama, but two things are fairly certain: (i) neither Democrat will rack up enough margin to break the 3-3 delegate deadlock; (ii) if Obama does carry Indiana, he'll probably have had to win this district. It's also a district that votes, so turnout should be quite high. Prediction: Clinton 50.4, Obama 49.6; 3-3 Delegate Split.
CD-2 (NNW / South Bend): A fairly average Indiana district, and probably the closest Indiana comes to a swing district in general elections. It's southern portions are small town and industrial, but Obama will do better along the Lake Shore and might get an assist from Notre Dame in South Bend. On a good day, Clinton could pull out a 4-2 delegate split win, but that's not the most likely outcome. Prediction: Clinton 54.1, Obama 45.9; 3-3 Delegate Split.
CD-3 (NE / Fort Wayne): Looks like trouble for Obama at first, as it borders OH-5, which Clinton carried 60:40. But this district is a bit more heterogeneous, with at least a handful of black voters in and around Fort Wayne, and a somewhat younger, better educated electorate than OH-5. It also has the highest percentage of WASPs in the state. My prediction, I think, nevertheless looks a little optimistic for Obama, especially if the Limbaugh vote turns out. But it would be an extremely tall order for Clinton to win a 3-1 delegate split, which would require carrying 62.5% of the vote. Prediction: Clinton 50.8, Obama 49.2; 2-2 Delegate Split.
CD-4 (Central / Lafayette): Obama is saved by Purdue University in what would otherwise be a mediocre district for him; PU is even better than a usual college for Obama because it's an engineering school and therefore tilts male. This is a strongly Republican district, so who wins the popular vote will be determined by crossover votes, but the delegate split will almost certianly be 2-2. Prediction: Clinton 50.3, Obama 49.6; 2-2 Delegate Split.
CD-5 (Central / Marion): Home to Indiana's bourgeoisie, the demographic here is well-educated and WASPy Indianapolis suburbanites. Accordingly, Obama rates as the slight favorite, though this tends to be a very reliable and high-turnout Republican district and so the Limbaugh vote could be a factor. Yet again, this district is nondramatic from a delegate perspective, as the split should be 2-2. Prediction: Obama 53.1, Clinton 46.9; 2-2 Delegate Split.
CD-6 (East / Muncie): The oldest, most rural, and least-educated district in Indiana, it's very probably Obama's worst, although none of these demographics are as extreme as Obama's poorest-performing areas in Pennsylvania and Ohio. It's also the only district in the state with an odd number of delegates, and Clinton will win the spare, even if Obama is having a good day otherwise. Prediction: Clinton 58.8, Obama 42.2; Clinton 3-2 Delegate Win.
CD-7 (Indianapolis): And we jauntily go from Obama's worst district to his best, the largely urban CD-7, where he has the endorsement of newly-elected André Carson. The key number to watch is 58.3%, which is the vote split that Obama would need to pick up a 4th delegate. That should be a fairly safe proposition, although this is also the district most likely to be impacted by Indiana's voter ID requirement. Prediction: Obama 61.5, Clinton 38.5; Obama 4-2 Delegate Win.
CD-8 (SE / Evansville): About the only positive for Obama is that the district shares a border with Illinois, although not really any of its media markets. With that said, the demographics, while leaning fairly strongly Clinton, are not especially extreme. Clinton would only need to overperform by a point or so to pick up a 4-2 delegate victory, an outcome I consider as likely as not. Prediction: Clinton 57.5, Obama 43.5; 3-3 Delegate Split.
CD-9 (SW / Bloomington): Bordering Kentucky, this would otherwise be a just awful district for Obama but for two things: the endorsement of Lee Hamilton and, more importantly, Indiana University in Bloomington. The reason you're seeing a lot of campaign activity here is because it's quite close to the threshold where Clinton could win a 4-2 delegate split, but if Bloomington turns out, I don't quite see that happening -- CD-8 is a better target for her. Prediction: Clinton 57.1, Obama 43.9; 3-3 Delegate Split.
Projected Statewide Results and Turnout
I have also somewhat revised my turnout model. By pulling together data from Swing State Project, I was able to estimate the number of votes that John Kerry received in each CD in 2004. This was accomplished by multiplying Kerry's vote share by the turnout in the House of Representatives election in each district. (Nationwide, there was about a 7.5% undervote in the combined House of Representatives vote relative to the Presidential vote. However much of this is because some Congressional candidates had no opposition, and so I assume that the Presidential Election turnout was 2.5% higher in each district than the Congressional Election turnout).
Thus, I am able to estimate the primary turnout in each district as a percentage of the Kerry vote. This varies according to several factors, the most important of which are whether the state has an open primary (turnout increases by about 30% when it does), and the combined number of days that the candidates spent in the state in the 30 days leading to the election (I assume that Obama and Clinton will each be spending a couple more days in Indiana over the weekend). Other factors that have a smaller degree of influence are:
- Black and Hispanic turnout: minorities have turned out as a slightly larger share of the primary electorate this year
- The urban/rural distribution of the state: turnout has slightly overperformed in rural areas
- The number of college and graduate students in the state: these voters may have registration issues and are slightly less likely to turn out in the primaries than in the general election, and
- Whether the contest occurred on Super Tuesday: contests held on Super Tuesday had somewhat lower turnout than those held before or afterward.
We are projecting overall turnout of about 925,000 persons in Indiana. Overall results follow.
Our projection is for a Clinton win by 2.0 percentage points, but for an even split in the delegate count. From a delegate perspective, the swing districts in this contest are in CD-8 and CD-9, where Clinton could earn a 4-2 split on a good day. From a popular vote perspective, however, it will be Obama's ability to turn out his base in CD-1 and CD-7 that counts.
Remember, however, that there is a standard error of roughly 6 points around this estimate, so anything from a Clinton win by 8 points to an Obama win by 4 points would not be a major surprise. Let's close by looking at three wild cards in Indiana that could lead the actual results to come in toward the higher or lower end of that range.
Wild Card #1. The Limbaugh Vote.
After calling off his dogs on Tuesday, Rush Limbaugh has re-initiated Operating Chaos, asking his listeners to vote for Hillary Clinton to prolong the Democrats' nomination process. There is, I think, reasonable circumstantial evidence from the exit polls that Limbaughs's instructions netted Clinton somewhere in the range of 2-4 points in Texas, Ohio, and Mississippi. Mitigating factors: Limbaugh's effort seems to have a somewhat lower profile this time around, and there is also an obscure (though probably unenforceable) Indiana law that allows voters to be 'challenged' if they don't intend to vote for the Democratic candidate in the General Election. On the other hand, the right-wing media has been so harsh to Obama recently that it's easy to imagine a lot of Republican voters sort of picking up on that message subliminally, and turning out to vote for Clinton. On the other other hand, Republicans no longer believe that Obama is the tougher opponent for McCain, so if they're really voting tactically, perhaps some Republicans will want to ensure a contest against Barack Hussein McBitter. My horse sense is that this dynamic plays out favorably for Clinton overall, and is worth a couple of points.
Wild Card #2. Indiana's Voter Identification Requirement.
On Monday, the Supreme Court upheld a law requiring Indiana voters to show a government-issued photo ID at the polls. Note that this is not a new law. It went into effect in 2005, and so if the campaigns have really been doing their homework, it's something they should have been prepared for. Voters may also cast a provisional ballot without an ID, but it is their burden to prove their residency if they do. (If the election turns out to be especially close, one can imagine a hanging chads type of situation in the days after the primary, with AFSCME and SEIU shuttling nearsighted grannies back and forth between the DMV and their local election board).
Academic studies suggest that this can be expected to depress turnout by 2-3 percent. But the groups that it is supposed to affect cut pretty evenly across Clinton and Obama demographics. Old voters seem to be disproportionately affected, but so do students. Black voters seem to be affected more strongly, but there are also more women without identification than men. Poorer people are more affected than older people.
I actually think this factor tips slightly to Obama for a couple of reasons. A higher-information voter will conceivably go through more trouble to make sure their identification is current than a lower-information voter, and that description tends to apply to Obama supporters. And Obama may have a slight organizational advantage in Indiana, with an army of student and union volunteers streaming in from Chicago. Also worth noting: the only other state with a similar requirement is Georgia, and Obama blew out his polling numbers there. Call it somewhere between 0 and 0.5 points for Obama.
Wild Card #3. Home Field Advantage?
To some extent, any neighboring-state advantage that Obama has might be reflected in his fundraising numbers, which form part of our regression model. But it's worth looking at whether the model would have underpredicted his margin in Congressional Districts that border Illinois. There are three CDs in Wisconsin and four in Missouri that border Illinois, not counting MO-8, which borders Illinois but also borders Arkansas. And here is how Obama performed against his prediction in each of those districts:
District Predicted Actual +/-
WI-1 51.6 52.7 +1.1
WI-2 61.6 66.0 +4.4
WI-3 54.8 57.1 +2.3
MO-1 73.1 73.7 +0.6
MO-2 55.5 53.5 -2.0
MO-3 49.9 51.1 +1.2
MO-9 48.0 47.8 -0.2
AVERAGE 56.4 57.4 +1.0
Overall, Obama outperformed his projected vote share by about one point in these districts, which would translate to a 2-point swing since a vote he gains is one taken away from Clinton. Also, although we didn't include Iowa in this table because it's a caucus, Obama performed quite a bit better in IA-1 and IA-2, which border Illinois, than he did throughout the rest of the state.While this is hardly overwhelming evidence -- the trend is nowhere near statistically significant -- it would not contradict the idea that Obama might overperform slightly in CD-1, and to a lesser extent perhaps in CD-2 and CD-8.
*-*
The Limbaugh vote appears to me the most tangible of these three factors. That coupled with the still-tepid state of Obama's prevailing media narrative would lead me to bet on Hillary's side of our Clinton +2 projection. But things should nevertheless be very close, and Indiana remains either candidate's state to lose.
4.30.2008
Voters v. Media: a concerning result
@ 11:42 PM
The MSNBC evening lineup gave a fair amount of play tonight to a couple of results from the new NBC/Wall Street Journal Poll. This was sort of like a non-partisan push poll: voters were read a series of nine negative statements -- three for each of the three remaining candidates -- and asked about their level of concern on each one.
MSNBC spent most of their time focusing on two of these issues -- what I call "Obama/bitter" (his recent comments on religion and guns) and "Obama/values" (his associations with Jeremiah Wright and William Ayers). You might assume, from having seen the programming, that these were the foremost concerns to their poll respondents. But of course they weren't.
Below, I have translated their polling results to a Likert Scale, where 1 represents "No Real Concern" and 4 represents a "Major Concern".
Judged by this metric, "Obama/bitter" and "Obama/values" rank just fifth and sixth on the list, respectively. Instead, the issue that concerns the voters the most is "Clinton/flip-flops". That is followed by two McCain issues -- his closeness with George W. Bush and his own flip-flops (to its credit, MSNBC did report the Bush result on air) -- and then another Clinton ("Clinton/honesty") before we get to the first two Obama issues. Meanwhile, the two issues that strike me as unambiguously being cheap shots -- "McCain/age" and "Obama/patriotism" -- are of very little concern to voters.
If you had to rank these nine issue in terms of how much attention the media has given to them recently, what would that result look like? My guess is something like:
1. Obama / values (6)
2. Clinton / Bill (7)
(tie) Obama / bitter (5)
(tie) Clinton / honesty (4)
5. McCain / age (8)
6. McCain / Bush (2)
(tie) Clinton / flip-flops (1)
(tie) Obama / patriotism (9)
9. McCain / flip-flops (3)
Of course, this is a subjective exercise, and my judgments about where the media has been spending its time could be wrong (I really think the week I did the best job of political analysis was the week that my cable was broken). But from what I can tell, there is actually something of an inverse relationship between those issues the media has spent the most time focusing on, and those that are of the most concern to voters in the media's own polling.
Not that this is breaking any news.
MSNBC spent most of their time focusing on two of these issues -- what I call "Obama/bitter" (his recent comments on religion and guns) and "Obama/values" (his associations with Jeremiah Wright and William Ayers). You might assume, from having seen the programming, that these were the foremost concerns to their poll respondents. But of course they weren't.
Below, I have translated their polling results to a Likert Scale, where 1 represents "No Real Concern" and 4 represents a "Major Concern".
Judged by this metric, "Obama/bitter" and "Obama/values" rank just fifth and sixth on the list, respectively. Instead, the issue that concerns the voters the most is "Clinton/flip-flops". That is followed by two McCain issues -- his closeness with George W. Bush and his own flip-flops (to its credit, MSNBC did report the Bush result on air) -- and then another Clinton ("Clinton/honesty") before we get to the first two Obama issues. Meanwhile, the two issues that strike me as unambiguously being cheap shots -- "McCain/age" and "Obama/patriotism" -- are of very little concern to voters.
If you had to rank these nine issue in terms of how much attention the media has given to them recently, what would that result look like? My guess is something like:
1. Obama / values (6)
2. Clinton / Bill (7)
(tie) Obama / bitter (5)
(tie) Clinton / honesty (4)
5. McCain / age (8)
6. McCain / Bush (2)
(tie) Clinton / flip-flops (1)
(tie) Obama / patriotism (9)
9. McCain / flip-flops (3)
Of course, this is a subjective exercise, and my judgments about where the media has been spending its time could be wrong (I really think the week I did the best job of political analysis was the week that my cable was broken). But from what I can tell, there is actually something of an inverse relationship between those issues the media has spent the most time focusing on, and those that are of the most concern to voters in the media's own polling.
Not that this is breaking any news.
Clinton +2 in North Carolina?
@ 11:02 PM
Polling Southern primaries tends to be difficult, and Insider Advantage does at least a competent job at it. But their new poll showing Clinton 2 points ahead of Barack Obama is a little odd. Clinton leads 54-33 among white voters, a result that is not great news for Obama but is fairly consistent with other polling in the state, including polls where he has a double-digit lead.
The difference is in the black vote. Firstly, Obama leads that vote "only" 64-20, with 15% undecided, a vote he has won 83-15 over the course of the primary season. Secondly, the poll shows only 25% of the turnout being black, which would be a very low result. The general rule of thumb is that black turnout equals about 150% of a state's African-American population, which in North Carolina's case would work out to 33%.
Of course it's always possible that there has been a black backlash of sorts against Obama's distancing himself from Jeremiah Wright. But Insider Advantage has a history of weird things going on in their cross-tabular reporting. They also have a history, not unlike ARG, of publishing weird results in the run-up to the primaries (Clinton +33 in Tennessee, Clinton +8 and Edwards +7 over Obama in Iowa) and then fixing those results with more reasonable looking results just before the primary actually takes place. One disadvantage of our pollster rating methodology is that we only look at the last poll before an election. Insider Advantage could be into something here, but if not, we might need to revisit that assumption.
The difference is in the black vote. Firstly, Obama leads that vote "only" 64-20, with 15% undecided, a vote he has won 83-15 over the course of the primary season. Secondly, the poll shows only 25% of the turnout being black, which would be a very low result. The general rule of thumb is that black turnout equals about 150% of a state's African-American population, which in North Carolina's case would work out to 33%.
Of course it's always possible that there has been a black backlash of sorts against Obama's distancing himself from Jeremiah Wright. But Insider Advantage has a history of weird things going on in their cross-tabular reporting. They also have a history, not unlike ARG, of publishing weird results in the run-up to the primaries (Clinton +33 in Tennessee, Clinton +8 and Edwards +7 over Obama in Iowa) and then fixing those results with more reasonable looking results just before the primary actually takes place. One disadvantage of our pollster rating methodology is that we only look at the last poll before an election. Insider Advantage could be into something here, but if not, we might need to revisit that assumption.
This Morning's Polls, 4/30
@ 7:44 AM
In New Jersey, a Monmouth/Gannett poll shows impressive-sized leads for both Democrats: Hillary Clinton leads John McCain by 14 points, while Barack Obama leads him by fully 24 points. This result is so optimistic for the Democrats as to frankly stretch credulity, and I know very little about the Monmouth/Gannett polling shop. At the same time, there have been a handful of decidely optimstic polling results for the Democrats in recent days (such as their performance in Indiana in the Selzer poll), while at the same time both are doing fairly well in the Rasmussen tracker. The conventional wisdom that the enlongated primary season may be hurting the Democrats does not have a lot of evidenciary wisdom behind it. As I have said before, it isn't when it ends that the problem, but how it ends that counts.
Also, in Arizona, an ASU/Cronkite poll has Barack Obama trailing home-stater John McCain by 9 points, and Hillary Clinton trailing him by 16. Home state effects typically run at about 6-7 points (although vary a lot from candidate to candidate), and so Arizona might be a fringe-competitive state for Obama if John McCain were not the nominee, though probably not for Clinton.
Although some of this is the result of our recent methodological changes, Barack Obama's Win Percentage is presently the highest it has been since April 2, and his electability edge over Clinton is the largest it has been since March 31.
Also, in Arizona, an ASU/Cronkite poll has Barack Obama trailing home-stater John McCain by 9 points, and Hillary Clinton trailing him by 16. Home state effects typically run at about 6-7 points (although vary a lot from candidate to candidate), and so Arizona might be a fringe-competitive state for Obama if John McCain were not the nominee, though probably not for Clinton.
Although some of this is the result of our recent methodological changes, Barack Obama's Win Percentage is presently the highest it has been since April 2, and his electability edge over Clinton is the largest it has been since March 31.
Pollster Ratings, v3.1
@ 7:38 AM
Thanks to some helpful comments on the previous version of these ratings, I have cleaned up several sloppy mathematical assumptions. I have also added new polling data from roughly 12 contests in the 2004 primaries into the database.
I am going to go ahead and duplicate most of the (long) discussion from the previous version, such that this article stands on its own. For those who are just browsing through, however, here are the topline results: our estimate of the amount of error that each pollster introduces as a result of methodological error. Note that these estimates are designed to be applied to general election polling data; the errors would be much higher across the board if we were looking at the primaries.
Full Methodological Description
I presently have a database of 171 different electoral contests since 2000 that have been surveyed by at least three of the 32 pollsters that I include in my study. These include contests for President, Senate and Governor, as well as polls from the presidential primaries. A poll is included if (i) it is the last poll that an agency put out before the election; (ii) it was released no later than two weeks from the election date.
My goal is to isolate the instance of methodological error, or what I call Pollster-Introduced Error (PIE). If you look at a poll at any given time before an election, there are essentially three different sources of error:
Total Error = Sampling Error + Temporal Error + PIE
Sampling Error is the error that pollsters typically report in their margin of error calculations -- the error intrinsic to sampling only a subset of the entire voting population. All polls have sampling error, though of course a pollster can reduce it by including more interviews in its sample.
Temporal Error is the error introduced by taking a poll weeks or months before an election. Temporal Error is a major consideration now, in April, when we are looking at polls of the November general election; many things can happen between now and then, and (contrary to the common perception) it is not up to the pollster to predict the future. Temporal Error is incorporated into our model in terms of the uncertainty we build into our estimates. For purposes of the pollster ratings, however, we can ignore Temporal Error (that is, assume it to be zero), because we are limiting our evaluation to polls taken very near to the election date.
That leaves us with our final source of error, Pollster-Introduced Error. PIE is what a tennis aficionado might call "Unforced Error"; it is error that results from poor methodology. As a matter of practice, all pollsters have some PIE, which is why the actual margins of error are always larger than those espoused by the pollster. However, this amount varies fairly significantly from agency to agency -- which is the impetus for producing these ratings.
To find the PIE, we will first calculate the Total Error for each pollster, and then deduct its Sampling Error. We begin with a calculation called the Raw Total Error (RTE), which is simply the average number of 'points' by which a pollster missed the final margin in a given contest, weighted by the number of pollsters that surveyed that contest. We then compare this against what we call the Iterated Average Error (IAE), which is average error for other pollsters that surveyed the same contest, as determined by a multiple iteration method. This step is functionally equivalent to that which we employed in Version 2.0 of our pollster analysis and is described at greater length there. We then subtract the IAE from the RTE to produce a +/- rating. A negative rating means that the pollster outperformed its peers, while a positive rating means that it underperformed. The +/- score for each pollster in our database is indicated below:
The next step is to move from Raw Total Error to Adjusted Total Error (ATE); this in turn requires two substeps, each of which are fairly simple.
Firstly, we add back in the average error for all general election polls in our database to the +/- rating. This figure happens to be 3.81 points. Note that, in Version 3.0 of these ratings, the global average error was determined based on a combination of primary and general election polling. This is because I was worried that the average error in primary polls had been especially high thus far in 2008 (7.17 points), and I was concerned that there was something intrinsic to this electoral cycle that was leading to the higher errors. However, I have now added more polls from previous years' primaries to the database, and determined that the average error was just as high in those primary cycles (in fact, somewhat higher; the average error was 8.27 points over about 16 contests between the 2000 and 2004 primaries). Polling primaries is inherently much, much more difficult than polling general elections for three principal reasons: (i) voters selecting from among several different candidates in their own party often like several of the candidates -- it therefore takes less to move them from one candidate to another; (ii) voters have less information about the candidates at the primary stage of the process, and their preferences may change quickly as they obtain more information -- they also often make up their minds late; (iii) turnout is much less predictable in primaries than it is in general elections. However, since this is a website dedicated to general election polling, we use general election polls to calibrate our results.
The next step is to regress the results of this calculation to the mean. Contrary to what you might hear from a pundit, who might evaluate a pollster based on the success of its most recent poll, there is a lot of 'luck' involved in polling. Looking at just a couple of polls will tell you very little about a pollster, and even with as many as 100 polls, there is still fairly significant regression to the mean.
This regression is based on a calculation of the standard error of the mean -- more specifically, we regress to the mean error expected based on the pollster's sample size. Note, however, that we do not regress to the mean for two agencies: Zogby Interactive and Columbus Dispatch. This is because these two pollsters use unconventional methodologies -- Internet-based polls and mail polls, respectively, which evidently have resulted in very poor outcomes. There is no reason to give a pollster credit for regression toward the mean when it uses an untested methodology that should intrinsically be associated with larger methodological errors.
Adding back in the global average error and then regressing to the mean gives us our Adjusted Total Error. We are almost ready to infer the Pollster-Introduced Error, but first we need to determine how much of the ATE is attributable to Sampling Error.
The expected sampling error associated with a given sample size is fairly trivial to determine, and can be inferred by means of a binomial distribution. It is approximated by the formula...
80 * n^(-.5)
...where 'n' is the number of respondents in the poll. Technically, the expected sampling error varies according to the intrinsic distribution of candidate preferences throughout the population; the sampling error will be somewhat larger when preferences are divided 50:50 than when they are skewed 80:20 toward one candidate. This can be understood intuitively by the fact that when 100 percent of the population prefers a given candidate -- think a Baath Party election -- your poll will return a perfect result regardless of its sample size. This effect is so trivial, however, that it may be ignored for purposes of this analysis.
Thus, we simply average the expected sampling error for all polls that we have in our database from a given pollster, weighted based on the number of pollsters that surveyed that contest. This result is known as the Average Expected Sampling Error (AESE). The AESE varies from pollster to pollster because some pollsters habitually include more respondents than others, although all bit a few gravitate toward the 500-700 respondent range over the long run.
The PIE -- Pollster Introduced Error -- can then be determined by deducting the AESE from the ATE. However, this is not a simple, linear subtraction; instead the errors are related by the sum-of-squares formula. (This was one of those steps that we had gotten wrong in Version 3.0 of this analysis). Specifically:
...and therefore:
The PIEs for each pollster (the same ones we listed in the very first table in this article) are provided below:
The basic way to interpret the PIE is that it is the amount of error that a pollster introduces because of imperfect methodology, in addition to any error that results from its finite sample size. A SurveyUSA poll, for instance, adds only about six-tenths of a point of error more than a methodologically perfect pollster would, while a Marist poll adds more than two-and-a-half points of error. We also apply a generic PIE of +2.19 to all unknown pollsters. This is slightly higher the PIE of the average pollster because we have found that there is some relationship between the number of polls that a pollster produces and its average error -- pollsters that release a lot of polls tend to be better than average (no doubt because they get more repeat business).
The essential lesson here is not to give deference to 'name-brand' pollsters, or those that are associated with large news organizations. Two of our top three are Internet-friendly operations that use the IVR ('robocall') method, while several other highly-rated pollsters are boutique or academic shops that limit their polling to a given region. The strongest pollsters associated with a large news organization are Mason-Dixon, which conducts polls for The McClatchy Company and MSNBC, and Market Shares, which polls for Tribune Company. However, other major media polling shops, such as FOX/Opinion Dynamics and CBS/New York Times, have considerably underperformed over time. Likewise some 'brand name' pollsters like Zogby and Gallup have had quite poor results.
The Final Step: Translating Ratings into Weightings
However, we are not quite done yet. While these numbers are interesting in the abstract, they do not tell us how to weight polls when we have more than one poll for a given contest. This final step is resolved by figuring what I call the Effective Sample Size.
Suppose that we have a Rasmussen poll with 500 respondents. We have specified Rasmussen's long-run PIE to be 1.22 points, whereas the average sampling error associated with a poll of 500 respondents is 3.58 points. The total expected error for this poll is determined by the sum-of-squares of thse two figures, or 3.78 points.
To determine the Effective Sample Size, we must answer the following question: how many respondents would a methodologically perfect have to include for it to have an expected error of 3.78 points? That can be determined by the following equation:
Effective Sample Size = 6400 * (Total Expected Error^-2)
448 = 6400 * (3.78 ^ -2)
In this case, the answer is 448. That is, we would be indifferent between a Rasmussen poll consisting of 500 respondents and a theoretically perfect poll consisting of 448 respondents. Thus, the Effective Sample Size for this poll is 448. Rasmussen is a strong pollster, and so this is a strong result. By contrast, a Zogby poll consisting of 500 respondents would have an Effective Sample Size of 353, whereas a Columbus Dispatch poll of 500 respondents would have an Effective Sample Size of just 93!
The weight we assign to each poll is directly proportional to the Effective Sample Size. For aesthetic purposes, we represent the weighting as a ratio, taken relative to a poll of 600 respondents by a pollster of average quality (long-run PIE of 1.96). Such a poll would have an Effective Sample Size of 441. Thus, the weighting for the Rasmussen poll would be 448/441 = 1.02, for the Zogby poll would be 353/ 441 = 0.80, and for the Columbus Dispatch poll would be 93 / 441 = 0.21. Note that this is before applying any penalties for the recentness of a poll. The recentness rating is still applied as described in the FAQ, and multiplied by the Effective Sample Size calculation to produce the final weighting.
This method has one very interesting quality -- namely, that the extent to which we weight one poll relative to another depends on the amount of data we have. When the sample sizes are small, the Sampling Error is large relative to the Pollster-Introduced Error, and therefore there is relatively little difference in the weights assigned to pollsters of varying quality. As sample sizes approach infinity, however, the Sampling Error dwindles toward zero, and nearly the entire difference in the quality of different polls is determined by the Pollster-Introduced Error.
This can be seen in the graph below, where we compare the Effective Sample Sizes of a 'good' pollster (SurveyUSA) and a 'bad' pollster (American Research Group) at various levels of actual sample sizes.
As you can see, the Effective Sample Size for the ARG poll levels off much more quickly than the Effective Sample Size for the SurveyUSA poll. If we have just a little bit of data from each pollster, our primary concern is simply the reducing the effects of Sampling Error, and so we are relatively indifferent about which pollsters we use. However, once we have a lot of data, we have the luxury of being far more discriminating. We'll take nearly as much SurveyUSA data as we can get our hands on, for instance, but for ARG, we are much more cautious, because no matter how much ARG data we have, it is still going to be subject to ARG's methodological deficiencies. In fact, even if we had an infinite amount of ARG data, it would still max out at an Effective Sample Size of 1,113! (SurveyUSA's theoretical maximum, by contrast, goes all the way up to 17,335).
You might wonder why I have the graph extending all the way out to 5,000 respondents when in practice it is very rare to find a poll with more than about 1,200 or 1,500 respondents. The reason is that while we might not find any one poll with a very large number of respondents, we may get effectively the same thing if we have several different polls from the same agency. For example, in the run-up to the Pennsylvania primary, Strategic Vision conducted general election trial heats of 1,200 respondents in each of five consecutive weeks. In many respects, this is similar to one poll of 6,000 respondents. And from the standpoint of our pollster weightings, we treat it largely the same way.
Specifically, when we have more than one poll from a given agency in a given state, we aggregate the sample sizes, and perform the Effective Sample Size calculation on this basis. We then subtract out the Effective Sample Sizes of any more recent surveys from that pollster to determine the Marginal Effective Sample Size, or MESS (yes, I know you are getting sick of all these acronyms). For example, for a set of polls like the Strategic Vision polls, the calculation would work as follows:
The central concept is pretty intuitive, which is that the more and more data we get from a given pollster, the more we encounter diminishing returns.
Also note that in addition to the above calculation, we also punish older polls based on their recentness rating. So effectively we have two different ways to penalize redundant polling from the same agency. But it is a mistake to throw out old polls from the same agency completely; getting SurveyUSA's sloppy seconds may be as good as getting virgin results from a lot of pollsters.
I am going to go ahead and duplicate most of the (long) discussion from the previous version, such that this article stands on its own. For those who are just browsing through, however, here are the topline results: our estimate of the amount of error that each pollster introduces as a result of methodological error. Note that these estimates are designed to be applied to general election polling data; the errors would be much higher across the board if we were looking at the primaries.
Full Methodological Description
I presently have a database of 171 different electoral contests since 2000 that have been surveyed by at least three of the 32 pollsters that I include in my study. These include contests for President, Senate and Governor, as well as polls from the presidential primaries. A poll is included if (i) it is the last poll that an agency put out before the election; (ii) it was released no later than two weeks from the election date.
My goal is to isolate the instance of methodological error, or what I call Pollster-Introduced Error (PIE). If you look at a poll at any given time before an election, there are essentially three different sources of error:
Total Error = Sampling Error + Temporal Error + PIE
Sampling Error is the error that pollsters typically report in their margin of error calculations -- the error intrinsic to sampling only a subset of the entire voting population. All polls have sampling error, though of course a pollster can reduce it by including more interviews in its sample.
Temporal Error is the error introduced by taking a poll weeks or months before an election. Temporal Error is a major consideration now, in April, when we are looking at polls of the November general election; many things can happen between now and then, and (contrary to the common perception) it is not up to the pollster to predict the future. Temporal Error is incorporated into our model in terms of the uncertainty we build into our estimates. For purposes of the pollster ratings, however, we can ignore Temporal Error (that is, assume it to be zero), because we are limiting our evaluation to polls taken very near to the election date.
That leaves us with our final source of error, Pollster-Introduced Error. PIE is what a tennis aficionado might call "Unforced Error"; it is error that results from poor methodology. As a matter of practice, all pollsters have some PIE, which is why the actual margins of error are always larger than those espoused by the pollster. However, this amount varies fairly significantly from agency to agency -- which is the impetus for producing these ratings.
To find the PIE, we will first calculate the Total Error for each pollster, and then deduct its Sampling Error. We begin with a calculation called the Raw Total Error (RTE), which is simply the average number of 'points' by which a pollster missed the final margin in a given contest, weighted by the number of pollsters that surveyed that contest. We then compare this against what we call the Iterated Average Error (IAE), which is average error for other pollsters that surveyed the same contest, as determined by a multiple iteration method. This step is functionally equivalent to that which we employed in Version 2.0 of our pollster analysis and is described at greater length there. We then subtract the IAE from the RTE to produce a +/- rating. A negative rating means that the pollster outperformed its peers, while a positive rating means that it underperformed. The +/- score for each pollster in our database is indicated below:
The next step is to move from Raw Total Error to Adjusted Total Error (ATE); this in turn requires two substeps, each of which are fairly simple.
Firstly, we add back in the average error for all general election polls in our database to the +/- rating. This figure happens to be 3.81 points. Note that, in Version 3.0 of these ratings, the global average error was determined based on a combination of primary and general election polling. This is because I was worried that the average error in primary polls had been especially high thus far in 2008 (7.17 points), and I was concerned that there was something intrinsic to this electoral cycle that was leading to the higher errors. However, I have now added more polls from previous years' primaries to the database, and determined that the average error was just as high in those primary cycles (in fact, somewhat higher; the average error was 8.27 points over about 16 contests between the 2000 and 2004 primaries). Polling primaries is inherently much, much more difficult than polling general elections for three principal reasons: (i) voters selecting from among several different candidates in their own party often like several of the candidates -- it therefore takes less to move them from one candidate to another; (ii) voters have less information about the candidates at the primary stage of the process, and their preferences may change quickly as they obtain more information -- they also often make up their minds late; (iii) turnout is much less predictable in primaries than it is in general elections. However, since this is a website dedicated to general election polling, we use general election polls to calibrate our results.
The next step is to regress the results of this calculation to the mean. Contrary to what you might hear from a pundit, who might evaluate a pollster based on the success of its most recent poll, there is a lot of 'luck' involved in polling. Looking at just a couple of polls will tell you very little about a pollster, and even with as many as 100 polls, there is still fairly significant regression to the mean.
This regression is based on a calculation of the standard error of the mean -- more specifically, we regress to the mean error expected based on the pollster's sample size. Note, however, that we do not regress to the mean for two agencies: Zogby Interactive and Columbus Dispatch. This is because these two pollsters use unconventional methodologies -- Internet-based polls and mail polls, respectively, which evidently have resulted in very poor outcomes. There is no reason to give a pollster credit for regression toward the mean when it uses an untested methodology that should intrinsically be associated with larger methodological errors.
Adding back in the global average error and then regressing to the mean gives us our Adjusted Total Error. We are almost ready to infer the Pollster-Introduced Error, but first we need to determine how much of the ATE is attributable to Sampling Error.
The expected sampling error associated with a given sample size is fairly trivial to determine, and can be inferred by means of a binomial distribution. It is approximated by the formula...
80 * n^(-.5)
...where 'n' is the number of respondents in the poll. Technically, the expected sampling error varies according to the intrinsic distribution of candidate preferences throughout the population; the sampling error will be somewhat larger when preferences are divided 50:50 than when they are skewed 80:20 toward one candidate. This can be understood intuitively by the fact that when 100 percent of the population prefers a given candidate -- think a Baath Party election -- your poll will return a perfect result regardless of its sample size. This effect is so trivial, however, that it may be ignored for purposes of this analysis.
Thus, we simply average the expected sampling error for all polls that we have in our database from a given pollster, weighted based on the number of pollsters that surveyed that contest. This result is known as the Average Expected Sampling Error (AESE). The AESE varies from pollster to pollster because some pollsters habitually include more respondents than others, although all bit a few gravitate toward the 500-700 respondent range over the long run.
The PIE -- Pollster Introduced Error -- can then be determined by deducting the AESE from the ATE. However, this is not a simple, linear subtraction; instead the errors are related by the sum-of-squares formula. (This was one of those steps that we had gotten wrong in Version 3.0 of this analysis). Specifically:
...and therefore:
The PIEs for each pollster (the same ones we listed in the very first table in this article) are provided below:
The basic way to interpret the PIE is that it is the amount of error that a pollster introduces because of imperfect methodology, in addition to any error that results from its finite sample size. A SurveyUSA poll, for instance, adds only about six-tenths of a point of error more than a methodologically perfect pollster would, while a Marist poll adds more than two-and-a-half points of error. We also apply a generic PIE of +2.19 to all unknown pollsters. This is slightly higher the PIE of the average pollster because we have found that there is some relationship between the number of polls that a pollster produces and its average error -- pollsters that release a lot of polls tend to be better than average (no doubt because they get more repeat business).
The essential lesson here is not to give deference to 'name-brand' pollsters, or those that are associated with large news organizations. Two of our top three are Internet-friendly operations that use the IVR ('robocall') method, while several other highly-rated pollsters are boutique or academic shops that limit their polling to a given region. The strongest pollsters associated with a large news organization are Mason-Dixon, which conducts polls for The McClatchy Company and MSNBC, and Market Shares, which polls for Tribune Company. However, other major media polling shops, such as FOX/Opinion Dynamics and CBS/New York Times, have considerably underperformed over time. Likewise some 'brand name' pollsters like Zogby and Gallup have had quite poor results.
The Final Step: Translating Ratings into Weightings
However, we are not quite done yet. While these numbers are interesting in the abstract, they do not tell us how to weight polls when we have more than one poll for a given contest. This final step is resolved by figuring what I call the Effective Sample Size.
Suppose that we have a Rasmussen poll with 500 respondents. We have specified Rasmussen's long-run PIE to be 1.22 points, whereas the average sampling error associated with a poll of 500 respondents is 3.58 points. The total expected error for this poll is determined by the sum-of-squares of thse two figures, or 3.78 points.
To determine the Effective Sample Size, we must answer the following question: how many respondents would a methodologically perfect have to include for it to have an expected error of 3.78 points? That can be determined by the following equation:
Effective Sample Size = 6400 * (Total Expected Error^-2)
448 = 6400 * (3.78 ^ -2)
In this case, the answer is 448. That is, we would be indifferent between a Rasmussen poll consisting of 500 respondents and a theoretically perfect poll consisting of 448 respondents. Thus, the Effective Sample Size for this poll is 448. Rasmussen is a strong pollster, and so this is a strong result. By contrast, a Zogby poll consisting of 500 respondents would have an Effective Sample Size of 353, whereas a Columbus Dispatch poll of 500 respondents would have an Effective Sample Size of just 93!
The weight we assign to each poll is directly proportional to the Effective Sample Size. For aesthetic purposes, we represent the weighting as a ratio, taken relative to a poll of 600 respondents by a pollster of average quality (long-run PIE of 1.96). Such a poll would have an Effective Sample Size of 441. Thus, the weighting for the Rasmussen poll would be 448/441 = 1.02, for the Zogby poll would be 353/ 441 = 0.80, and for the Columbus Dispatch poll would be 93 / 441 = 0.21. Note that this is before applying any penalties for the recentness of a poll. The recentness rating is still applied as described in the FAQ, and multiplied by the Effective Sample Size calculation to produce the final weighting.
This method has one very interesting quality -- namely, that the extent to which we weight one poll relative to another depends on the amount of data we have. When the sample sizes are small, the Sampling Error is large relative to the Pollster-Introduced Error, and therefore there is relatively little difference in the weights assigned to pollsters of varying quality. As sample sizes approach infinity, however, the Sampling Error dwindles toward zero, and nearly the entire difference in the quality of different polls is determined by the Pollster-Introduced Error.
This can be seen in the graph below, where we compare the Effective Sample Sizes of a 'good' pollster (SurveyUSA) and a 'bad' pollster (American Research Group) at various levels of actual sample sizes.
As you can see, the Effective Sample Size for the ARG poll levels off much more quickly than the Effective Sample Size for the SurveyUSA poll. If we have just a little bit of data from each pollster, our primary concern is simply the reducing the effects of Sampling Error, and so we are relatively indifferent about which pollsters we use. However, once we have a lot of data, we have the luxury of being far more discriminating. We'll take nearly as much SurveyUSA data as we can get our hands on, for instance, but for ARG, we are much more cautious, because no matter how much ARG data we have, it is still going to be subject to ARG's methodological deficiencies. In fact, even if we had an infinite amount of ARG data, it would still max out at an Effective Sample Size of 1,113! (SurveyUSA's theoretical maximum, by contrast, goes all the way up to 17,335).
You might wonder why I have the graph extending all the way out to 5,000 respondents when in practice it is very rare to find a poll with more than about 1,200 or 1,500 respondents. The reason is that while we might not find any one poll with a very large number of respondents, we may get effectively the same thing if we have several different polls from the same agency. For example, in the run-up to the Pennsylvania primary, Strategic Vision conducted general election trial heats of 1,200 respondents in each of five consecutive weeks. In many respects, this is similar to one poll of 6,000 respondents. And from the standpoint of our pollster weightings, we treat it largely the same way.
Specifically, when we have more than one poll from a given agency in a given state, we aggregate the sample sizes, and perform the Effective Sample Size calculation on this basis. We then subtract out the Effective Sample Sizes of any more recent surveys from that pollster to determine the Marginal Effective Sample Size, or MESS (yes, I know you are getting sick of all these acronyms). For example, for a set of polls like the Strategic Vision polls, the calculation would work as follows:
Sample Sizes
Date Actual Cumulative Effective MESS
April 19 1200 1200 697 697
April 12 1200 2400 982 285
April 5 1200 3600 1138 156
March 29 1200 4800 1235 97
March 22 1200 6000 1302 67What we are doing here is giving priority to a pollster's most recent poll in a given state. So Strategic Vision's most recent (April 19th) poll gets credit for its full Effective Sample Size, or 697 people. However, because we aggregate the sample sizes, further polling from Strategic Vision is penalized. If we aggregate the two most recent polls from Strategic Vision (April 19 and April 12), we get a combined Effective Sample Size of 987 respondents. But since we already spent 697 of that 982-person budget on the April 19th poll, we have only 285 people left over for the April 12th poll. This is the Marginal Effective Sample Size (MESS) for that poll.The central concept is pretty intuitive, which is that the more and more data we get from a given pollster, the more we encounter diminishing returns.
Also note that in addition to the above calculation, we also punish older polls based on their recentness rating. So effectively we have two different ways to penalize redundant polling from the same agency. But it is a mistake to throw out old polls from the same agency completely; getting SurveyUSA's sloppy seconds may be as good as getting virgin results from a lot of pollsters.
4.29.2008
One benefit of falling poll numbers...
@ 9:45 PM
Rush Limbaugh, citing the recent AP/Ipsos numbers, puts "Operation Chaos" on hold.
I was wondering whether we might see something like this. A very recent Rasmussen poll revealed that Republicans are now almost evenly divided on the question of whether Barack Obama or Hillary Clinton would be the stronger opponent against John McCain.
Democrats, however, still believe Obama to be more electable by a 51-37 margin. On the one hand, I tend to think that looking yourself in the mirror is the hardest thing: if Republicans have concluded that Obama has taken enough hits that he's now essentially equal to Clinton in electability, that might really tell us something. On the other hand, the media, and particularly the right-wing media, has had its sights squarely trained on Obama for about six weeks now. If Limbaugh hadn't suspended Operation Chaos, then he would be admitting, in a sense, to his own impotence in shaping public opinion.
The media narrative, however, appears as though it might shift back toward Barack Obama after his well-received disavowal of Jeremiah Wright today. While predicting shifts in public sentiment is extremely difficult -- predicting shifts in media sentiment tends to be very simple. Obama's remarks today give him a pivot point, one that otherwise might have been hard to come by given the lack of major public events between now and Indiana. I wouldn't quite go so far as to say that this could all turn out to be a net positive for Barack Obama -- but it might be the closest approximation of 'good' news that he was going to be able to get between now and the next primaries. If his campaign can follow this up with a big endorsement within the next 24-48 hours, it could create the sense of real momentum.
I was wondering whether we might see something like this. A very recent Rasmussen poll revealed that Republicans are now almost evenly divided on the question of whether Barack Obama or Hillary Clinton would be the stronger opponent against John McCain.
Democrats, however, still believe Obama to be more electable by a 51-37 margin. On the one hand, I tend to think that looking yourself in the mirror is the hardest thing: if Republicans have concluded that Obama has taken enough hits that he's now essentially equal to Clinton in electability, that might really tell us something. On the other hand, the media, and particularly the right-wing media, has had its sights squarely trained on Obama for about six weeks now. If Limbaugh hadn't suspended Operation Chaos, then he would be admitting, in a sense, to his own impotence in shaping public opinion.
The media narrative, however, appears as though it might shift back toward Barack Obama after his well-received disavowal of Jeremiah Wright today. While predicting shifts in public sentiment is extremely difficult -- predicting shifts in media sentiment tends to be very simple. Obama's remarks today give him a pivot point, one that otherwise might have been hard to come by given the lack of major public events between now and Indiana. I wouldn't quite go so far as to say that this could all turn out to be a net positive for Barack Obama -- but it might be the closest approximation of 'good' news that he was going to be able to get between now and the next primaries. If his campaign can follow this up with a big endorsement within the next 24-48 hours, it could create the sense of real momentum.
Electoral History Charts
@ 7:16 AM
Below are some charts that a few of you will like, and for others of you will be like reading train tables.
What follows is a each state's historical voting record for each Presidential Election since 1948. These charts should be fairly easy to interpret: an "r11" (Republican +11) for instance means that the Republican candidate beat the Democratic candidate by 11 points.
The only exception is when an independent or third-party candidate actually won the state. We indicate these cases with an "i7", meaning that the independent candidate won by a 7-point margin over the second-place finisher.
This next version is actually a little more interesting: this is a state's electoral history relative to the rest of the country. In 1972, for instance, Richard Nixon beat native son George McGovern by 9 points in South Dakota. However, since Nixon beat McGovern by 24 points overall, this was actually a relatively good showing for McGovern: he overperformed his nationwide popular vote margin by 15 points in South Dakota. Thus, this would be indicated in the table as a 'd15' (Democrat +15).
These tables will have a permanent home under the 'History' link in the navigation bar.
What follows is a each state's historical voting record for each Presidential Election since 1948. These charts should be fairly easy to interpret: an "r11" (Republican +11) for instance means that the Republican candidate beat the Democratic candidate by 11 points.
The only exception is when an independent or third-party candidate actually won the state. We indicate these cases with an "i7", meaning that the independent candidate won by a 7-point margin over the second-place finisher.
This next version is actually a little more interesting: this is a state's electoral history relative to the rest of the country. In 1972, for instance, Richard Nixon beat native son George McGovern by 9 points in South Dakota. However, since Nixon beat McGovern by 24 points overall, this was actually a relatively good showing for McGovern: he overperformed his nationwide popular vote margin by 15 points in South Dakota. Thus, this would be indicated in the table as a 'd15' (Democrat +15).
These tables will have a permanent home under the 'History' link in the navigation bar.
4.28.2008
Today's Polls, 4/28
@ 11:11 PM
In Wisconsin, a UW-Madison/Badger Poll has Barack Obama leading John McCain by 4 points, but Hillary Clinton trailing him by 6 points.
It is states like Wisconsin that are responsible for giving Obama at least as strong an electability argument as Hillary Clinton. Forget, for the moment, all that stuff about Obama expanding the map. While the Obama electoral map might be more "fun" than Hillary's, and has the potential to be a little bit more robust in terms of electoral strategy, for the time being Obama is a little too far out of range for states like North Carolina or Montana to be characterized as something other than curiosities. If Obama gains 2-3 points against McCain across the board -- sure, some of those states will become interesting. (For that matter, Hillary Clinton could put states like Kentucky and Tennessee into play if her numbers improve globally). But for the time being, it's probably best to focus on more earthbound sorts of advantages.
The thing is however that Obama does have some significant advantages over Clinton, even without the map-expanders. These are in states that I call the Group of Nine: Wisconsin, Iowa, Minnesota, Michigan, Washington, Oregon, Colorado, Nevada, and New Hampshire. Presently, we have 61 polls listed from among these states. Of those 61 polls, Obama outperforms Clinton in 57 of them, and ties her in two others.
These states, somewhat spread out over the country, otherwise have a fair amount in common. They are, as a group, whiter than the rest of the country, better educated than the rest of the country, more Mainline Protestant than the rest of the country, and more rural than the rest of the country -- some of these characteristics fit with the media-approved caricatures of the race while others do not. They are purple states, but they are not 'centrist' states in the way that, say, Ohio or Missouri are. Instead, they have high numbers of independents, and they tend to weave together different political traditions from the left and the right that somewhat balance one another out. They tend to have active and engaged political bases, but can be somewhat anti-establishment.
The Group of Nine states account for 80 electoral votes; Florida, Ohio and Pennsylvania collectively account for 68. That 12-EV gap, not coincidentally, is very similar to the advantage that Obama presently holds over Clinton in our simulations.
p.s. There's also a new Capital Survey poll out of Alabama that shows McCain leading both Democrats 55-36 (+19). Alabama and Oklahoma are the only two states in the country where both Democrats presently have 0% win percentages.
It is states like Wisconsin that are responsible for giving Obama at least as strong an electability argument as Hillary Clinton. Forget, for the moment, all that stuff about Obama expanding the map. While the Obama electoral map might be more "fun" than Hillary's, and has the potential to be a little bit more robust in terms of electoral strategy, for the time being Obama is a little too far out of range for states like North Carolina or Montana to be characterized as something other than curiosities. If Obama gains 2-3 points against McCain across the board -- sure, some of those states will become interesting. (For that matter, Hillary Clinton could put states like Kentucky and Tennessee into play if her numbers improve globally). But for the time being, it's probably best to focus on more earthbound sorts of advantages.
The thing is however that Obama does have some significant advantages over Clinton, even without the map-expanders. These are in states that I call the Group of Nine: Wisconsin, Iowa, Minnesota, Michigan, Washington, Oregon, Colorado, Nevada, and New Hampshire. Presently, we have 61 polls listed from among these states. Of those 61 polls, Obama outperforms Clinton in 57 of them, and ties her in two others.
These states, somewhat spread out over the country, otherwise have a fair amount in common. They are, as a group, whiter than the rest of the country, better educated than the rest of the country, more Mainline Protestant than the rest of the country, and more rural than the rest of the country -- some of these characteristics fit with the media-approved caricatures of the race while others do not. They are purple states, but they are not 'centrist' states in the way that, say, Ohio or Missouri are. Instead, they have high numbers of independents, and they tend to weave together different political traditions from the left and the right that somewhat balance one another out. They tend to have active and engaged political bases, but can be somewhat anti-establishment.
The Group of Nine states account for 80 electoral votes; Florida, Ohio and Pennsylvania collectively account for 68. That 12-EV gap, not coincidentally, is very similar to the advantage that Obama presently holds over Clinton in our simulations.
p.s. There's also a new Capital Survey poll out of Alabama that shows McCain leading both Democrats 55-36 (+19). Alabama and Oklahoma are the only two states in the country where both Democrats presently have 0% win percentages.
Wine States versus Beer States
@ 6:14 PM
Hmm, that wasn't nearly as cool as I thought it was going to be. This is the states split up 50:50 based on their relative consumption levels of wine and beer, as tracked down from this Matt Yglesias article.
For what it's worth, Obama leads Clinton 13-10 in wine states, and 12-7 in beer states. It should be noted that the entire Midwest is a swing region, though.
Clinton's Bounce
@ 4:16 PM
For the record, she got 2 points in Rasmussen, and 10 points in Gallup, or 6 points on average, by the standards I set up last week to evaluate this stuff. The bounce has since receded slightly to a 4.5-point average.
But what's interesting about numbers in that 4-6 point range is that this is about what she'd need to have an even-steven chance of winning the +Florida popular vote count -- which likely won't win her an argument in Denver but might get her through the doors.
It's harder to tell what's going on at the state level. In Indiana, Clinton either gained or lost points from the previous SurveyUSA poll, depending on what you consider a SurveyUSA poll and what you don't. PPP showed some significant movement to her in North Carolina, but PPP also probably had some work to do on its model after its poor performance in Pennsylvania.
Obama would benefit from a change in the media cycle -- that much I'm pretty certain about.
But what's interesting about numbers in that 4-6 point range is that this is about what she'd need to have an even-steven chance of winning the +Florida popular vote count -- which likely won't win her an argument in Denver but might get her through the doors.
It's harder to tell what's going on at the state level. In Indiana, Clinton either gained or lost points from the previous SurveyUSA poll, depending on what you consider a SurveyUSA poll and what you don't. PPP showed some significant movement to her in North Carolina, but PPP also probably had some work to do on its model after its poor performance in Pennsylvania.
Obama would benefit from a change in the media cycle -- that much I'm pretty certain about.
Call me contrarian, but...
@ 8:53 AM
...by going on a mini publicity blitz that, by the standards of the mainstream media, is so obviously contrary to the best interests of the Obama campaign, is not Reverend Jeremiah Wright effectively distancing himself from Obama?
Pollster Ratings v3.0
@ 7:01 AM
For a couple of weeks now, I've been working from a new version of my pollster ratings, which account in a more sophisticated way for the relationship between the quality of the pollster, and that poll's sample size. The Pennsylvania primary provides as good an excuse as for me to refresh those ratings, as well as to do a more thorough job of explaining them.
And be forewarned ... the explanation that follows is very thorough, and fairly technical in some places. Eventually, I will endeavor to do a quicker-and-dirtier version of this, but I want to have the detail on the record.
I presently have a database of 159 different electoral contests since 2000 that have been surveyed by at least three of the 32 pollsters that I include in my study. This includes contests for President, Senate and Governor, as well as polls from the presidential primaries (almost all of which are from 2008).
The central difference between the approach I took originally and the one I'm taking this time around is that I'm trying to specifically isolate the instance of methodological error, or what I call Pollster-Introduced Error (PIE). If you look at a poll at any given time before an election, there are essentially three different sources of error:
Total Error = Sampling Error + Temporal Error + PIE
Sampling Error is the error that pollsters typically report in their margin of error calculations -- the error intrinsic to sampling only a subset of the entire voting population. All polls have sampling error, though of course a pollster can reduce it by including more interviews in its sample.
Temporal Error is the error introduced by taking a poll weeks or months before an election. Temporal Error is a major consideration now, in April, when we are looking at polls of the November general election; many things can happen between now and then, and (contrary to the common perception) it is not up to the pollster to predict the future. Temporal Error is incorporated into our model in terms of the uncertainty we build into our estimates. For purposes of the pollster ratings, however, we can ignore Temporal Error (that is, assume it to be zero), because we are limiting our evaluation to polls taken very near to the election date.
That leaves us with our final source of error, Pollster-Introduced Error. PIE is what a tennis aficionado might call "Unforced Error"; it is error that results from poor methodology. As a matter of practice, all pollsters have some PIE, which is why the actual margins of error are always larger than those espoused by the pollster. However, this amount varies fairly significantly from agency to agency -- which is the impetus for producing these ratings.
For purposes of our study, PIE is inferred by taking the total error, and subtracting out the Sampling Error. For example, in its recent poll of Pennsylvania, Mason-Dixon predicted Hillary Clinton to beat Barack Obama by 5 points. In actuality, Hillary Clinton beat Barack Obama by 9.1 points. That is a total error of 4.1 points. However, some of this error is not the pollster's "fault", but instead results from its finite sample size. Specifically, Mason-Dixon surveyed 625 people for its poll; on average, a poll of 625 respondents will miss the final margin by 3.2 points because of sampling error alone. Thus, the Pollster-Introduced Error for this poll is 4.1 points less 3.2 points, or 0.9 points. If Mason-Dixon had surveyed more people, we would attribute less of its error to Sampling Error, and more to Pollster-Introduced Error.
(Note: the average error associated with a given sample size is fairly trivial to determine, and can be inferred by means of a binomial distribution. It is approximated by the formula...
80 * n^(-.5)
...where 'n' is the number of respondents in the poll. Technically, the average sampling error varies according to the intrinsic distribution of candidate preferences throughout the population; the sampling error will be somewhat larger when preferences are divided 50:50 than when they are skewed 80:20 toward one candidate. This can be understood intuitively by the fact that when 100 percent of the population prefers a given candidate -- think a Baath Party election -- your poll will return a perfect result regardless of its sample size. This effect is so trivial, however, that it may be ignored for purposes of this analysis).
In this way, we calculate the PIE for every poll in our database. (In some cases, the PIE for an individual poll will be less than zero. While it is impossible for a pollster to have a PIE less than zero over the long run, we retain any sub-zero result for purposes of calculating the pollster's average PIE.) The PIE is then compared, by means of an iteration method, against that of other pollsters which surveyed the same contest, a result that we describe as Iterated Average Error (IAE). This step is functionally equivalent to that which we employed in the previous version of our pollster analysis; as before, we weight the results by the number of pollsters that surveyed a given state.
The PIEs and IAEs for each pollster in our database is indicated below:
The salient number in this table is the +/- rating. This is the extent to which the pollster overperformed or underperformed the other pollsters that surveyed the same races. For example, a SurveyUSA poll has introduced on average 1.13 points of error (PIE) less than that of an average pollster, while a CBS/New York times poll has introduced 0.81 points of error more than average.
In order to translate these numbers into something that we can apply as pollster weightings, we need to take three additional steps. Two of these steps are trivial, while the third is relatively complicated.
The first step is to regress the +/- rating to the mean. This is based on a straightforward calculation of the standard error of the mean. However, note that we do not regress to the mean for two agencies: Zogby Interactive and Columbus Dispatch. This is because these two pollsters use unconventional methodologies -- Internet-based polls and mail polls, respectively, which evidently have resulted in very poor outcomes. There is no reason to give a pollster credit for regression toward the mean when it uses an untested methodology that should intrinsically be associated with larger methodological errors.
The second step is to add back in the average PIE for all pollsters. For example, a SurveyUSA poll might be 1.13 points better than an average poll (or 0.77 points better following regression to the mean), but what is its PIE in absolute terms? The average PIE for all polls in our sample is approximately 1.80 points; however, this includes a disproportionate number of primary election polls, which inherently tend to be associated with larger errors, whereas this site is focused on general election polls. To somewhat reduce these effects, we take the average PIE from among the five election cycles that we have in our database, which works out to 1.49 points:
Adding the global average PIE to the +/- score for our pollster (following regression to the mean) produces its long-run PIE. For example, SurveyUSA's raw +/- score is -1.13, or -0.77 following regression to the mean. Adding 1.49 to this figure produces 0.72. That is, SurveyUSA introduces, on average, 0.72 more points of error than a methodologically perfect poll would (e.g. one having only sampling error, but no pollster-introduced error). This as it happens is an exceptionally strong result -- the best result in our database The long-run PIEs for all 32 pollsters are provided below.
The essential lesson here is not to give deference to 'name-brand' pollsters, or those that are associated with large news organizations. Of our six strongest pollsters, the first two are Internet-friendly operations that use the IVR ('robocall') method, while the next four are boutique or academic shops that limit their polling to a given region. The strongest pollster associated with a large news organization is Mason-Dixon, which conducts polls for The McClatchy Company and MSNBC. However, other major media polling shops, such as FOX / Opinion Dynamics and CBS/New York Times, have considerably underperformed over time. Likewise, some 'brand name' pollsters, like Zogby and Gallup, have had quite poor results.
The Final Step: Translating Ratings into Weightings
However, we are not quite done yet. While these numbers are interesting in the abstract, they do not tell us how to weight polls when we have more than one poll for a given contest. This final step is resolved by figuring what I call the Effective Sample Size.
Suppose that we have a Rasmussen poll with 500 respondents. We have specified Rasmussen's long-run PIE to be 0.88 points, whereas the average sampling error associated with a poll of 500 respondents is 3.58 points. The total expected error for this poll is the sum of thse two figures, or 4.46 points.
Total Expected Error = Sampling Error + Long-Run PIE
4.46 = 3.58 + 0.88
To determine the Effective Sample Size, we must answer the following question: how many respondents would a methodologically perfect have to include for it to have an expected error of 4.46 points? That can be determined by the following equation:
Effective Sample Size = 6400 * (Total Expected Error^-2)
322 = 6400 & (4.46 ^ -2)
In this case, the answer is 322. That is, we would be indifferent between a Rasmussen poll consisting of 500 respondents and a theoretically perfect poll consisting of 322 respondents. Thus, the Effective Sample Size for this poll is 322. Since Rasmussen is a strong pollster, this is actually a fairly good result. By contrast, a Zogby poll consisting of 500 respondents would have an Effective Sample Size of 228, whereas a Columbus Dispatch poll of 500 respondents would have an Effective Sample Size of just 64!
The weight we assign to each poll is directly proportional to the Effective Sample Size. For aesthetic purposes, we represent the weighting as a ratio, taken relative to a poll of 600 respondents by a pollster of average quality (long-run PIE of 1.49). Such a poll would have an Effective Sample Size of 283. Thus, the weighting for the Rasmussen poll would be 322/283 = 1.14, for the Zogby poll would be 228 / 283 = 0.81, and for the Columbus Dispatch poll would be 64 / 283 = 0.23. Note that this is before applying any penalties for the recentness of a poll. The recentness rating is still applied as described in the FAQ, and multiplied by the Effective Sample Size calculation to produce the final weighting.
This method has one very interesting quality -- namely, that the extent to which we weight one poll relative to another depends on the amount of data we have. When the sample sizes are small, the Sampling Error is large relative to the Pollster-Introduced Error, and therefore there is relatively little difference in the weights assigned to pollsters of varying quality. As sample sizes approach infinity, however, the Sampling Error dwindles toward zero, and nearly the entire difference in the quality of different polls is determined by the Pollster-Introduced Error.
This can be seen in the graph below, where we compare the Effective Sample Sizes of a 'good' pollster (SurveyUSA) and a 'bad' pollster (American Research Group) at various levels of actual sample sizes.
As you can see, the Effective Sample Size for the ARG poll levels off much more quickly than the Effective Sample Size for the SurveyUSA poll. If we have just a little bit of data from each pollster, our primary concern is simply the reducing the effects of Sampling Error, and so we are relatively indifferent about which pollsters we use. However, once we have a lot of data, we have the luxury of being far more discriminating. We'll take nearly as much SurveyUSA data as we can get our hands on, for instance, but for ARG, we are much more cautious, because no matter how much ARG data we have, it is still going to be subject to ARG's methodological deficiencies. In fact, even if we had an infinite amount of ARG data, it would still max out at an Effective Sample Size of 2,053! (SurveyUSA's theoretical maximum, by contrast, goes all the way up to 12,499).
You might wonder why I have the graph going all the way out to 5,000 respondents when in practice it is very rare to find a poll with more than about 1,200 or 1,500 respondents. The reason is that while we might not find any one poll with a very large number of respondents, we may get effectively the same thing if we have several different polls from the same agency. For example, in the run-up to the Pennsylvania primary, Strategic Vision conducted general election trial heats of 1,200 respondents in each of five consecutive weeks. In many respects, this is similar to one poll of 6,000 respondents. And from the standpoint of our pollster weightings, we treat it largely the same way.
Specifically, when we have more than one poll from a given agency in a given state, we aggregate the sample sizes, and perform the Effective Sample Size calculation on this basis. We then subtract out the Effective Sample Sizes of any more recent surveys from that pollster to determine the Marginal Effective Sample Size, or MESS (yes, I know you are getting sick of all these acronyms). For example, for a set of polls like the Strategic Vision polls, the calculation would work as follows:
The central concept is pretty intuitive, which is that the more and more data we get from a given pollster, the more we encounter diminishing returns. (Also note that in addition to the above calculation, we also punish older polls based on their recentness rating. So effectively we have two different ways to penalize redundant polling from the same agency).
* * *
I recognize that this is an awful lot to digest, so set this aside and come back later with an aperitif. In the meantime, perhaps we'll try another Q&A session later today.
And be forewarned ... the explanation that follows is very thorough, and fairly technical in some places. Eventually, I will endeavor to do a quicker-and-dirtier version of this, but I want to have the detail on the record.
I presently have a database of 159 different electoral contests since 2000 that have been surveyed by at least three of the 32 pollsters that I include in my study. This includes contests for President, Senate and Governor, as well as polls from the presidential primaries (almost all of which are from 2008).
The central difference between the approach I took originally and the one I'm taking this time around is that I'm trying to specifically isolate the instance of methodological error, or what I call Pollster-Introduced Error (PIE). If you look at a poll at any given time before an election, there are essentially three different sources of error:
Total Error = Sampling Error + Temporal Error + PIE
Sampling Error is the error that pollsters typically report in their margin of error calculations -- the error intrinsic to sampling only a subset of the entire voting population. All polls have sampling error, though of course a pollster can reduce it by including more interviews in its sample.
Temporal Error is the error introduced by taking a poll weeks or months before an election. Temporal Error is a major consideration now, in April, when we are looking at polls of the November general election; many things can happen between now and then, and (contrary to the common perception) it is not up to the pollster to predict the future. Temporal Error is incorporated into our model in terms of the uncertainty we build into our estimates. For purposes of the pollster ratings, however, we can ignore Temporal Error (that is, assume it to be zero), because we are limiting our evaluation to polls taken very near to the election date.
That leaves us with our final source of error, Pollster-Introduced Error. PIE is what a tennis aficionado might call "Unforced Error"; it is error that results from poor methodology. As a matter of practice, all pollsters have some PIE, which is why the actual margins of error are always larger than those espoused by the pollster. However, this amount varies fairly significantly from agency to agency -- which is the impetus for producing these ratings.
For purposes of our study, PIE is inferred by taking the total error, and subtracting out the Sampling Error. For example, in its recent poll of Pennsylvania, Mason-Dixon predicted Hillary Clinton to beat Barack Obama by 5 points. In actuality, Hillary Clinton beat Barack Obama by 9.1 points. That is a total error of 4.1 points. However, some of this error is not the pollster's "fault", but instead results from its finite sample size. Specifically, Mason-Dixon surveyed 625 people for its poll; on average, a poll of 625 respondents will miss the final margin by 3.2 points because of sampling error alone. Thus, the Pollster-Introduced Error for this poll is 4.1 points less 3.2 points, or 0.9 points. If Mason-Dixon had surveyed more people, we would attribute less of its error to Sampling Error, and more to Pollster-Introduced Error.
(Note: the average error associated with a given sample size is fairly trivial to determine, and can be inferred by means of a binomial distribution. It is approximated by the formula...
80 * n^(-.5)
...where 'n' is the number of respondents in the poll. Technically, the average sampling error varies according to the intrinsic distribution of candidate preferences throughout the population; the sampling error will be somewhat larger when preferences are divided 50:50 than when they are skewed 80:20 toward one candidate. This can be understood intuitively by the fact that when 100 percent of the population prefers a given candidate -- think a Baath Party election -- your poll will return a perfect result regardless of its sample size. This effect is so trivial, however, that it may be ignored for purposes of this analysis).
In this way, we calculate the PIE for every poll in our database. (In some cases, the PIE for an individual poll will be less than zero. While it is impossible for a pollster to have a PIE less than zero over the long run, we retain any sub-zero result for purposes of calculating the pollster's average PIE.) The PIE is then compared, by means of an iteration method, against that of other pollsters which surveyed the same contest, a result that we describe as Iterated Average Error (IAE). This step is functionally equivalent to that which we employed in the previous version of our pollster analysis; as before, we weight the results by the number of pollsters that surveyed a given state.
The PIEs and IAEs for each pollster in our database is indicated below:
The salient number in this table is the +/- rating. This is the extent to which the pollster overperformed or underperformed the other pollsters that surveyed the same races. For example, a SurveyUSA poll has introduced on average 1.13 points of error (PIE) less than that of an average pollster, while a CBS/New York times poll has introduced 0.81 points of error more than average.
In order to translate these numbers into something that we can apply as pollster weightings, we need to take three additional steps. Two of these steps are trivial, while the third is relatively complicated.
The first step is to regress the +/- rating to the mean. This is based on a straightforward calculation of the standard error of the mean. However, note that we do not regress to the mean for two agencies: Zogby Interactive and Columbus Dispatch. This is because these two pollsters use unconventional methodologies -- Internet-based polls and mail polls, respectively, which evidently have resulted in very poor outcomes. There is no reason to give a pollster credit for regression toward the mean when it uses an untested methodology that should intrinsically be associated with larger methodological errors.
The second step is to add back in the average PIE for all pollsters. For example, a SurveyUSA poll might be 1.13 points better than an average poll (or 0.77 points better following regression to the mean), but what is its PIE in absolute terms? The average PIE for all polls in our sample is approximately 1.80 points; however, this includes a disproportionate number of primary election polls, which inherently tend to be associated with larger errors, whereas this site is focused on general election polls. To somewhat reduce these effects, we take the average PIE from among the five election cycles that we have in our database, which works out to 1.49 points:
Cycle PIE
2000 - General 1.35
2002 - General 1.24
2004 - General 0.30
2006 - General 0.64
2008 - Primaries 3.90
Global Average 1.49You should note that we still give some weight to the primary election polls from 2008 in calculating our average PIE. This is because the polls in this election cycle have been much, much less accurate than polls in previous (general election) cycles. While my suspicion is that this has mostly to do with the nature of polling primaries, rather than the nature of the 2008 electoral cycle, I cannot say this for certain because I have very little data from previous years' primaries to look at. Therefore, we hedge our bets a little bit (this should be considered a conservative assumption).Adding the global average PIE to the +/- score for our pollster (following regression to the mean) produces its long-run PIE. For example, SurveyUSA's raw +/- score is -1.13, or -0.77 following regression to the mean. Adding 1.49 to this figure produces 0.72. That is, SurveyUSA introduces, on average, 0.72 more points of error than a methodologically perfect poll would (e.g. one having only sampling error, but no pollster-introduced error). This as it happens is an exceptionally strong result -- the best result in our database The long-run PIEs for all 32 pollsters are provided below.
The essential lesson here is not to give deference to 'name-brand' pollsters, or those that are associated with large news organizations. Of our six strongest pollsters, the first two are Internet-friendly operations that use the IVR ('robocall') method, while the next four are boutique or academic shops that limit their polling to a given region. The strongest pollster associated with a large news organization is Mason-Dixon, which conducts polls for The McClatchy Company and MSNBC. However, other major media polling shops, such as FOX / Opinion Dynamics and CBS/New York Times, have considerably underperformed over time. Likewise, some 'brand name' pollsters, like Zogby and Gallup, have had quite poor results.
The Final Step: Translating Ratings into Weightings
However, we are not quite done yet. While these numbers are interesting in the abstract, they do not tell us how to weight polls when we have more than one poll for a given contest. This final step is resolved by figuring what I call the Effective Sample Size.
Suppose that we have a Rasmussen poll with 500 respondents. We have specified Rasmussen's long-run PIE to be 0.88 points, whereas the average sampling error associated with a poll of 500 respondents is 3.58 points. The total expected error for this poll is the sum of thse two figures, or 4.46 points.
Total Expected Error = Sampling Error + Long-Run PIE
4.46 = 3.58 + 0.88
To determine the Effective Sample Size, we must answer the following question: how many respondents would a methodologically perfect have to include for it to have an expected error of 4.46 points? That can be determined by the following equation:
Effective Sample Size = 6400 * (Total Expected Error^-2)
322 = 6400 & (4.46 ^ -2)
In this case, the answer is 322. That is, we would be indifferent between a Rasmussen poll consisting of 500 respondents and a theoretically perfect poll consisting of 322 respondents. Thus, the Effective Sample Size for this poll is 322. Since Rasmussen is a strong pollster, this is actually a fairly good result. By contrast, a Zogby poll consisting of 500 respondents would have an Effective Sample Size of 228, whereas a Columbus Dispatch poll of 500 respondents would have an Effective Sample Size of just 64!
The weight we assign to each poll is directly proportional to the Effective Sample Size. For aesthetic purposes, we represent the weighting as a ratio, taken relative to a poll of 600 respondents by a pollster of average quality (long-run PIE of 1.49). Such a poll would have an Effective Sample Size of 283. Thus, the weighting for the Rasmussen poll would be 322/283 = 1.14, for the Zogby poll would be 228 / 283 = 0.81, and for the Columbus Dispatch poll would be 64 / 283 = 0.23. Note that this is before applying any penalties for the recentness of a poll. The recentness rating is still applied as described in the FAQ, and multiplied by the Effective Sample Size calculation to produce the final weighting.
This method has one very interesting quality -- namely, that the extent to which we weight one poll relative to another depends on the amount of data we have. When the sample sizes are small, the Sampling Error is large relative to the Pollster-Introduced Error, and therefore there is relatively little difference in the weights assigned to pollsters of varying quality. As sample sizes approach infinity, however, the Sampling Error dwindles toward zero, and nearly the entire difference in the quality of different polls is determined by the Pollster-Introduced Error.
This can be seen in the graph below, where we compare the Effective Sample Sizes of a 'good' pollster (SurveyUSA) and a 'bad' pollster (American Research Group) at various levels of actual sample sizes.
As you can see, the Effective Sample Size for the ARG poll levels off much more quickly than the Effective Sample Size for the SurveyUSA poll. If we have just a little bit of data from each pollster, our primary concern is simply the reducing the effects of Sampling Error, and so we are relatively indifferent about which pollsters we use. However, once we have a lot of data, we have the luxury of being far more discriminating. We'll take nearly as much SurveyUSA data as we can get our hands on, for instance, but for ARG, we are much more cautious, because no matter how much ARG data we have, it is still going to be subject to ARG's methodological deficiencies. In fact, even if we had an infinite amount of ARG data, it would still max out at an Effective Sample Size of 2,053! (SurveyUSA's theoretical maximum, by contrast, goes all the way up to 12,499).
You might wonder why I have the graph going all the way out to 5,000 respondents when in practice it is very rare to find a poll with more than about 1,200 or 1,500 respondents. The reason is that while we might not find any one poll with a very large number of respondents, we may get effectively the same thing if we have several different polls from the same agency. For example, in the run-up to the Pennsylvania primary, Strategic Vision conducted general election trial heats of 1,200 respondents in each of five consecutive weeks. In many respects, this is similar to one poll of 6,000 respondents. And from the standpoint of our pollster weightings, we treat it largely the same way.
Specifically, when we have more than one poll from a given agency in a given state, we aggregate the sample sizes, and perform the Effective Sample Size calculation on this basis. We then subtract out the Effective Sample Sizes of any more recent surveys from that pollster to determine the Marginal Effective Sample Size, or MESS (yes, I know you are getting sick of all these acronyms). For example, for a set of polls like the Strategic Vision polls, the calculation would work as follows:
Sample Sizes
Date Actual Cumulative Effective MESS
April 19 1200 1200 452 452
April 12 1200 2400 672 220
April 5 1200 3600 824 150
March 29 1200 4800 941 117
March 22 1200 6000 1035 94What we are doing here is giving priority to a pollster's most recent poll in a given state. So Strategic Vision's most recent (April 19th) poll gets credit for its full Effective Sample Size, or 452 people. However, because we aggregate the sample sizes, further polling from Strategic Vision is penalized. If we aggregate the two most recent polls from Strategic Vision (April 19 and April 12), we get a combined Effective Sample Size of 672 respondents. But since we already spent 452 of that 672-person budget on the April 19th poll, we have only 220 people left over for the April 12th poll. This is the Marginal Effective Sample Size (MESS) for that poll. The central concept is pretty intuitive, which is that the more and more data we get from a given pollster, the more we encounter diminishing returns. (Also note that in addition to the above calculation, we also punish older polls based on their recentness rating. So effectively we have two different ways to penalize redundant polling from the same agency).
* * *
I recognize that this is an awful lot to digest, so set this aside and come back later with an aperitif. In the meantime, perhaps we'll try another Q&A session later today.
4.27.2008
The Road Veers Right
@ 3:14 PM
Why did Barack Obama appear on Fox News Sunday this morning? For the same reason that Hillary Clinton (see video below) is campaigning on a proposal originally introduced by John McCain: a summer holiday from the federal gas tax.
The reason is that the remaining primary calendar veers decidedly right of center. Below is a graph of the 50 states, sorted by the percentage of the vote that George W. Bush received in 2004. The states that have yet to vote are highlighted in red.
With the exception of Oregon, which is more conservative than is generally acknowledged, the other six states all fall within a relatively narrow band toward the right-center of the political spectrum. And four of those six states (Kentucky and South Dakota are the exceptions) have open primaries.
So have the Democrats begun to pander to the right? It depends on your definition of pander, I suppose. But on balance, it is probably helpful to them to have these right-center states on the docket as we transition gradually into the general election cycle.
It does seem to me, however, that their respective actions are illustrative of the different approaches they might take toward courting the political center in a general election campaign. Obama will have a conversation with a right-of-center news outlet, which asks a series of questions framed from a right-of-center point of view, but will not actually espouse any particularly right-of-center policy proposals. Clinton, on the other hand, will take on a position that taken criticism from both the left-wing and the right-wing establishments, but which probably plays relatively well on the stump.
The reason is that the remaining primary calendar veers decidedly right of center. Below is a graph of the 50 states, sorted by the percentage of the vote that George W. Bush received in 2004. The states that have yet to vote are highlighted in red.
With the exception of Oregon, which is more conservative than is generally acknowledged, the other six states all fall within a relatively narrow band toward the right-center of the political spectrum. And four of those six states (Kentucky and South Dakota are the exceptions) have open primaries.
So have the Democrats begun to pander to the right? It depends on your definition of pander, I suppose. But on balance, it is probably helpful to them to have these right-center states on the docket as we transition gradually into the general election cycle.
It does seem to me, however, that their respective actions are illustrative of the different approaches they might take toward courting the political center in a general election campaign. Obama will have a conversation with a right-of-center news outlet, which asks a series of questions framed from a right-of-center point of view, but will not actually espouse any particularly right-of-center policy proposals. Clinton, on the other hand, will take on a position that taken criticism from both the left-wing and the right-wing establishments, but which probably plays relatively well on the stump.
Subscribe to:
Posts (Atom)
