Following are our updated pollster ratings, accounting for the Democratic contests in North Carolina, Indiana, West Virgina, Kentucky and Oregon. We've also updated to include the final, state-certified voting margin in a couple of states like Ohio where the numbers changed from the unofficial estimates. Apart from refreshing this data, the methodology is entirely identical to Version 3.1.
Moving up: Zogby (who did much better than the consensus in Indiana and North Carolina) and Public Policy Polling (who made up for their poor result in Pennsylvania with a spot-on call in Oregon).
Most of the hit appears to have been absorbed by SurveyUSA, which by no means performed badly in this cycle, but had been so far ahead of the curve that even an average performance drags their numbers downward somewhat. The mathematics of the ratings calculation are also such that the ratings tend to be especially sensitive for those pollsters toward the top of the chart.
Even the pollsters who did not poll in this cycle may have seen their ratings affected slightly, as everything is taken relative to the performance of other pollsters. For example, since our opinion of Zogby improved in this version of the ratings, a pollster now gets more credit for beating Zogby in a particular state than it had gotten before.
The new version of the ratings will be incorporated beginning with our refresh of the polling data tomorrow morning. As I doubt that we'll get much polling data in Montana and South Dakota, this may well be the final version of the ratings until the general election takes place.
Wednesday, May 28, 2008
Pollster Ratings, v3.1.1
-- Nate Silver at 12:35 AM
Labels: methodology, pollster ratings, pollsters, site
32 comments
Poblano -
I can tell you love a statistical challenge, so here's one: how large a difference in PIE is required to reach statistical signifigance? The top-ranked appears much better than the bottom-ranked, but how about #7 vs. #12?
This is a great looking set of numbers, but without MOE or confidence intervals (or what-not) how can we know if a difference in PIE is meaningful?
To answer the above:
It might be possible to construct a confidence interval(Though I'd bet it would be difficult, I don't see any obvious pivotal qualities)
But since the purpose of this site is forecasting, not actual pollster ratings, it makes sense to use the data given, regardless of significance.
To give an example, suppose you are given a weighted coin. You flip it twice, and it comes up once as heads and once as tails.
Using the data available, it is best to assume that the probability of heads is 1/2, even if that find is not statistically significant.
This site is now the first I check when it comes to politics...better than Kos, RCP, etc because it is interesting and on point. How about some more posts/posters to make it even better? Your political analysis is as interesting as the #'s, and more of that would lead to even greater viewership. Take it to the next level.
So it’s Ohio all over again?
At least according to your most recent analysis.
Only 10 states appear to be even marginally competitive – NH, VA, PA, OH, MI, IN, WI, CO, NM and NV. If you cede Obama the states he leads in – PA, MI, WI, CO and NM, and McCain the states he leads – NH, IN, NV, you get Obama 265, McCain 253. Ohio, which you have exactly tied, casts the deciding vote.
2008 looks like 2004.
Anon #2,
I'm not following your math. If Obama and McCain win the states you outline (I assume you give McCain Virginia), I get Obama 269, McCain 249. Even if McCain takes Ohio, they tie at 269 EVs, and Obama wins the election in the House of Representatives.
Poblano: Judging by the accuracy attained by SUSA, what do you make of their recent VEEP polling series which seems to across the board register an above average Democratic Party ID, and a below average number of African American Voters?
Strangly these two don't cancel each other out and it's probably the strongest series of polls that Obama has had all cycle. I buy into the thesis that it's because it's one of the first Non-Hillary General Election Polling Cycles, and without her even in the questioning, it's getting him better results somehow. How that is remains anyone's guess.
Omar: Stick around, you'll find that often the commentors are as good as Poblano. Such solid strong analysis attacks a really good cadre of readers... This site is awesome as is.
If I remember correctly your pollster rating is based purely on last-minute polls - please correct me if I'm wrong.
If so, we have an interesting problem. Could it be that some pollsters have methodologies that are good at getting the results at the very last minute, but are less good long range, and vice versa? This is not fanciful: the choice of questions may lead to more sensitivity to the day-to-day noise of shifting moods, or you may push respondents more or less, all of which having an effect on your accuracy at long range.
I think many people have voiced the intuition in particular that SUSA is a good last-minute pollster but a weak long-range pollster.
This is obviously important for your project because (a) you very wisely use old polls as part of your model; (b) for the next 155+ days or so, we are only going to have long-range polls.
Poblano: Why are there no Gallup polls in your state poll list over there at the right?
In its most recent "swing state analysis" (reported on Pollster.com), are those results just an allocation to each state from their national tracking polls -- rather than separate states? What are the N's per state? What are the trends?
To add to the above comment, since your state poll analysis does not include Gallup, it seems to me you could do a direct replication of Gallup's "Swing State Analysis" (reported on Pollster), but using non-Gallup data.
Further, if you backed up in time using your database you could show trends in the swing states as of March 15, April 15, May 15, and today. A lot of work but it would give some depth to trends, whereas Gallup does not show trends at all because of their method of pooling data over 3-week period in May only.
doktarr - I'm using the map found at http://www.270towin.com/
I get 265-253. I'm giving NH to McCain right now (and yes, VA). If you give NH to Obama it is 269-249 w/o Ohio, but McCain seems to lead in the NH polls.
I don't know what other 4 EV state it could be.
Poblano, are you planning to correct old poll numbers based on national movement since then? It is a way to use the national poll numbers - though you'd have to smooth out the minor variations with a broad enough average. You could be conservative in your correction - multiply it by 0.9, or in some way subtract out a MOE.
If you are going to do this, you should definitely have the system in place before Clinton drops out, because that's just the start of the serious national movement.
Please respond, I think this idea is at least worth consideration, and I'd be happy to clarify how I see it working. Your model is still giving some weight to polls from February, and that will only get worse as the general comes on and pollsters shift some focus away from the states.
(if you wanted to be more complex about it, you could do your national correction by party-id subgroups. That way, as the lambs return to the fold, your correction would not just make the whole map bluer - some red states could get redder at the same time).
Here's my suggestion for using national polls:
1. Treat them as polls of a state, USA, without electoral college votes (and with the demographics of the USA).
2. Use them as another target for training the 538 regression.
3. Adjust higher the weight of the 538 regression.
Same anon:
Check Hawaii and Rhode Island. I'm quite sure the scenario you suggest is a tie - I get a tie using 270towin. In fact, it's the most common tie in Poblano's tie analysis, happening a whole .04% of the time last he checked.
I'm new to this site and I do not understand some of the numbers. For example:
- The Swing State Analysis for Ohio shows Obama vs McCain as 15.8% in BLUE and Clinton vs McCain as 8.1% in RED. What the heck does this mean?
- The Win Percentage Tracker shows what? The percentage of states a given candidate would win if the general election were held on a given day?
- The Regional Analysis tables shows what? The statistical probability that a given candidate will carry the state?
I would appreciate SOME simple explanations.
Thanks.
I posted something along these lines on DailyKos, but I thought you might be the perfect person to run the analysis.
I have a hypothesis that Clinton is actually doing a little better than Obama amongst African-American voters, while Obama may be performing a little bit better with working class whites and older women than exit polling suggests. The exit polls have been unusually inaccurate in this cycle, and I believe that pollster bias may be affecting the answers they are receiving (for example, an African-American voter may be less inclined to tell an African-American pollster that they supported Clinton).
I'm wondering if there is a way to prove this by looking at the cross-tabs of the automated polls and comparing those the cross-tabs of the exit polls and the cross-tabs of the non-automated polls.
This might be useful in an attempt to filter out the biases in future non-automated polls.
How many polls has Seltzer conducted? SUSA polls every contest.
How many polls has Seltzer conducted? SUSA polls every contest.
What's with the enormous error in SurveyUSA and yet they rank second amongst pollsters? They are all over the place. Obama is gonna win North Dakota and Indiana yet barely win Massachussetts and tie in New Jersey? Meanwhile, McCain barely wins North Dakota, South Carolina and Nebraska, gets blown away in Ohio and only wins by 11 in Utah (where Bush won by about 40).
I smell some serious bias here, or just incompetency.
Does your ranking system account for regression toward t