Real Credibility Problems

EDIT: Please see also my follow-up post here.

Last week, I discussed Real Clear Politics' decision to exclude the Research 2000 daily tracking poll from their national averages. After a lengthy discussion with RCP founder John McIntyre, I decided to defend them, pointing out that while RCP may have a debatable framework for deciding which polls they do or do not include, they at least appeared to have applied this framework consistently.

My gut-level takeaway from my conversation with McIntyre was somewhat different from what I represented, however. My gut-level feeling was that RCP was in fact cherry-picking those results that were to its liking, and then coming up with post-facto rationalizations to justify its decisions.

RCP made a decision yesterday that convinces me I should have trusted my gut. Specifically, it was their decision to re-include polls from American Research Group (ARG) in their state-by-state averages.

ARG had been effectively "banned" from RCP for several months now, going back to the Democratic primaries. Sometimes RCP listed ARG polls with an asterisk and did not include them in their averages; more often they didn't list them at all.

Why didn't RCP include ARG's polls in its averages? Presumably because they had an exceptionally erratic performance during this year's primaries, a performance that -- when coupled with ARG's relative lack of disclosure about its methodology -- had led RCP to conclude that their polls were not credible.

As you may know, I am not much of a fan of ARG myself. However, while ARG ranks toward the back end of our pollster ratings, so do a lot of other pollsters that RCP has no problems with.

What I do know, however, is that whatever RCP's reasons were for excluding ARG from its averages, none of those reasons should have changed over the past 48 hours. Did ARG suddenly improve its level of disclosure? No. Did their polls suddenly become more accurate? How could they, when we haven't had any elections to evaluate them.

No. The only thing that changed is that ARG released a set of three polls yesterday that had considerably more favorable numbers for John McCain than other contemporaneous polls of those states. In each of those states -- North Carolina, Virginia, Nevada -- ARG is presently the outlier on the John McCain side (tied with Mason-Dixon in the case of Virginia). RCP did not feel any compulsion to include ARG's numbers when ARG cycled through all 50 states a couple of weeks ago (including many where there has been very little polling). Only when ARG released this gravity-defying set of polling in Virginia and North Carolina and Nevada did RCP suddenly have a change of heart.

This is not RCP's sole problem with consistency. Take a look, for example, at which polls they presently include in their averages for Virginia:



You'll see three polls -- SurveyUSA, Mason-Dixon, and ABC/Post -- that were conducted at essentially the same time. The ABC/Post poll was in the field from the 18th to the 21st, Mason-Dixon from the 17th to the 22nd, and SurveyUSA from the 19th to the 21st. And yet, the Mason-Dixon and ABC/Post polls are included in its average whereas SurveyUSA is not. Guess which one had the strongest numbers for Obama?

A similar example from Minnesota. The Star-Tribune conducted polling from September 10th through September 12th; SurveyUSA conducted polling from September 10th to September 11th. The Star-Tribune poll, which shows a tie, is included; the SurveyUSA poll, which showed Obama ahead by 2, is not.

As long as we're having fun with this, how about Alaska? Anchorage-based pollster Ivan Moore had released a poll in July, then showing just a 3-point race between McCain and Obama. We included it, Pollster.com included it, and RCP did not. But then last month, when the same firm released a fresh poll showing the profound effect that Sarah Palin had on the race (McCain +19), RCP decided they were a worthy pollster.

Or how about North Carolina Senate? RCP presently includes two weeks-old polls from SurveyUSA and Research 2000, whch had shown Elizabeth Dole with a solid lead in her race against Kay Hagan. But they didn't bother to include more recent polls from Elon and Civitas, which show the race essentially tied.

Why is one of Mitch McConnell's internal polls included in their Kentucky Senate averages?

Why, a week or so ago, did RCP include a poll from the Marketing Resource Group of Lansing, Michigan, which the Detroit News describes as a "Republican consulting firm", while excluding a Nevada poll that came out on the same day from Project New West, a Democratic strategy firm? Might it have had anything to do with the fact that the Michigan poll showed McCain ahead -- and the Nevada poll showed him behind?

Look -- I'm not going to tell you that my site is completely devoid of spin. I am a Democrat, and I see the world through a Democratic lens. But what I can promise you is that we'll keep the spin separate from our metrics. The spin is a side dish, which you can choose to consume or ignore.

Unfortunately, that is not a choice you have at RCP. Their partisan leaning is infused into their numbers. If RCP disclosed their methodology -- articulated their rationale for excluding or including certain polls -- I would give them the benefit of the doubt. But they do not, so I do not.

You do, however, have a number of good alternatives in the marketplace. So here is a recipe for RCP detox.

For polling averages, you can of course look here. But otherwise, I would recommend Pollster.com. Pollster.com has the most unbiased standard imaginable: they simply include every poll.

For content aggregation, I would recommend Memeorandum or PoliticsHome, each of whom refresh their material on a 24/7 basis.

We're not asking that you go cold turkey. Do bookmark and continue to read Jay Cost at RCP, who is an outstanding analyst.

Full disclosure: RCP is a competitor, so I have every incentive to tell you this. (I'm pretty sure they see me as a competitor too, as I don't think they've linked to me in about a month). But the fact is that there's not an exceptionally high degree of difficulty in simply collecting a bunch of polls and averaging them together. There's no excuse not to do it right.