I just concluded a long, and understandably fairly contentious phone conversation with John McIntyre of Real Clear Politics.
John strongly disputes the notion that he is cherry-picking polls to achieve a particular partisan result (pro-GOP or otherwise).
It is clear to me that there is substantial subjectivity in how RCP selects the polls to include in its averages. RCP does not publish an FAQ, or any other set of standards. Nor, in my conversation with John, was he willing to articulate one. In my view, the fact that RCP does not disclose a set of standards means ipso facto that they are making judgment calls -- that there is some subjectivity involved -- in how their polls are selected.
However, it does not necessarily follow that these judgment calls reflect any deliberate partisan leaning, i.e. any "bias". That is a much stronger accusation, and it is the one that John objects to. I take John at his word that this is not the case. I hope to have a spirited rivalry with RCP, but my post this morning may have made things too personal.
At the same time, I believe that RCP leaves themselves open to this type of accusation until and unless they improve their level of disclosure. I have strongly encouraged John to publish an FAQ or other set of standards on his website.
10.02.2008
RCP Follow-Up
@ 1:27 PM
Subscribe to:
Post Comments (Atom)
110 comments
Maybe you can have a televised debate?
methinks RCP doth protest too much
Ok, so he's not biased, but he can't prove it or be bothered to explain why certain polls are used only when they are pro McCain?
Actions speak louder than words, McIntyre.
I think your post from this morning was very well done, and you're points are valid.
I also really appreciate the way your present your stats, excellent number work that is then put in real people speak.
So he's not a blatant partisan hack, he's just random and has no actual methodology.
That clears things up.
So, he gave no explanation for the recent reversal? None at all?
OPEN.
THAT.
CAN.
OF.
WHOOP.
ASS.
Whatta dickbag.
Nate, you cannot accuse anyone of bias.
When Obama was trailing you dug your heels in and began using PPP (a dem pollster) and weighing it more than the more historically accurate ones.
You admitted in the beginning you were pro-Bama and your bias is obvious.
I've refrained from posting for weeks because of this.
McCain was a POW for years, and that absolves him of any imputation of partisan behavior. He thus doesn't have to disclose anything.
Why, they were lucky to have fingers to count on in his North Vietnamese hellhole, much less fancy polls on the internets.
-- Oh, you said "McIntyre." Well, same difference.
An Internet web site rivalry is not a zero sum game...There's really no need to continue to push this. Nate called out RCP; now it's RCP's turn to give a reasonable explanation and publish a set of standards to avoid nonobjective decisions in the future.
Seems to me that he's protesting the fact you pointed it out, not that it was true...
I still say I have observed them moving results from one day to the next (or previous) to help McCain look better on any given day. Just what I have seen. If they don't like that accusation, then let them leave the polls where they first place them!
I don't have in mind that you have any intentional or unintentional bias in your statistical estimations. But in the name of disclosure, it would be interesting to see more of what is in your simulation model. It is after all the source of the most prominent statistic on the whole site, the Obama win percentage.
Nate - who cares what he says. This is your site and your free to say whatever the h$*^ you want.
I think everything you said earlier was peachy - you stated your opinion and backed it up with valid details. It's certainly obvious they don't care for putting in the work to publish sound stat's, they're just free-wheeling it.
And who cares what anyone says - you're site is popular because you have in place sound statistical methodologies.
A Republican that shrouds their methods in secrecy and who attacks the messenger when those methods are questioned?
NEVER!
(/sarcasm)
party sheep to party sheep brawl. typical.
each party sheep comment you make, brings your credibility down.
stooping the the level the candidates are at, fantastic. more unhelpful garbage.
Nate has been very open about how and why he uses what Polls.
According to the Pollster Ratins, PPP has a slightly below average Pollster Introduced error rating.
Yes, Nate's a Democrat. That does not mean that his statistical analysis should be judged flawed by association. That he has a FAQ and clearly details how he sets his model puts him on a reliability standard above that of most, if not all, online election polling and prediction sites.
They have a budget for bus ads. You've transparency and the truth.
Keep up the good fight.
TJB, he also uses Strategic Vision, which is a Republican pollster and both fall within his given criteria, and I believe he has done so consistently since the primaries, so your implied accusation of bias is unfounded.
Objectively, and using math as the only means to determine the answer, Nate's results are more accurate than RCP's.
It would be nice and sweet for them to open up their methodology, but secrets in Washington are worth money or power. I'd be shocked if they did such a thing. They have a nice little niche going in GOP circles. To open it up now could damage their other business relationships.
In other words, McIntyre said that it only looks as if he is biased, but he really isn't.
That is weak.
Obviously, short of telepathy or some kind of voluntary admission, it is impossible to prove that someone maintains a bias.
This is a defence that could be employed by the most bigoted individuals. "Sure, I called him a n*****, but I'm not prejudiced. My actions appear to indicate bias, but you can't prove that there is bias in my thoughts. So therefore I resent your implication. You may apologise now."
Again, profoundly weak.
tjb - did you not read Nate's post? He very clearly admitted his partisanship, but also stated that he expresses it only in his commentary. To Nate, as to any good datamonkey, numbers are numbers and ideas are ideas and never the twain shall meet. It's called reality. Welcome.
Conservatives are, by nature, incapable of understanding that scientists are capable of putting aside their bias in order to produce realistic results. Conservatives can't comprehend that true scientists are always trying to prove themselves WRONG, in order to learn more about the subject at hand.
Conservatives just don't understand, nor should they be expected to understand. In the end, they're just conservatives after all. There's no need to expect any more from them.
I figured your post from earlier today would generate a telephone call. Be not dismayed. Sounds like your hunches were accurate. You operate on a platform of FULL DISCLOSURE. Unless RCP does likewise, they are irrelevant. Thank you.
Don't let the bully push you around.
You didn't trust your gut the first time and you regretted it. I can tell from the tone of this post that you still don't buy what RCP is selling but you feel compelled to issue a qualified mea culpa, since you probably just got berated pretty good.
Trust your gut, and don't apologize for it.
Keep it real Nate! You're only making everyone else around you better.
Worrisome CO poll from Ciruli (sp?) shows A LOT of Udall voters crossing over for McCain...
Obama 44%
McCain 43%
The MSM, vis-a-vis RCP, wants to make the race appear as tight as possible. A tight race = more advertising dollars.
Reality has a liberal bias.
Nate,
I can't laud you strongly enough for these two posts. One of the things that makes fivethirtyeight so great is that you and Sean are offering transparency about your numeric methods, your political positions, and how you try to keep the two separate.
RCP is frequently cited in the media as an authoritative source on numbers. If there's bias there -- intended or not -- it colors public perception. And this is politics -- perception counts.
I am only one of a few who has wondered if there's something fishy at RCP (and probably less than and more recently than many around here). You've done a great service by explaining what's up. If John McIntyre is willing to be more transparent, that's great. If he's not, I guess the ad revenue at pollster.com will go up.
Finally, I haven't thought of this site and RCP as competitors. I, probably like others, had thought of them as a source for raw data, which is clearly naive. It's eye-opening to realize that there's more to it than that.
Anyone who does political research and/or reportage has bias. The degree to which bias could affect the research debatable. In order to mitigate effects of bias, good researchers completely disclose their methodologies. Nick Silver has fully disclosed his political bias, his methodology and where appropriate (and perhaps where unclear), he has provided complete explanations for his methodological choices. RCP is not being transparent. That does not make RCP wrong or biased, it should make consumers suspicious of their data and analysis.
Hmmm? He won't post a FAQ or identify his methodology. Sounds shady to me!
Why is it that Republitards always seem to have something to hide? Not very 'American' if you ask me.
I'm sure that the RCP people aren't intentionally biasing the results in the republican direction. They're doing their best. But Nate makes a great point -- if you have a set method that you don't change around and you publish exactly what you're doing, then you're approaching the matter in a scientific and unbiased manner. If you rather make a judgment call each time a poll is published, then you open yourself up to making systematic mistakes. It doesn't matter if you're doing it on purpose, you'll still tend to make judgments that favor what you want if you have a partisan lean. It's why scientists use the scientific method. Even if they WANT to be fair and honest all of the time, they still come into their experiments with hypotheses and they'll take actions that unfairly support these hypotheses if they don't carefully use the scientific method. Nate and Sean are VERY obviously democratic and their commentary smacks quite heavily of this. But you can just never read the blog and look at the numbers only. There may be some decisions regarding how to count the polls that have overall helped Obama, but they made these decisions far before knowing what they'd resolve in, and they've stayed consistent. So the numbers are not biased on this site. Only the blog part. Again, as for RCP, if there is not a solid set-in-stone method, the judgment calls will tend to subconsciously favor what the author wants to be true. This is a fact of psychology.
Finally some good news for McCain. Mason-Dixon still has him up 3 in VA. (48-45)
Getting back to RCP, it's a FACT that they included the ARG polls a day after they came out. Not when they first came out - 2 days ago, but yesterday when the deluge of pro-Obama polls came out.
Sorry for the off-topic post, but I found this alarming:
DJIA on 1/20/2001: 10,587.60
DJIA on 10/2/2008: 10,571.18 (as of 1:48pm ET)
This has Colbert-O'Brien Rivalry written all over it.
RCP isn't partisan; they're biased. They were founded by a couple of guys who are free market aficianados. It wouldn't be terribly surprising if they adjusted their take on the polls accordingly. Their own editorials also reflect that philosophy, but in their defense, the guest commentaries span the range from left to right.
You know if the data is still available you could prove one way or the other if RCP has a bias (intentional or not, conscious or unconscious). Nate presented evidence that supports his premise of bias. It is difficult to prove intention. Is there contrary data to support RCP's claim?
I would wonder why RCP has included the Democracy Corps national poll from yesterday, showing Obama up by 4, but not the Democracy Corps Ohio poll from today, showing Obama up by 6 there.
Acta non verba, Mr. McIntyre.
The ball's in their court. They can open up about their methodology, or they can lose credibility.
November 4th will tell the tale, one way or the other. No finessing numbers then.
Via Politico - McCain abandons Michigan.
The road to 270 just got a hell of a lot harder for Johnny Drama.
ha ha, shap! well done. yes, i'm thinking we'll survive..
McCain: Obama lead growing because 'life isn’t fair'
http://politicalticker.blogs.cnn.com/2008/10/02/mccain-obama-lead-growing-because-%e2%80%9clife-isn%e2%80%99t-fair%e2%80%9d/
McCain Campaign ~ Best Campaign Ever ~
McCain said goodbye to campaign in Michigan.
McCain pulls out of Michigan.
http://www.politico.com/blogs/jonathanmartin/1008/McCain_pulling_out_of_Michigan.html
"John McCain is pulling out of Michigan, according to two Republicans, a stunning move a month away from Election Day that indicates the difficulty Republicans are having in finding blue states to put in play.
McCain will go off TV in Michigan, stop dropping mail there and send most of his staff to more competitive states, including Wisconsin, Ohio and Florida. Wisconsin went for Kerry in 2004, Ohio and Florida for Bush.
McCain's campaign didn't immediately respond to a request for comment."
Richard Nixon nailed it when he said, "Watch what we do, not what we say."
RCP has been cherry picking entirely from the right side of the tree. Of course, they're biased for the Republicans. They know that there aren't any serious consequences among their right-wing readers, so what the hell, why not.
A key difference is that you can choose what to focus on here, because all of the information is present. If you don't like Nate's model, you can simply look at the polling averages. If you don't like his pollster weighting system, you can just look at the raw polling data. It strikes me that this site is superior to RCP in access to the data, except that they provide links to the polls whereas Nate does not. The only actual polling data that Nate excludes, however, are internal polls, but he is consistent in not allowing them no matter which party is releasing them.
RCP would be wise to post an explanation of what polls they include and why. I'm surprised that they didn't give Nate an answer for this. Could they have not even thought it through? That seems impossible for a polling site.
You should worry about your own site Nate, not start a e-war.
I think this is all a tempest in a teapot. Frankly Nate, I don't think your post about this was really even necessary.
RCP doesn't purport to be doing predictive poll summarizing. Their primary purpose is personal commentary and summarizing the daily punditry. Poll averaging is a sideline. They just take what comes in and combine most of it to get a number. Over time I've had my own questions about what their method is for doing this, but I don't think it biases their results in any way that changes the outcome significantly. Anyone who looks at their results from 2004 would be hard pressed to declare their approach way off or biased.
On the other hand, 538 exists ENTIRELY for the purpose of predictive polling. As such, the choice of poll, how much credit it is given, and for how long are very much open to interpretation and comment. As is the model being used. The approach here is NOT the same as RCP. We shouldn't hold either site up to the standards of the other.
Finally, I just don't think we should be arguing about whether the polls have Obama up 5 or 6 points on OCTOBER 2nd. This is navel gazing. The only poll that matters is the one in November. I doubt that even a biased RCP is going to affect that, especially when you consider the ragtag audience of numbers geeks and political junkies that go there (and here) on a daily basis. To most Americans, this is a completely under the radar argument.
McCain pulling out of Michigan
http://www.politico.com/blogs/jonathanmartin/1008/McCain_pulling_out_of_Michigan.html
McCain Campaign ~ Best Campaign Ever ~
McCain already won MI
good decision !
You know, just like when an alcoholic wakes up in a puddle of their own puke in a gutter behind the bar... I've realized today that I'm a hopeless political junkie. I'm checking and rechecking 538 a couple dozen times a day, for God's sake. For what? An update on the polls. This is bad, really bad. If I can't get to 538 for my hourly fix, I start to shake uncontrollably and I lash out at those I love (okay, I do that anyway)... but my point is... what the hell am I going to do after the election???
Occam's razor.
I'd say RCP's bias is blatant.
You can't just drop ARG and then pick them up when they start to change. Was there change in methodology or was there change that favored the Repub narrative?
That said, it must've really pained McIntyre to post those Qunnipiac and CNN State polls, as both seem to have a D house effect.
And oh yeah.....THEY WERE BRUTAL!! :):):):):):)
Mason-Dixon has a buzzkill M+3 in VA. Someone needs to rip it's internals out and find out WTF is goin' on.
Greg Kuperberg said...
I don't have in mind that you have any intentional or unintentional bias in your statistical estimations. But in the name of disclosure, it would be interesting to see more of what is in your simulation model. It is after all the source of the most prominent statistic on the whole site, the Obama win percentage.
Keep in mind that this isn't just some dude's blog. Whether Nate makes any money from the site, I don't know, but he has a career centered around statistics, and the 538 model may very well contain some of his tricks of the trade. Whether it does or not, though, releasing it could pave the way for imitators and competitors to snatch up his ideas, repackage them, and get away with his profit. Imagine a site that had practically the same statistics, or even exactly the same, but with GOP oriented commentary. Imagine one that was able to be updated more frequently. Those two would probably take a big amount of the people already here.
While I am a fan of open source software, that's because it drives competition, and it's still possible to make a profit through selling things other than the idea itself (such as technical support). That doesn't exist with this-- either people are around to look at his statistics, read his commentary, and receive his ads (and presumably buy his books), or he's flipping burgers. (Kidding, Nate! I'm sure you'd make a fine accountant.)
Say he wants to make a fivethirtyeight book, chronicling the 2008 presidential campaign, and explaining how the various events impacted polls. How demographics have changed in the past four years. Maybe with the "Road to 270" and "On the Road" sections in there. I'd probably read it. But if this site loses popularity due to losing its ability to be competitive, he'd not only lose out on site revenue, but the huge impact advertising on his own site would bring him.
He also stands to gain a significant amount of career visibility through this, as evidenced by the fact that he's been showing up on news programs fairly often, which would also be a big loss for him if he lost readers.
Ripping the Mason-Dixon poll?
Here we go again:
I checked the internal makeup with Excel solver. ASSUMING that there is no "other" vote there would be 18% Blacks in the poll and 82% white. However, there were about 7% "others" in the 2004 Exit Polls. Let´s assume there are 6% in the Mason-Dixon-Poll, and they broke 60-40 Obama, since he IS in the lead with Hispanics, Asians, Natives or whatever you have there, I think.
This gives us the following race crosstabs:
Whites 78.5
Blacks 15.5
Other 6
I know that the NC exit polls overestimated black turnout, but since there ARE about 20% Blacks in Virginia, 15.5 looks a bit low. If you reweight the polls to match
75 White
20 Blacks
5 Other
you come up with a 48-46 Obama lead. And that sounds reasonable to me.
Just lie baby lie it's the GOP way.
test post
Sean said...
Worrisome CO poll from Ciruli (sp?) shows A LOT of Udall voters crossing over for McCain...
Obama 44%
McCain 43%
But you missed and important data from the ID party in this poll:
40% republican
28% democratic
So McCain down by 1 with a 12% party ID is not a good result for him.
I agree with the McCain campaigns decision to pull out of Michigan. It was recently pointed out here that he probably needs to choose between MI and PA, rather than try to take both. My confusion is in the realocation of those resources into WI. He is going to move resources from MI into a comparatively competetive state with 7 fewer electoral votes?
Why not double down in PA, where a win could really cripple Obama, rather than a state that would be offset by Obama picking off VA, NC, IN, or MO. Unless they see something really different about WI in their internals, that seems like a horrible waste of resources.
I suppose it's possible that this is just a headfake though, and they aren't seriously going to abandon MI for WI.
Rasmussen
New Mexico
Obama 49
McCain 44
Montana
McCain 52
Obama 44
Kentucky is McCain by 10.
You arrogant little twit.
"Firstly, we assign each poll a weighting based on that pollster's historical track record, the poll's sample size, and the recentness of the poll. More reliable polls are weighted more heavily in our averages."
So this is disclosure is it? This is empiricist mumbo-jumbo for "we make it up". The worst thing about you is you believe your own propaganda. Your site never struck me as one of the more statistically sound websites.
dario - not to mention the fact that the poll is ten days old. Incidentally, the party ID numbers are probably 40R 38D, closer to reality but still a decent number for Obama, considering that IND voters are grossly underrepresented in that poll.
was the site down.
otherwise it's a very slow day.
give us our poll fix!
Real joe, at least you.
Rasmus-
I think you're very close on the 2004 numbers for North Carolina.
But that was 2004. This is 2008.
North Carolina is one of the few states that lists registrants by race. The Obama ground game has been phenomenal here, and check this out.
African American registrants 2004:
1,116,278
African American registrants 9/27/2008:
1,258,564
That's an increase of 142,000 registrants! Blacks could be 22% of the electorate in this election.
See for yourself:
http://www.app.sboe.state.nc.us/NCSBE/VR/VR%20Stats/vr_stats_main.asp
No Matt, the ID are 40-28 for republicans.
The Montana numbers are very interesting. Bob Barr is on the ballot there, and he could draw 12-15%.
Just talked to a friend who is a newspaper editor in Billings, and he told me that in Yellowstone County (Billings), Dem registrations are outnumbering Rep registrations by about 700 per week.
If this is true all over the state, Montana is in play.
If McCain is pulling out of MI, that doesn't leave him many routes to the presidency. The most realistic is Bush states - IA -NM. It would mean that he has to play perfect defense, but it isn't completely implausible. Or he could win PA, which given the string of polls showing Obama with a high single-digit lead would seem very difficult. In theory he could go after WI or MN, but neither seem prone to flipping, and McCain has literally never had a lead in either since the start of the general election. Are there other options that I'm overlooking? My guess is that he's just hoping that his defense holds in CO, VA, OH, NC, NH, NV, FL, IN, and MO.
@Greg Kuperberg:
(& @ feba)
Nate has already published the basis of his model. At least one other poster I am aware of has replicated it. Anybody with the interest, the math skills and the sitzfleisch can do the same thing.
All you have to do is download or key in the data he has listed as what he uses, set them up in the spreadsheet and apply the proper regression analysis and voila, a replication.
He is under no obligation to to extra work just because you haven't read, or don't understand, the methodology disclosed in his FAQs.
TJB, you're an idiot, an asshole, and a fundamentally dishonest person -- i.e., a typical McCain supporter. Feel free to refrain from posting for a few more weeks.
People, note how he *changed the subject*. Don't play his asshole troll game by defending Nate --he doesn't need defending.
"Why not double down in PA, where a win could really cripple Obama"
Because McCain is trailing in every PA poll, most often in the 6-9 point range. I'm not sure that any amount of resources can make up that deficit.
"So this is disclosure is it? This is empiricist mumbo-jumbo for "we make it up". The worst thing about you is you believe your own propaganda. Your site never struck me as one of the more statistically sound websites."
You know, it would be easier to just post a simple message like: "I am an idiot who can't understand what Nate's model does." It would convey the same message but would be much more intellectually honest.
Not for nuthin' but you just had KKKarl Rove on Fixed Noise a few minutes ago trumpeting RCP numbers. Hmmm...
Seems you could just perform a statistical meta-analysis of RCP's selection of polls and grade RCP's subset for overall accuracy and bias, the same way you now grade individual pollsters.
@ mlewisham:
I am afraid that you are the arrogant twit. If you had taken the trouble to read the methodology section (under the keyword methodology or in the FAQs) you would have been able to see everything laid out.
You don't have to agree with Nate's opinions, and you can dispute his methodology (if you have the skills), but you should try to understand it first. Is that too much for your poor brain to deal with?
@Sedi
I completely agree that he is unlikely to win in PA, but I don't think the relative likelihood of winning WI justifies the 11 fewer EV's he is contesting.
The pollster average for the two states is nearly identical right now, and I think he would be better served in PA. I also question how much it matters though, unless the national numbers move I doubt McCain can win simply through allocation of resources.
Just to add another piece of information to the pile--if you to to RealClearPolitics' website, you see a wildly right-wing selection of views: Thursday's links include Mort Zuckerman, Pajamas Media, Andy McCarthy at National Review (attacking a real journalist, Gwen Ifill), Douthat @ the Atlantic, and the discredited Robert Novak, with Matt Yglesias as the only progressive voice. Moving down to the next batch, you've got The American Spectator--too far to the right for even most Republicans, Victor Davis Hanson, an article entitled "Liberals Sneer, Americans Cheer" for the right-wing Investors Business Daily, and the lunatic Brent Bozell, with only a link to the moderate Nicholas Kristof for "balance."
RCP got caught stacking the deck, and like most right-wingers, the whined and moaned about it. Don't back down, Nate. They can't be trusted, and you were right to call them out on it.
Wisconson is the logical choice.
McCain has to asume he can tighten the polls & win Ohio, Virginia & Florida ect. Then with Wisconsin he only needs only one of Colorado, New Hampshire or Nevada.
tkk13above has it exactly right. I worry that anyone would think "I'm not deliberately, consciously cherry-picking the polls that say what I want to say" would mean "my poll selection is unbiased".
They really should teach people about unconscious bias in schools...
Typical Republican! His counter-attack a complete red-herring!
"Bias" is a question of intent! "How can you accuse me of bias! You have no evidence! I strenuously deny it!"
Blah, blah, blah.
Who can probe his mind and see what his actual intent is? Is he cherry-picking polls because of an INTENT to skew the results in favor of Republicans?
Or is it institutional blindness?
What you are really accusing him of is INCOMPETENCE!
I notice NO refutation of the actual facts: that he runs a blatantly biased site, with the most outrageous hard right-wing articles all attacking Obama and giving advice to McCain and talking about how terrible it is that "Liberals" are criticizing Sarah Palin for being a "Movement conservative."
Everyone who criticizes her for being "inexperienced" is apparently a hypocrite: "Liberals Sneer, Americans Cheer!" http://www.ibdeditorials.com/IBDArticles.aspx?id=307752082486771
We're not even real Americans!
But, when he leaves Republican favoring polls on his site for weeks and ignores ones that favor Obama, that's not "bias."
Of course not! That's just his "scientific judgment."
He's a lying fraud! That's all.
He may not be conscious of his own mendacity, but to pretend to being objective when you're obviously not, is just hypocrisy.
At the law, intent is shown by ACTIONS. A person can be presumed to have had a mental state in accordance with their actions!
If we see someone push somebody off a cliff, we can infer they "intended" to do it, in the absence of contrary evidence.
Here, it's indisputable that he cherry-picks the polls and provides NO reasoning about why other, objectively more reliable polls are left off, except that they would favor Obama and Democrats.
He can deny all he wants, but it's pure B.S. He's arguing that he lacks SPECIFIC INTENT to defraud.
He doesn't need it. He's INSTITUTIONALLY BIASED which amounts to the same thing!
Cale,
I don't think he should have pulled out of MI. On the other hand, none of those states are looking very good for him right now, so McCain's team is really choosing from among a bunch of bad options. Protecting the Bush states as best he can is his most likely path to victory, but it requires playing a lot of difficult defense (CO, VA, OH, FL, NC, IN, MO).
mlewisham - Nate has posted in detail every calculation he makes, including how he calculates the pollster reliability ratings, how sample size affects weighting, and by what factor poll weightings decrease over time. It is entirely possible to replicate Nate's math, and he has clearly stated his policy for including or excluding polls. The fact that you're too lazy to read Nate's disclosure or too thick to understand it does not make him a twit; it makes you an idiot.
RCP seems to like keeping the race close by picking polls or adjusting averaging dates. I don't think it's necessarily partisan, but since Obama is mostly leading they might come across as pro-McCain. They are not alone, media is also keen on keeping the race close. I assume in Nate's model more general trends like "Obama is having difficulties in Ohio" get fair statistical treatment. RCP might have similar ideas but not being able to handle this perception in a fair statistical way, they cherry pick polls. A huge problem arises when actual trends change but RCP tries to hush it down. They'll catch up eventually.
RCP is biased, not even acting they are not. RCP started last election to try to influence the election IMO. I read them then, they picked stories that has the most right wing bias possible.
They have not changed, they are biased even as they try to hide it. They keep McCain polls up longer, they are pur bias.
QUIT GOING TO RCP! Go to cqpolitics.com or electicker2008.com
One last (probably not really last) attempt to argue for PA over WI.
Pollster.com: McCain down 4.1 in PA, 4.8 in WI.
538.com: McCain 14% to win PA, 9% to win WI.
Tipping point probability: PA 20% (#3), WI 8% (#15)
ROI Index: PA #11, WI not in top 15.
I don't see by what metric WI makes more sense than PA. I would in fact point out that in every metric besides the pollster.com average MI looks like a better target than WI.
I have written to RCP several times, with NOT EVEN AN ACKNOWLEDGEMENT from them, arrogant sods.
There's no FAQ, there's no rhyme or reason to their picks of polls and WHEN THEY DROP THEM OFF THE BOTTOM of the averages.
They are WANKERS.
McCain has quit MI, they failed in IA this week, if BO defends NH and PA and WI, or takes FL, NC, or VA - this is OVA!
You've presented strong circumstantial evidence of partisan bias. If he wants to rebut it, then he's welcome to.
If you want to look at an example of whether or not RCP is biased, The minnesota avg is a prime example. Normally when a candidate meets or exceeds a 4% margin in a state that state is moved from the tossup category to the leaning category by RCP. Minnesota is at +4.6% for Obama and is still called a tossup state.
http://www.realclearpolitics.com/epolls/2008/president/mn/minnesota_mccain_vs_obama-550.html
@cometboy -
"Finally, I just don't think we should be arguing about whether the polls have Obama up 5 or 6 points on OCTOBER 2nd. This is navel gazing. The only poll that matters is the one in November."
That is a bizarre thing to say on this site. If you believe this, what are you doing here?
RCP doesn't purport to be doing predictive poll summarizing. Their primary purpose is personal commentary and summarizing the daily punditry. Poll averaging is a sideline. They just take what comes in and combine most of it to get a number.
You have made Nate's case for him. In the case of its poll summary, RCP doesn't take its content, or its readers, seriously enough to care too much about the accuracy of what they're doing.
It's absolutely fair for Nate to call them on it. In fact, Nate is in the perfect position to do so, as someone who's given a great deal of thought and effort to these questions, and whose own techniques are objective by any standard I can imagine.
RCP isn't the only polling site that does this. Polling Report looks like they've occasionally played a game or two with including or not including ARG results in its listings of George W. Bush's polling results. They include ARG's results in some places on their site, but managed to drop ARG's very low Bush numbers from Bush's results.
RCP is widely followed. A friend of mine is the CEO of a company and follows RCP, and I think other top-level people do the same. Kudos to Nate for calling them out on their bias. Facts are facts.
Cale
McCain is not pulling out of PA.
He probably figures it is already saturated, and the extra won't make much difference.
The point is he never really tried in WI before. That means more low hanging fruit.
Think of effect as being something like logarithmic. 10 office pushes the vote 2%. 100 offices pushes it 4%. The same with advertising.
He could never do more than double his effort in PA, but perhaps a factor of 10 is posible in WI.
Nate (you incredibly sexy man - sorry, had to say it)...
Don't give him that much credit. He has proven that he is nothing but a right-wing hack, and you rightly called him out for it. If he can't stand the heat, he shouldn't be cooking the books.
We are with you Nate and appreciate someone who can read the polls telling us what they read in the polls. I only go to their site to see what they have as an 'average'... the articles that they 'cherry pick' are way too off to the right for me. I would rather stay here and just read comments then read what they select. You did what was right.
I have always wondered about the subjectivity involved in RCP's choice of polls, from as far back as the primary-era rejection of ARG... I recall searching the site for some time, fruitlessly of course, for any explanation of why certain polls were "asterisked" and therefore not included. I am just now beginning to realize the power inherent in the running of such a site with regard to the choice of polls to begin with; I suppose it's only natural in such an entrepreneurial society that the supply of third-party polls should be pretty staggering, but it raises legitimate questions of influence over numbers that ideally should speak for themselves. I have usually been pretty content with their moderation, especially in Jay Cost's really insightful posts on the HorseRaceBlog, but at times the Time Blog commentary has carried an unpleasant and unpleasantly obvious Republican bias. I've only recently discovered this site, but I have to credit you for the openness and responsibility with which your information is presented. I imagine that I personally will continue to check both sites religiously as the election period builds to its conclusion, but it is at once both unsettling and comforting to realize that, while numbers are not always presented in a natural way, there exists an alternative. Thank you for your work, and don't give up on your convictions.
hey nate silver
im a math/stat major and you're my hero.
you were beautifully predictive in primary polling, absolutely peerless.
thank you for this site.
;)
Nate:
How do we know if the average pollster is unbiased?
A key parameter in 538's (or anyone's) election modeling has to be the grand mean across all polls.
Speaking in rough terms, the grand mean is estimated as the average across all pollsters, and the grand mean is our best estimate of who is winning. But we don't have that large a universe of pollsters. It's entirely possible the actual sample of pollsters is biased on average. How would we know if they were or were not?
For example, Daily Kos (DK) (Research 2000) recently started a tracking poll. DK appears to have the strongest pro-Obama lean of all tracking polls at the moment. They may be right, they may be wrong, I can't tell. BUT, suppose that prior to DK starting up, we actually had an unbiased set of currently active pollsters. Then, when DK starts up, suddenly the average pollster will lean towards Obama.
One thought: 538 does have some background data on pollster quality from the primaries. Under the assumption that pollster quality carries over to the general election, then it could be possible to estimate the average bias of a set of pollsters. However, the incentives, (really, the utilities) of the various pollsters are different in the primaries and the general election. Whether you-the-pollster have a democratic or republican bias, you don't have much preference between primary candidates, (unless you are employed by a particular candidate). In contrast, in the general election, you will have a strong preference between the two candidates and thus may be inclined to tilt your polling.
If Sean's/Nate's discussions of the Obama ground game are correct, DK could come closest to estimating the final result correctly. And if public predictions in other areas of policy (oil prices for example) are symptomatic, its entirely possible that all pollsters could end up on the same side of the actual result. This usually doesn't happen in the political sphere, but do remember New Hampshire.
Am posting this over at my own blog as well...
I went to RCP regularly until I noticed that most of the articles they decide to include were pretty conservative-leaning, when I compared them with what appeared on Google and Yahoo news. Specifically, there are quite a number of articles from conservative blogs, very few from liberal blogs. Bloggers at RCP are also overwhelmingly right-winged. And yes, I notice how they seem to cherry-pick polls - most of their changes seem to favor McCain. There does appear to be some biases, intentional or not.
What's the hubbub about? I haven't spent any time at RCP except to occasionally check their Electoral College map at http://www.realclearpolitics.com/epolls/maps/obama_vs_mccain/?map=10 This clearly shows McCain/Palin losing badly, so can anyone explain to me how exactly they are in the tank for McCain?
In fact, looking right now this web site (538) shows Obama 333.2 vs. McCain 204.8 while RCP has Obama/Biden 353 vs. McCain/Palin at 185.... I'm not sure what this pissing match is all about.
Another possibility: RCP is cherry-picking polls to inflate McCain's prospects, but not because of partisan bias. Rather, traffic to their site depends on interest in the election, which depends on there being some uncertainty in the outcome.
Still another possibility: RCP is cherry-picking polls not necessarily to favor McCain but just to ensure that the relative rankings change from day to day. It's kind of like how US News & World Report changes the criteria for its college rankings from year to year, so that there is movement in the rankings among institutions that, realistically speaking, don't change much in the timeframe of a single year.
情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,潤滑液,自慰套,威而柔,威而柔,威而柔,威而柔,自慰套,跳蛋,按摩棒,充氣娃娃,自慰套,潤滑液,威而柔,FleshLight,跳蛋,按摩棒,充氣娃娃,跳蛋,按摩棒,FleshLight,充氣娃娃,情趣商品,情趣網站,情趣網站,潤滑液,性感內衣,充氣娃娃,按摩棒,情趣精品,跳蛋,情趣網站,情趣商品,跳蛋,FleshLight,充氣娃娃,情趣內衣,情趣精品,按摩棒,威而柔,自慰套,成人玩具,Nexus,lelo,聰明球,後庭,後庭g點,g點,美國fleshlight,STU訓練大師,Fleshgirls,Toys Heart,Tenga,日本 Vibratex,日本Toys Heart ,日本Tenga,美國aneros,rudeboy,英國rudeboy,英國Rocksoff,德國Fun Factory,Fun Factory,英國甜筒造型按摩座,甜筒造型按摩座,英國Rock Chic ,瑞典 Lelo ,英國Emotional Bliss,英國 E.B,荷蘭 Natural Contours,荷蘭 N C,美國 OhMiBod,美國 OMB,Naughti Nano ,音樂按摩棒,ipod按摩棒,美國 The Screaming O,美國TSO,美國TOPCO,美國Doc Johnson,美國CA Exotic,美國CEN,美國Nasstoy,美國Tonguejoy,英國Je Joue,美國Pipe Dream,美國California Exotic,美國NassToys,美國Vibropod,美國Penthouse,仿真按摩棒,矽膠按摩棒,猛男倒模,真人倒模,仿真倒模,PJUR,Zestra,適趣液,穿戴套具,日本NPG,雙頭龍,FANCARNAL,日本NIPPORI,日本GEL,日本Aqua Style,美國WET,費洛蒙,費洛蒙香水,仿真名器,av女優,打炮,做愛,性愛,口交,吹喇叭,肛交,魔女訓練大師,無線跳蛋,有線跳蛋,震動棒,震動保險套,震動套,TOY-情趣用品,情趣用品網,情趣購物網,成人用品網,情趣用品討論,成人購物網,鎖精套,鎖精環,持久環,持久套,拉珠,逼真按摩棒,名器,超名器,逼真老二,電動自慰,自慰,打手槍,仿真女郎,SM道具,SM,性感內褲,仿真按摩棒,pornograph,hunter系列,h動畫,成人動畫,成人卡通,情色動畫,情色卡通,色情動畫,色情卡通,無修正,禁斷,人妻,極悪調教,姦淫,近親相姦,顏射,盜攝,偷拍,本土自拍,素人自拍,公園露出,街道露出,野外露出,誘姦,迷姦,輪姦,凌辱,痴漢,痴女,素人娘,中出,巨乳,調教,潮吹,av,a片,成人影片,成人影音,線上影片,成人光碟,成人無碼,成人dvd,情色影音,情色影片,情色dvd,情色光碟,航空版,薄碼,色情dvd,色情影音,色情光碟,線上A片,免費A片,A片下載,成人電影,色情電影,TOKYO HOT,SKY ANGEL,一本道,SOD,S1,ALICE JAPAN,皇冠系列,老虎系列,東京熱,亞熱,武士系列,新潮館,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,整型,水噹噹,貸款,貸款,信用貸款,宜蘭民宿,花蓮民宿,未婚聯誼,網路購物,珠海,下川島,常平,珠海,澳門機票,香港機票,婚友,婚友社,未婚聯誼,交友,婚友,婚友社,單身聯誼,未婚聯誼,未婚聯誼,婚友社,婚友,婚友社,單身聯誼,婚友,未婚聯誼,婚友社,未婚聯誼,單身聯誼,單身聯誼,婚友,單身聯誼,未婚聯誼,婚友,交友,交友,婚友社,婚友社,婚友社,大陸新娘,大陸新娘,大陸新娘,越南新娘,越南新娘,外籍新娘,外籍新娘,台中坐月子中心,搬家公司,搬家,搬家,搬家公司,線上客服,網頁設計,線上客服,網頁設計,網頁設計,土地貸款,免費資源,電腦教學,wordpress,人工植牙,關鍵字,關鍵字,seo,seo,網路排名,自然排序,網路排名軟體,
徵信社, 感情挽回, 挽回感情, 徵信, 徵信社, 徵信, 捉姦, 徵信公司, 通姦, 通姦罪, 抓姦, 抓猴, 捉猴, 捉姦, 監聽, 調查跟蹤, 反跟蹤, 外遇問題, 徵信, 捉姦, 女人徵信, 外遇問題, 女子徵信, 外遇, 徵信公司, 徵信網, 徵信, 徵信社, 外遇蒐證, 抓姦, 抓猴, 捉猴, 調查跟蹤, 反跟蹤, 感情挽回, 挽回感情, 外遇沖開, 徵信, 徵信, 徵信社, 抓姦, 徵信, 徵信社, 外遇蒐證, 外遇, 通姦, 通姦罪, 贍養費, 徵信, 徵信社, 徵信社, 抓姦, 徵信社, 徵信社, 徵信, 徵信社, 徵信, 徵信, 徵信公司, 徵信社, 徵信, 徵信公司, 徵信社, 徵信社, 徵信社, 徵信社, 徵信社, 徵信, 徵信公司, 徵信社, 徵信, 徵信, 徵信公司, 徵信, 徵信社, 女人徵信
徵信, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵信, 徵信網, 徵信社, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵信,徵信網, 徵信, 徵信社, 外遇, 徵信, 徵信, 徵信社, 抓姦, 徵信, 徵信社, 外遇, 徵信, 徵信社, 徵信社, 徵信社, 抓姦, 徵信社, 徵信社, 徵信, 徵信社, 徵信, 徵, 徵信公司, 徵, 徵, 徵, 徵, 徵, 徵, 徵信, 徵信公司, 徵信社, 徵, 徵, 徵, 徵信, 徵信公司, 徵信社, 徵信, 徵, 徵, 徵, 徵, 徵信, 徵信社, 徵信公司, 徵, 徵, 徵, 徵信, 徵信社, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵, 徵
^^ nice blog!! ^@^
徵信, 徵信網, 徵信社, 徵信社, 徵信社, 徵信社, 感情挽回, 婚姻挽回, 挽回婚姻, 挽回感情, 徵信, 徵信社, 徵信, 徵信, 捉姦, 徵信公司, 通姦, 通姦罪, 抓姦, 抓猴, 捉猴, 捉姦, 監聽, 調查跟蹤, 反跟蹤, 外遇問題, 徵信, 捉姦, 女人徵信, 女子徵信, 外遇問題, 女子徵信, 徵信社, 外遇, 徵信公司, 徵信網, 外遇蒐證, 抓姦, 抓猴, 捉猴, 調查跟蹤, 反跟蹤, 感情挽回, 挽回感情, 婚姻挽回, 挽回婚姻, 外遇沖開, 抓姦, 女子徵信, 外遇蒐證, 外遇, 通姦, 通姦罪, 贍養費, 徵信, 徵信社, 抓姦, 徵信社, 徵信, 徵信公司, 徵信社, 徵信, 徵信公司, 徵信社, 徵信公司, 女人徵信, 外遇
徵信, 徵信網, 徵信社, 徵信網, 外遇, 徵信, 徵信社, 抓姦, 徵信, 女人徵信, 徵信社, 女人徵信社, 外遇, 抓姦, 徵信公司, 徵信社, 徵信社, 徵信社, 徵信社, 徵信社, 徵信社, 女人徵信社, 徵信社, 徵信, 徵信社, 徵信, 女子徵信社, 女子徵信社, 女子徵信社, 女子徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社,
徵信, 徵信社,徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 外遇, 抓姦, 離婚, 外遇,離婚,
徵信, 外遇, 離婚, 徵信社, 徵信, 外遇, 抓姦, 徵信社, 徵信, 徵信社, 徵信, 外遇, 徵信社, 徵信, 外遇, 抓姦, 徵信社, 征信, 征信, 徵信, 徵信社, 徵信, 徵信社, 征信, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信社, 徵信社, 徵信, 外遇, 抓姦, 徵信, 徵信社, 徵信, 徵信社,
^^ very nice
徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社,徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,
徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,徵信, 徵信社,
酒店經紀人,
菲梵酒店經紀,
酒店經紀,
禮服酒店上班,
酒店小姐兼職,
便服酒店經紀,
酒店打工經紀,
制服酒店工作,
專業酒店經紀,
合法酒店經紀,
酒店暑假打工,
酒店寒假打工,
酒店經紀人,
菲梵酒店經紀,
酒店經紀,
禮服酒店上班,
酒店經紀人,
菲梵酒店經紀,
酒店經紀,
禮服酒店上班,
酒店小姐兼職,
便服酒店工作,
酒店打工經紀,
制服酒店經紀,
專業酒店經紀,
合法酒店經紀,
酒店暑假打工,
酒店寒假打工,
酒店經紀人,
菲梵酒店經紀,
酒店經紀,
禮服酒店上班,
酒店小姐兼職,
便服酒店工作,
酒店打工經紀,
制服酒店經紀,
酒店經紀,
菲
梵,
Post a Comment