Scrap the Squigglys

I'm with Mike Murphy on this one. The presence of on-screen results from dial-testing groups is something that needs to be reconsidered during future presidential debates.



It's not that the squiggly lines aren't fun to watch. Rather, they're too much fun to watch. It's hard to avert your eyes from them. It's hard to separate your own, independent reaction from theirs. And it's certainly hard to integrate back into to the non-squiggly universe once you've gotten hooked on the squigglys.

It was only a matter of time before one of the networks figured this out and started carrying the dial-testing results with their live debate feed. That network turned out to be CNN, which made a terrificly smart programming decision, and was rewarded with ratings comparable to the major networks. (Which channel do you think I watched last night?)

The problem is that the squigglys may give thirty random strangers from Bumbleweed, Ohio just too damned much power to influence public perception. The squigglys influence the home viewers, the home viewers participate in the snap polls, the snap polls influence the pundits, the pundits influence the narrative and -- voilà! -- perceptions are entrenched.

Mind you, I'm not complaining about the post-debate snap polls really, like the ones that CNN and CBS conduct. I'd certainly rather look at those numbers than watch the pundits babble for hours on end, especially as pundits tend to watch for all the wrong things during the debates.

But whereas the snap polls are scientific instruments with sample sizes of 500 or more, the probability of getting an unrepresentative reaction from a 30-person dial-testing group is much, much higher. First and foremost is the matter of sample size. You'd never see a poll conducted with just 30 respondents, because the margins of error would be around 18 (!) points.

In addition, as Mark Blumenthal points out, these focus groups depart significantly from truly random samples. Let me quote from him at length:

Focus groups do have important limitations that are not well understood. Although focus group recruiters try to make the participants as representative as possible, the focus group is not a projective random sample, like a poll. Participants usually live near the facility. As the response rates are miniscule given the time commitment, participants usually receive monetary incentives (usually $50-$75) to encourage participation. Recruiters also seek to fill specific quotas for specified demographic characteristics (a mix of ages, for example). Thus, we simply cannot count answers in a focus group to estimate the reactions of a larger population. In other words, if 20 of 30 "undecided voters” react a certain way to the debate tonight, we cannot conclude that 66% of all undecided voters nationally feel the same way.

A second limitation is what researchers call "group dynamic.” In a focus group, participants are often influenced or cowed by the opinions of others in the group. If one dominant personality loudly stakes out a position, others tend to hide or modify their contradictory views. [...]

Finally, the artificial nature of the focus group is often a poor way to judge how the information from advertising (or the fallout of a debate) will be processed in the real world. For example, focus group participants often express genuine antipathy for negative advertisements and reject the information contained in them as false and unfair. Yet in the real world, as the recent campaign has demonstrated powerfully, such advertising can still communicate negative information with ruthless effectiveness. Also, People no doubt watch advertising much more closely and critically in focus group than in their living rooms.

Mark is talking about interactive focus groups, which are a slightly different beast from the squiggly-line groups that CNN and the other networks use, but most of the criticisms carry over. You're not really getting a random sample when everyone has to be sitting in the same physical location at the same time; maybe voters in Ohio were really grooving on Obama's message last night, but voters in Florida weren't. Moreover, people may react differently when they feel as though they're being watched, and that their reactions are being broadcast in real time to 9 million Americans on CNN. The dial-testing groups may also be paying too much attention to the debate to mimic real-world conditions ... they're sitting there in a room with absolutely nothing else to do but watch the debate and twiddle their knobs. That's not how most people watch the debates. Most people are flipping channels between the debate and Dodgers-Phillies, or trying to put their kids to bed, or are chatting on the phone, or are four beers into a six pack, or all of the above.

What I'd suggest is that the CPD ask the network to refrain from including focus-group reactions in their live broadcasts of the debates. If the networks want to include the squigglys in their re-broadcasts of the debates, or perhaps on their Internet streams, I'd be all for that. But I think the viewer should be entitled to formulate her own, independent reaction to the debate, rather than having to share her television with Joe the Plumber and some guys from his neighborhood.