Real Oklahoma Students Ace Citizenship Exam; Strategic Vision Survey Was Likely Fabricated

In detailing some of the evidence against Strategic Vision LLC, a pollster I am now almost certain is disreputable and fraudulent, I pointed in particular to a poll that they conducted on behalf of the Oklahoma Council of Public Affairs, an conservative-leaning educational thinktank. The poll purported to show that Oklahoma's high school citizens were deficient in some of the most basic aspects of citizenship. Only 23 percent of them knew that George Washington was the first president, the poll claimed! Just 43 percent knew that the Democrats and Republicans are the two major political parties!

These conclusions seemed dubious to me on their face. Several years ago, at my old consulting job, I participated in a project for the State of Ohio's public schools which involved sitting down in a third or fifth grade classroom for the better part of a day and seeing how the students were learning. Most of these observations took place in poor, post-industrial towns, which were still suffering the effects of the steel mill or the axle plant that had long ago left town. What struck me, most of all, was how smart the kids were, relative to my expectations. These kids might not have been the highest achievers -- but I'm pretty sure that more than 90 percent of them would have known who George Washington was. And these were third and fifth graders.

There were other hints too, that Strategic Vision's poll may have been fake. The scores that Strategic Vision claimed the kids had gotten, for instance, were strangely underdispersed. And they seemed to contradict results from Oklahoma's own standardized testing, which asked much more difficult citizenship questions and found most of the students doing just fine.

It turns out that I was not the only person who had doubts about the survey. So did Ed Cannaday, the State Representative from Oklahoma's 15 House District.

In a telephone interview yesterday, Cannaday told me he was shocked when he heard of the results, which had received widespread media attention. "When I saw the statistics, I was just flabbergasted and said it cannot be true," he told me.

There were two items in particular that sent up warning flags for him: the one claiming that only 23 percent of the students knew the identity of George Washington, and another that claimed that about one in every ten students had listed the two major political parties as "Republican and Communist".

"Given the dialog of today, if they had said Republican and socialist, then maybe," Cannaday told me. "But communist -- that's just not something that you throw out there any more. I don't think Sarah Palin even used that term."

Cannaday, age 69, would be in a position to know. Before entering the State Legislature three years ago, he had spent decades in education, first as a teacher in a large public school in Kenosha, Wisconsin, and then in Oklahoma where he set up an alternative school. After a stint in private business, Cannaday returned to classroom, first as a teacher and then as a principal, and then -- finding he missed the one-on-one interaction with his students -- as a teaching principal at a small school in House District 15. He now serves on the House's education committee in Oklahoma City, and continues to pay regular visits to the schools in his district. "Most schools like to have me once a month," he says, to talk about legislation pending before the state.

Cannaday therefore had little difficulty setting up an experiment: he arranged to have all the seniors in the 10 secondary schools in his district take the Strategic Vision/OCPA survey. Cannaday tried to replicate the Strategic Vision survey to the greatest extent possible. The same exact questions were used, and as in the case of the original survey, the answers were open-ended rather than multiple choice. The survey was administered to a total of 325 seniors, including special education students.

Cannaday's survey however, found his students doing just fine: They answered an average of 7.8 out of the 10 questions correctly. By comparison, the high school students that were purportedly surveyed by Strategic Vision had gotten just 2.8 out of the items correct. 98 percent of the students on Cannaday's survey -- not 23 percent -- knew that George Washington was the first President. 81 percent -- not 14 percent -- knew that Thomas Jefferson had written the Declaration of Independence. 95 percent -- not 43 percent -- knew that the Democrats and Republicans are the major political parties. There was just no comparison between the two.



Cannaday distributed his results via e-mail to the constituents on his mailing list, including Karina Henderson, who published his findings in a dairy at Daily Kos. He also sent hard copies to each of the schools in his district, as well as all of Oklahoma's state legislators. The reaction so far has been entirely positive -- "even from the Republicans," said Cannaday, a Democrat.

Cannaday also sent his results to OCPA, the thinktank that had commissioned the survey, but has yet to receive a response. In October, before the results of Cannaday's survey had surfaced, OCPA had told the Oklahoma Gazzette that they were taking "a closer look at the raw data and the methodology,” behind the Strategic Vision survey but were not yet ready to "toss out" the results.

House District 15 is generally quite representative of Oklahoma, especially its Eastern portion, but is somewhat poorer than the state as a whole. "Rural" was the first adjective that came to mind when I asked Cannaday to describe his district -- no town has more than 3,000 people. Most of the residents make their living in the natural gas industry, commute to service-sector jobs in the comparatively large towns of Muskogee, Oklahoma or Fort Smith, Arkansas, or are engaged in what Cannaday calls "cow/calf operations". The five counties that make up the district range from middle-class to impoverished. Haskell County, for instance, where the town of Stigler is located, has one of the highest unemployment rates in the state and one of the largest proportions of its students on free and reduced lunch programs, the preferred benchmark of socioeconomic status in public education. House District 15 has no private schools.

Cannaday is proud of the achievements of his students -- particularly their low drop-out rate, which is about five percent, and their success in the state's mock trial tournaments, where they've frequently finished in the top 5 in the state competing against much larger schools. He has seen his students become doctors, attorneys, optometrists and accountants, he told me. "Any time you can have one of your former students in your district who's on speed dial in Oklahoma City as a physician, that's not too bad," he said.

But the schools in House District 15, which sends 40-50 percent of its students to college and sees 20-25 percent compete it -- are not exceptional in any obvious way. The students at Haskell High School, for instance, received below-average scores in 5 of the 7 categories tested by Oklahoma's standard exam, including in U.S. History.

There is no reason to think, in other words, that the students in House District 15 should have gotten such profoundly superior results to the "students" in Strategic Vision's survey. Nor could Strategic Vision's results have been the result of any sort of mathematical or methodological oddity. Consider their claim that literally none of the 1,000 students they surveyed were able to answer more than 7 of the 10 questions correctly -- lower than the average score achieved in Cannaday's test.

There are, rather, only two possibilities. Either the Strategic Vision survey was entirely fabricated -- or Cannaday's was.

I would put every dollar to my name on Cannaday, who has kept the surveys and is happy to show them to them to anyone who comes asking.

Next week is Celebrate Freedom Week in Oklahoma, with public schools students to be taught from a special curriculum highlighting the Declaration of Independence. "If were going to be pass education reform then we need to be out in the classroom demonstrating it," Cannaday said of his fellow legislators. "I will be in a classroom Friday," he told me. "I enjoy it."

I e-mailed to David E. Johnson, the CEO of Strategic Vision, a draft of this article and asked for any comments. The entirety of his comments were as follows:

"Thank you for the opportunity to respond. Our company did survey the Oklahoma students grades 9-12 for the Oklahoma Council of Public Affairs. Our client has all of the raw data, cross tabs, methodology from the survey."

Johnson did not reply to a second e-mail asking for his interpretation of the substantial difference between his results and those found by Cannaday.