I first came across this study by George Mason University's Daniel Klein and Zogby International's Zeljka Buturovic, which appeared as a journal article in Econ Journal Watch, which Klein edits, in a link at Tyler Cowen's site several weeks ago. Cowen links to about a dozen interesting pieces every day, and I thought Klein's study was so obviously flawed that it wasn't really worth commenting on. But now it has re-appeared in the editorial pages of the Wall Street Journal, with the somewhat non-sequitur title, "Are You Smarter Than a Fifth Grader?".
Here's what Klein and Buturovic did. They took a survey using one of Zogby's internet panels, which is by far the worst polling instrument that they could have selected. The panel was not weighted and was not in balance. For example, McCain led Obama 49-43 among respondents to the survey, even though roughly the opposite outcome was observed in the actual election -- and only about 4 percent of the respondents were Hispanic and only 39 percent were female. Then they asked 16 "questions of basic economics", as the Journal's sub-head describes them, and arbitrarily included eight of them in their analysis but threw the other eight out.
If you were expecting fifth grade questions about supply and demand, you'd be wrong. Let me just say: I come at this as a University of Chicago economics graduate who indeed disagrees with the liberal orthodoxy on many economic matters. But questions such as "[Does] poverty cause crime?", which was one of the questions that Klein and Buturovic excluded without explanation, are more like Zen meditations than matters of basic economics.
Others were poorly phrased, for instance: "A company that has the largest market share is a monopoly?". This is confusing; having the largest market share is a necessary, though hardly sufficient, condition for being a monopoly, and no alternative definitions were presented. Is this question really an objective basis for determining whether someone is more or less "enlightened" (that's actually the term that Klein uses!) about economics?
Some come closer to having a technically correct answer, but are more within the realm of trivia. For instance, "In the USA, more often than not, rich people were born rich?". Notwithstanding that the definition of "rich" is ambiguous, you could probably develop an empirical answer for this question based around studies of social mobility. But unless you'd spent a great deal of time reading the academic literature on mobility -- something that few laypeople will do -- there's really no way that you'd know it.
Finally, there are some questions about which there is considerable disagreement even within circles of academic economists -- as Klein should know, since he's commissioned several surveys of them. Economists are about evenly split, for instance, when it comes to the minimum wage. There is much closer to being a consensus on free trade, but there are an ample number of heterodox views. And in other cases -- like the Rand Paulian view that "More often than not, employers who discriminate in employee hiring will be punished by the market?" -- there is very little in the way of recent academic research at all.
So basically, what you're left with a number of questions in which people respond out of their ideological reference points because the questions are ambiguous, substanceless, or confusing. Klein is blaming the victims, as it were.
There would have been much better ways to construct a study like this one. For instance, questions could have been developed from standardized tests of high school students, like the AP Economics exam, or from surveys of academic economists. Such studies might well support Klein's thesis. But between the poorly-considered questions and the poor choice of survey partner, this amounts to junk science.