Thursday, December 26, 2013

The problem with fact checking

Here's a recent verdict issued by PolitiFact: "The Heritage tweet oversimplifies the debate. The statement is partially accurate but leaves out important details."

Don't worry about the specific issue being examined. Instead, just ponder those two sentences for a minute. Focus on the word tweet. Is PolitiFact doing self-parody now?

I have a low opinion of the fact-check movement, since the supposedly "objective" fact-checkers are influenced by the same biases that reporters bring to other types of news. For specific examples, see my Manhattan Institute article, which also includes links to some delightful skewerings of the fact-checkers by Mark Hemingway.

At its worst, fact checking is an attempt to control political discourse by defining the boundaries of acceptable speech--all under the banner of objectivity and fairness. Take out the subjective and salacious ratings--"mostly false," "three Pinocchios," etc.--and re-label the pieces as "analysis" or "opinion," and I would have much less to complain about.

In the midst of the 2012 election, I annoyed my friends by sending them this self-written parody of a fact check: 
During his convention speech last night, Mitt Romney said that he would be a "great" president. We have called him on this falsehood before, but both he and his surrogates keep repeating it.
The FACTS: Romney never says exactly what he means by "great." The vagueness of the word allows him to twist its meaning in any way he pleases, which is a trademark of dishonest politicians. He is also implying that he can predict the future--an ability for which he has provided no evidence in the past. Strictly from a probabilistic perspective, it is unlikely that Romney will be "great" in any generic sense, since a recent poll of historians labeled just a half dozen past presidents "great." It is possible that Romney also includes the category "near great" within his definition of great, but that has never been explicitly stated, and even if true would not generate a more than 50 percent chance that his prediction will be borne out.

The VERDICT: After the first time Romney falsely claimed he would be a "great" president, we rated the comment just "mostly false," given the lack of specifics provided. However, even after we have called him on the lie, he and his surrogates continue to repeat it over and over. A spurned Fact Checker is an angry Fact Checker, and so we now rate this claim: Pants on Fire!

The most common reaction I received was, "Are you sure that's a parody?"

Monday, December 16, 2013

Ryan-Murray and the problem with bipartisanship

"I can bring people together." That's what every presidential candidate says during a campaign, and perhaps for good reason: polls consistently show that voters respond favorably to promises of goodwill, compromise, and bipartisanship.

But is it really a good thing when politicians "come together" to "get things done"? For smaller-government advocates, the answer is usually no. When bitter partisanship rules the day, Congress generally refrains from increasing spending or taxes--in fact, Congress refrains from doing much of anything in those situations. For conservatives upset with an endlessly expanding public sector, just slowing the growth is a real victory.

At least they're not enlarging the government.

Sometimes even extreme divisiveness produces good results. Take the sequester. Its forced spending cuts (including for defense) were seen as so unpalatable that the parties would simply have to agree to an alternative. Well, they didn't. Negotiations broke down, both sides pointed the finger at the other, and the sequester went through, delivering a rare treat for small-government conservatives and libertarians: genuine spending cuts.

But now suddenly Republicans and Democrats are in a bipartisan mood, and all the good feelings have produced a two-year budget known as the Ryan-Murray plan. The plan increases both spending and revenue today, with promises of future cuts that may or may not materialize.

How disappointing. It seems that when politicians work together in the spirit of goodwill and cooperation, the government usually gets bigger. So maybe divisiveness isn't so bad.

(For my take on the pension provisions in the Ryan-Murray plan, please read my piece for National Review here.)

Friday, December 13, 2013

Earnest suggestion or sarcastic insult?

Here's a headline from the Department of Unintentional Humor, care of the Washington Post:

h/t Robert VerBruggen

Monday, December 9, 2013

Questioning the PISA

International test scores are regularly cited by both sides in the debate over American education policy. Finland outscores us? Must be the highly-paid teachers! Or is it the back-to-basics curriculum? Or the nationalized structure? Or the homogeneous student population? Or the country's low gini coefficient? No one really knows.

In fact, the scores might not mean much of anything. Take the Program for International Student Assessment (PISA). With the new PISA scores released last week, some advocates are scrambling to spin the results. Others are more skeptical. As I wrote for National Review:
[Rick] Hess points to the mixing of city and country data, ...the inexplicable fall of Finland, and the extreme sensitivity of the rankings to the choice of test questions.

What about the mechanics of PISA test administration? Did every country follow the same strict procedures? Almost certainly not. Consider sample drop-out, which is one of the most frequent problems we confront in program evaluation.... PISA response rates vary widely from country to country. For instance, Finland tested 96 percent of its nationally-representative sample, but Mexico somehow tested only 63 percent of its own. One need not be a cynic to suspect some gamesmanship there.
And even if we believe that PISA scores have some reasonable degree of validity, can we really attribute most of the country-to-country differences to education policy? No. As I noted in a separate post for National Review, "While the U.S. gets mediocre scores, it also has demographic and socioeconomic challenges that the more homogeneous European and East Asian countries do not."

Interestingly, Mark Steyn took issue with that statement, arguing that European nations also have high levels of immigration from poor countries, yet they still seem to outscore the United States. Maybe the U.S. performs poorly even after controlling for demographic factors? Tino Sanandaji did a good job of testing that theory with the 2009 PISA data. To create a comparison that is as apples-to-apples as possible, he limited the European sample to students who were not first or second generation immigrants. He then compared these native scores to the scores of white students in the United States:

Sanandaji found that this comparison makes the U.S. look pretty good. Again, I don't believe that much can be learned from PISA scores one way or the other, but there is no evidence that the U.S. education system is failing miserably.

Monday, December 2, 2013

Social planners must feel the weight of history

When Nick Kristof recently extolled Oklahoma's universal pre-school for four-year-olds, it was deja vu for those of us who are hard-headed about evaluating social programs.

Oklahoma pre-school has never been subject to a "gold standard" evaluation--meaning a large-scale, randomized experiment. Kristof's column came just weeks after a gold standard evaluation of pre-school in Tennessee found essentially no differences between the treatment and control groups. And that evaluation follows decades of experiments showing little to no effect of similar programs. Nevertheless, Kristof's enthusiasm for Oklahoma is unchecked by the weight of history.

There's a pattern here. As I wrote for National Review recently:
Skeptics of government social programs have long been frustrated by a familiar pattern of media coverage. It starts with a new social program portrayed as the key to fixing problems like poverty or low test scores. The media proceed to either ignore the long history of similar initiatives found to have little effect, or they acknowledge the history but report that this program is fundamentally different from everything that has come before.
Either way, the media trumpet the positive results from early evaluations of the program, even when those results are based on small samples, short time horizons, non-experimental data, or all of the above. Years later, researchers find that a randomized experiment shows that the program has no effect, or that the effects quickly fade, or that the effects cannot be replicated on a larger scale. These findings are quietly reported in a technical article, with no fanfare and little public attention. By that time, the media have already moved on to promoting another new social program.
Read the whole thing here.