Questioning the PISA

International test scores are regularly cited by both sides in the debate over American education policy. Finland outscores us? Must be the highly-paid teachers! Or is it the back-to-basics curriculum? Or the nationalized structure? Or the homogeneous student population? Or the country's low gini coefficient? No one really knows.

In fact, the scores might not mean much of anything. Take the Program for International Student Assessment (PISA). With the new PISA scores released last week, some advocates are scrambling to spin the results. Others are more skeptical. As I wrote for National Review:

[Rick] Hess points to the mixing of city and country data, ...the inexplicable fall of Finland, and the extreme sensitivity of the rankings to the choice of test questions.

What about the mechanics of PISA test administration? Did every country follow the same strict procedures? Almost certainly not. Consider sample drop-out, which is one of the most frequent problems we confront in program evaluation.... PISA response rates vary widely from country to country. For instance, Finland tested 96 percent of its nationally-representative sample, but Mexico somehow tested only 63 percent of its own. One need not be a cynic to suspect some gamesmanship there.

And even if we believe that PISA scores have some reasonable degree of validity, can we really attribute most of the country-to-country differences to education policy? No. As I noted in a separate post for National Review, "While the U.S. gets mediocre scores, it also has demographic and socioeconomic challenges that the more homogeneous European and East Asian countries do not."

Interestingly, Mark Steyn took issue with that statement, arguing that European nations also have high levels of immigration from poor countries, yet they still seem to outscore the United States. Maybe the U.S. performs poorly even after controlling for demographic factors? Tino Sanandaji did a good job of testing that theory with the 2009 PISA data. To create a comparison that is as apples-to-apples as possible, he limited the European sample to students who were not first or second generation immigrants. He then compared these native scores to the scores of white students in the United States:

Sanandaji found that this comparison makes the U.S. look pretty good. Again, I don't believe that much can be learned from PISA scores one way or the other, but there is no evidence that the U.S. education system is failing miserably.