Pants on Fire for Politifact

Every once in a while there is a Politifact whose little emoticon (true, mostly true, pants on fire, etc.) is so gobsmackingly wrong that it leaves you speechless. Here’s the latest.

The statement to be checked is an observation by gubernatorial candidate Mary Burke that Wisconsin’s school choice program “has no research that shows its going to improve student learning.” Writing for Politifact, Tim Kertscher rates that the statement “mostly true.”

He’s got it completely wrong. In fact, the only evidence that exists on whether the voucher program improves student learning says that it does.

In other words, the cute little emoticon (an obfuscatory bit of "entertainment" fluff which I desperately wish the paper would get rid of) is pretty much backward.

Here’s half the mistake. Because the average of all MPS scores (as reported by the Department of Public Instruction and essentially republished by the Public Policy Forum) is higher than the average of all voucher students, Politifact concludes that this “gives a clear edge to public school students over voucher students.”
 

No it doesn’t. Not even close.
 

Any researcher will tell you that in assessing student achievement as a result of some program or instructional strategy, one must control for the socioeconomic status and other relevant characteristics of the students being compared. In other words, it is essential to compare like to like; apples to apples, if you will.
 

Researchers also will tell you that, in assessing whether student learning has“improved,” some type of longitudinal study is required, i.e., you must determine whether the achievement of particular students who have been exposed to the innovation under study approved over a period of time. A snapshot of test scores doesn’t do that. Even a series of annual snapshots of large groups of students doesn’t do that because we can’t be sure that the groups we are comparing consist of the same students from year to year.
 

As Paul Peterson, Henry Lee Shattuck Professor of Government at Harvard, explains, simply reporting the “average math and reading scores of all students” “often [says] less about the quality of instruction at the school than about the students’ family backgrounds.” He also calls for a “growth-based measuring stick.”
 

The DPI and PPF reports don’t do either of these things. They compare all (or almost all) MPS students to voucher students who, until quite recently, were all low income (some working class families are now eligible) and likely to be children whose parent were dissatisfied with their performance in public school.  Even if the percentage of disadvantaged and minority students are similar for both groups as a whole, one has to control for relevant factors and crunch the numbers to determine whether one group has "outperformed" the other.

Neither report attempts to measure whether the improvement of students who move to a choice school compared to comparable children who remain in a public school.  (In fact, at least for PPF, there would be no way to do so with the data available to them.)

We could go on.  It is nonsense, for example, to compare the test scores of public and voucher students in Racine in 2014 because the choice program had only just begun in Racine and the voucher students had been in their new schools for only a few weeks. Some folks argue, moreover, that DPI treats students who have opted out of tests in  a way that disfavors voucher schools. There are other issues as well.

I would add that it is meaningless to treat voucher schools as a group since they are not subject to common control and differ dramatically from one another. If, in fact, well-run choice schools improve student performance then it would also be wrong to say that there is “no evidence” the program works. In fact, a quick look at the 2013 PPF report reveals that a majority of choice schools appear to have matched or exceeded the MPS average for at least one subject during the years in question and that a substantial majority of Lutheran or Catholic schools did so. The 2014 results, while not quite as clear, are comparable. Perhaps certain types of choice schools do improve student learning.

So the numbers that Politifact says give public school students a “clear edge” do no such thing.

There are, as far as I know, four studies that attempt to do what DPI and PPF do not. One very small study in the early years of the voucher program – when there was a lottery to participate in Milwaukee – showed that students who got into choice schools did better than those who lost the lottery. Two more recent studies, which Politifact cites but then ignores, shows that the presence of school choice is associated with improvement of test scores in MPS. That was an early rationale for school choice; it was hoped that competitive pressure would cause public schools to improve.

And then there’s the University of Arkansas School Choice Demonstration Project study – the only real study of whether the choice program“improved student learning.” The SCDP study found that Choice students were significantly more likely to graduate and go on to college. The study found that voucher students showed a greater increase in reading scores than MPS students. (That this happened in the last year of the study and the first year in which voucher schools had to, like MPS, publicize their scores doesn’t change that.) There were some other advantages for voucher students but there were not statistically significant, i.e., the researchers could not say that they were not a product of chance.

You can dismiss the statistically significant improvements as “too small” although they are at least as robust as the evidence for a variety of educational nostrums that are commonly touted. And these gains are accomplished at a bit over half the cost of a public school education. You can quibble over what caused the measured improvements. But the fact remains that they are “evidence”of improvement that Mary Burke falsely said does not exist.

How Burke’s statement can possibly be called “mostly true”is beyond me. The Politifact author regards the evidence for improvement as "thin." I'm not sure that statistically significant findings of improvement in the only properly designed study that exists can be so readily dismissed. But let's grant him this . Concerns about the extent of improvements or what really caused it are worth discussing.

But she said there was no evidence. That is false. It cannot be mostly true. This Politifact was not fact checking. It was an imposition of opinion.

Cross posted at Purple Wisconsin