John Hood: Getting policy right takes time

Published 1:08 am Thursday, August 2, 2018

RALEIGH — Well, never mind, then. According to the only large-scale, randomized controlled trial of a statewide preschool initiative — Tennessee’s pre-K program —intervening early to boost long-term academic performance doesn’t work.

Thousands of low-income students in Tennessee were randomly assigned to the pre-K program or to a control group. While there were some academic gains for participants at the kindergarten stage, by the third grade the benefits disappeared. In fact, pre-K participants actually scored lower than their otherwise-comparable peers.

States aren’t just wasting taxpayer money, then. Preschool programs are actively harming their intended beneficiaries. If N.C. leaders care about disadvantaged children, they should shut down our state’s pre-K program.

Is that what you were expecting me to say? Do you think such a conclusion would be reasonable based on a single study, however well-constructed? If your answer to either or both of these questions is yes, I would respectfully urge you to consider carefully what it means to apply scientific concepts and findings to public policy.

One of those concepts is that the scientific method isn’t really about proving propositions true. As a great 20th-century philosopher of science, Karl Popper, argued, science is really about proving propositions false. Although a scientific theory may seem highly useful in explaining the past and predicting the future, there is always the possibility that some valid future experiment will disprove it.

“No matter how many instances of white swans we may have observed,” Popper wrote, “this does not justify the conclusion that all swans are white.” He didn’t mean that science wasn’t useful. The point is that if a proposition can’t be falsified by experiment, it isn’t a scientific proposition.

But what does falsification look like in the real world? Another great 20th-century philosopher of science, Thomas Kuhn, argued that the model of a broadly accepted scientific theory being disproved by a single experiment is unrealistic. What really happens is that the defenders of the theory assume the experiment is flawed in some way, or only applies to some situations. In other words, they explain the inconvenient finding away.

Instead, what ought to happen, and what does happen, is that scientists see a puzzle to be solved. They replicate the contradictory experiment. If the finding is the same, they seek to explain it without abandoning the established theory.

To return to the question of preschool intervention, the Tennessee study, an earlier and comprehensive evaluation of Head Start, and new statistical work by the Brookings Institution’s Russ Whitehurst all suggest little to no benefit. But you’ll find a number of other, smaller-scale studies pointing the other way.

Similarly, while dozens of studies show that school choice and competition are associated with academic and other benefits, initial studies of statewide voucher programs showed scant or even negative effects, although subsequent findings were more promising.

In response to such evidence, it would be reasonable to temper one’s short-run expectations and recognize that scaling up policy ideas can be challenging. But it would not be reasonable to abandon those ideas altogether. That’s not how science works. And it’s certainly not how public policy works.

John Hood is chairman of the John Locke Foundation.