RALEIGH – Good public policy analysis means, as you’ve no doubt heard many times, comparing apples to apples and not apples to oranges.
Easier said than done.
All too often, politicians and policymakers of all political persuasions make assertions and jump to conclusions based on anecdotal information that doesn’t allow for legitimate comparisons between alternative states. How do you prove that something wouldn’t have happened without a government program being in place to foster it? How do you compare trends that can be affected by a variety of factors, some under the control of lawmakers and some far outside it?
Analysts use several different means of transforming the policy equivalent of citrus fruits into apples. They construct models that include multiple variables so as to test the impact of one while holding others constant. They take particular examples, delve deeply into them, and produce case studies that attempt to explore the specific causes and effects. And often, they simply admit the fuzziness of the data and the imprecise nature of the comparison while nevertheless attempting to draw at least some useful conclusions.
Education policy is a field that is especially littered with multi-colored fruit. Here’s one example: Why do Wake County public schools outperform Mecklenburg County public schools on state tests? There are many possibilities. Wake’s average income is higher and its poverty rate is lower. Mecklenburg used forced busing to a far greater extent (until recently) than Wake to foster integration. The two systems have differed in leadership style, in the composition of their school boards, in policies for special-needs students, and in political leadership on their respective county commissions. Any or all of these could be factors. The honest answer should be – but likely won’t be, given the political stakes involved – that no one knows for sure.
Charter schools have suffered since their inception in 1996 from faulty comparisons. Critics have noted that their overall test-score performance has been lower than the average for district-run public schools. But that’s just the beginning of the story. North Carolina’s charters populate both the top and the bottom of the distribution when it comes to state test scores. Moreover, many of those at the bottom are so small that the inclusion or exclusion of just a few children can have a significant impact on ABC scores and rankings. Plus, many of these charters are essentially schools for students with severe behavioral or learning problems. Within school districts, these students are often assigned to alternative schools and their test scores aren’t counted in district or state averages. But if the same students are in charter schools, no such accommodation is granted in the reporting of scores.
Three scholars at the Manhattan Institute in New York have been studying this problem of charter-school comparisons for a while. The preliminary results of their efforts can be found at the Manhattan Institute web site. In brief, they pulled out “targeted charter schools” – those that essentially operate as alternative schools for special-needs kids – from the rest of the charter school population and then compared the latter to district-run public schools in several states.
The result may not be a perfect apples-to-apples comparison, but at least the charter-school oranges get polished to a modest shine. (Hey, I know I’m squeezing the metaphor a bit, but I hope it’s to your taste). The Manhattan scholars found a modest but statistically significant increase in performance among students at untargeted charter schools vs. their district-run counterparts. The effect was particularly strong for charter schools in Florida and Texas, where the educational benefits go beyond the modest. On the other hand, the effect was particularly weak in our own state of North Carolina, where the charters were associated with a small boost in scores but the results were not statistically significant.
As always, more research into the effects of charters, parental choice, and competition will be welcome and greatly appreciated. But this study represents a promising hint that, if a fair comparison can be made, charters can play a useful role in reducing the achievement gap and expanding educational opportunity.
Hood is president of the John Locke Foundation and publisher of Carolina Journal.