Since its reauthorization, the Elementary and Secondary Schools Act, or No Child Left Behind law, has required that school improvement programs be grounded in “scientifically based research.” According to Lynn Olson of Education Week, “Those words, or an approximation, appear more than 100 times in the reauthorization…”
So, the U.S. Department of Education and the Institute of Education Sciences issued a document, Identifying and Implementing Educational Practices Supported By Rigorous Evidence: A User Friendly Guide, to clarify what “scientifically supported research” means in the education field.
The idea behind the guide is to help schools determine which educational reforms have the research to back up effectiveness claims.
The Institute of Education Sciences has also established the What Works Clearinghouse, a central source of evidence for what actually works in education.
Effectiveness is increasingly the watchword in education circles. The N.C. Department of Public Instruction reported in January that for 2003, top students in the state’s public schools made far less progress than did their lower-achieving classmates.
According to the Raleigh News & Observer, State Board of Education Chairman Howard Lee focused on the fact that overall pass rates on the state’s achievement tests were up. “What I’m hearing is that we’re all fairly comfortable with where we are and we ought to let sleeping dogs lie for now,” Lee said.
William Sanders of the SAS Institute, and co-director of the SAS in School Research Project, noted at a John Locke Foundation event that students who are just under the adequate yearly progress bar, are “worth more” to a school’s success under No Child Left Behind than the children who are already making adequate progress.
They can make or break the school’s adequate yearly progress standing, and cause a school either to pass or to be labeled “needs improvement,” he said.
The need to get as many of these “close” children as possible up to the adequate progress level directs schools’ attention and resources away from high-achieving and very low-achieving (those with no chance of making adequate progress) students, Sanders said. Sanders calls this the “shed effect,” and predicts that both groups will lose out under No Child Left Behind mandates.
The N&O report confirms that, for 2003, top students in the state have, indeed, lost out.
This type of critical analysis will become increasingly important as North Carolina moves further into the NCLB process. According to state Accountability Director Lou Fabrizio, some students “may not be getting what they need” with current programs and testing. So schools will be looking for reforms that can deliver achievement benefits to all students.
In 2003, 53 percent of North Carolina public schools missed at least one of their adequate yearly achievement targets. A second year of missed targets would put children in the transfer-eligible category. That means the clock is now ticking toward mandatory change. Education reforms and other interventions, such as supplemental tutoring, are likely to be part of the overall accountability and school improvement process.
But for many parents, transfer is not desirable, and may not even be possible unless enough seats can be had in non-failing schools. In the 2003 Phi Delta Kappa/Gallup poll on public attitudes toward public education, parents expressed a strong desire to improve the schools their children currently attend, rather than to send them into an unknown situation at a different school.
The No Child Left Behind achievement timetable directs schools to reach 100 percent proficiency with every child by 2013-14. In this race for quality, the luxury of trial-and-error in finding programs that build achievement no longer exists. Using an analogy to pharmaceutical testing, the Guide is intended to help determine which education “interventions” can pass both the well-designed and effectiveness tests.
The U.S. Department of Education’s User Friendly Guide provides a checklist for evaluating the research on education practices. With reliable research to back them up, school officials should be able to choose program designs that have a high likelihood of success.
The Guide identifies a “gold card” standard for educational program design and research, starting with a randomized-controlled trial to evaluate program effectiveness.
According to the Guide, trial programs should start with well-matched candidates. Student demographics, the type of tests they will take, the time period for the trial program, and methods of collecting data all can affect measured outcomes, and must be as uniform as possible.
Children participating in the research study should be assigned the control or the program group via random selection. This helps eliminate unintentional bias in the results.
Researchers have to be particularly wary of factors such as parent or student motivation, the Guide says. These factors can create significant differences among otherwise similar students.
A control, or comparison group, is key. The results of pre-and-post tests alone don’t generate adequate scientific research, officials say.
Without a comparison group, it’s impossible to tell whether differences in test scores and other measures are related to the education program, or to other variables.
Pilot programs that are well-designed should give a reliable indication of their effectiveness, but they need not be perfect, officials say.
Programs with some flaws, such as less well-matched participants, higher-than-desirable attrition rates, or small sample size may still be acceptable, but come under the “possibly effective” banner. This means that schools and districts should approach them with some reservation, and the Guide warns against overstating expected success.
Finally, scientifically based research should report all of the program’s results, not just positive ones. In addition, researchers need to publish the results of statistical tests.
Statistical significance and the standard errors that arise in measuring outcomes give a good idea of whether the program is effective. If so, it should be possible to replicate those results in more than one school setting.
As Linda Darling-Hammond points out in “Standards and Assessments: Where We Are and What We Need,” at least 47 states have created standards for student learning, but “not all of these initiatives have accomplished the goals that early proponents of standards-based reforms envisioned.”
The heaviest emphasis in accountability has been test-based, partly motivated by No Child Left Behind, and partly, no doubt, motivated by states’ desires to avoid a low or last-place ranking when it comes to comparisons using nationally standardized tests.
According to Darling-Hammond, however, schools now using “test-based accountability [that] emphasizes sanctions for students and teachers have often produced greater failure, rather than greater success, for their most educationally vulnerable students.”
Can North Carolina afford the type of well-researched, well-documented reforms that the No Child Left Behind law requires? In many states, school officials complain that NCLB imposes unfunded mandates in terms of testing, ,transfers, and supplemental services.
The answer is “yes,” according to Sen. Elizabeth Dole’s office, which reports that North Carolina has more than $54 million in federal funding, earmarked for No Child Left Behind, that is uncommitted.
Palasek is a policy analyst with the John Locke Foundation and assistant editor at Carolina Journal.