North Carolina students performed better than expected on nationally standardized tests, according to a new study that ranks states and students by a composite “teachability index.” The report, “The Teachability Index: Can Disadvantaged Students Learn?” by Jay Greene and Greg Forster of the Manhattan Institute, seeks to separate the effects of being disadvantaged from the effects of education reform and school spending. By doing this, the Teachability Index may be useful for judging the value of school reform.
Nationally, the authors show that students are “somewhat more teachable now than they were in 1970,” a ‘teachability’ gain of just under 9 percent. Most of the improvement has occurred since 1994, and has continued at least through 2001, according to the Manhattan Institute study.
North Carolina has not fared so well in student teachability. Roughly speaking, the index estimates that North Carolina’s students are about 14 percent more difficult to teach now than they were in 1970.
Teachability isn’t the only thing that determines academic outcomes. The study is quick to point out that in some states school performance and school efficiency more than compensate for harder-to-teach students. North Carolina is one of those states. School performance in North Carolina is, according to the report, about 9 percent better than would be expected, given teachability and other factors, and school efficiency statewide is about 22 percent above expectations.
Greene has become something of an education index guru. He has produced studies that allow cross-state comparisons of measures such as the SAT, graduation rates, educational freedom, and college preparedness. The latest report combines measures from six indexes, including race, family, and community among others, and incorporates 16 different explanatory factors in all.
“If hard-to-teach students can’t be brought up to minimal levels of basic skills even by schools that have benefited from reforms, while easy-to-teach students will pick up the same skills even in mediocre schools, then clearly reform is a waste of time,” Greene said. In an atmosphere of ongoing concern about student outcomes, discussion about the level and type of reforms we implement takes on added significance.
In 1983 the Excellence Commission on Education produced a report titled A Nation At Risk. In it the commission recommended sweeping reforms in K-12 education, both in academics and in structure. The reforms were designed to boost rigor in the classroom and make American students academically competitive with students from other nations.
By 2003, however, it was apparent that the suggested reforms had never really been implemented. American students were still far behind most of the world. This was especially critical in the fields of math and science (CJ May 2003). In the followup report Our School and Our Future: Are We Still At Risk? the Koret Task Force on Education at the Hoover Institution found little improvement 20 years after the initial recommendations appeared.
If academic outcomes have remained essentially constant since 1970, school spending has not. Greene notes that “inflation-adjusted education spending per pupil has doubled in the past thirty years, while student achievement and graduation rates have remained flat.”
The flat results Greene quotes are from the National Assessment of Educational Progress Long-Term Trends Report. He examines the scores of 17-year-olds on NAEP reading and math exams from 1969 to 1999.
NAEP tests are nationally standardized examinations used to measure student proficiency and make comparisons of students across states. The long history of the NAEP allows for a longitudinal analysis of student achievement in a number of disciplines. Since scores from 17-year-old students are a reasonable reflection of the cumulative effects of public education, and other factors that affect learning, their use makes sense in this type of study. Because of the sampling method of the long-term NAEP assessment, state-level reports are not available for 17-year-olds.
In 1972 the average American 17-year-old achieved a score of 304 on the NAEP math test, but by the time the Excellence Commission began its report in 1982 that score had fallen to 299. In 1999, students achieved a 30-year peak score of 309, but fell to the 1972 level of 300 the following year. The appearance of A Nation At Risk in 1983 did not fix the achievement malaise in math scores. Students in 2000 were no more proficient in math than their 1972 counterparts.
Reading performance has been almost literally flat over the 30-year period in the Greene and Forster study. Seventeen-year-olds scored a 285 on the NAEP reading exam in 1972, a 285 in 1983, and a 288 in 2000. Despite huge additional spending, the claim that student outcomes have been ‘flat’ for three decades seems justified by the NAEP results.
What is a Teachability Index? Greene’s measurement system uses 16 factors to create six preliminary indexes: readiness, economics, community, health, race, and family. By examining these factors separately Greene hopes to explain the role that advantages and disadvantages play in student performance.
Statistical techniques allowed researchers to test the relationship between the Teachability Index and academic outcomes. Those outcomes are measured by NAEP scores, the percentage of students with “basic” or above proficiency on the NAEP, and high school graduation rates. To avoid confusing proficiency with spending effects, the analysis controlled for differences in education spending in each state.
A Teachability Index can tell researchers whether schools are facing students with greater challenges and how well they are teaching the students they enroll.
While teachability measures make it possible to estimate how well students can be expected to do, the School Performance Index measures statewide school effectiveness with a given set of students. North Carolina ranks 43rd nationally for teachability, but fifth nationally for school performance, with students scoring 9 percent above expectations on the NAEP tests.
Trying to separate social, economic, racial, and gender effects from spending requires a look at efficiency as well. Here the question is: Which states get the best results per education dollar spent? Cost-of- living differences in each state were factored out using data from the American Chamber of Commerce Research Organization for the third quarter of 2003.
Based both on teachability and spending, the School Efficiency Index ranks North Carolina ninth in the nation. School efficiency, according to the authors, is 22 percent above what the other indexes would lead investigators to anticipate.
If children are forced to focus on personal safety, live in single-parent homes, and deal with untreated health problems, poverty, or issues that are generally adult responsibilities, they will become less ‘teachable,’ Greene and Forster conclude. Although some states have experienced a decline in the teachability of their K-12 students, teachability is not an excuse for poor performance.
Evidence from states like North Carolina demonstrates that schools can sometimes outperform expectations.
It is also clear that states can underperform. These results undermine the idea that education spending drives results. Since “inflation-adjusted education spending per pupil has doubled in the past thirty years, while student achievement and graduation rates have remained flat,” there is more in the mix than additional spending. “States with low scores on the index do not inevitably produce low-scoring students” and “states with high scores on the index do not inevitably produce high scoring students.” Claims that students are harder to teach don’t explain a long-term plateau in American educational progress.
One bright spot is a positive correlation between accountability and choice in the School Performance Index. Under Greene’s formulation of Education Freedom (CJ April 2003), the availability of nontraditional options has a positive effect on learning. Accountability, through student testing, is also positive.
“In explaining school outcomes, education experts have long stressed school inputs—money and students’ backgrounds—often to the exclusion of other factors.” Teachability suggests that they look harder at what they do with those inputs.
Dr. Karen Palasek is assistant editor of Carolina Journal.