Thursday’s release of 2002 reading test scores from the National Assessment of Educational Progress has prompted North Carolina politicians to claim significant achievement gains since 1998 — but most or all of these gains may well have nothing to do with a real improvement in student learning, according to a preliminary analysis by a Raleigh-based research group.

Policy analyst Karen Palasek at the North Carolina Education Alliance said that the reported nine-point gain in the state’s average NAEP score for reading in the 4th grade from 1998 to 2002, and a three-point gain for eighth-graders during the same period, may appear significant and worthy of celebration. But North Carolina’s policies for testing and excluding its public-school students are so different from the national norm that, even according to the federal officials releasing the scores, it may be impossible to make valid comparison over time and among states.

“Educators, political leaders, reporters, and parents need to greet these reported test scores with great skepticism, and to read the fine print in the NAEP report,” Palasek said. “The reality is far different from what you might see in press releases or superficial reports about the scores.”

The main issue is how North Carolina and other states decide which students should be assessed and which should be excluded, in the latter case either for having a learning disability or for a limited proficiency in English. According to the 2002 NAEP report, only 7 percent of 4th graders nationwide were excluded from the reading tests for one of these two reasons. Moreover, the same percentage was excluded nationwide in 1998, making the reading scores for the two years roughly comparable.

In North Carolina, however, there was a much greater likelihood of these students — who typically score poorly on the test — being excluded from the NAEP sample. Fully 12 percent of the state’s 4th-graders were excluded in 2002, which was the highest exclusion rate in the nation. Perhaps more importantly, only 7 percent of the state’s 4th-graders were excluded in 1998. Thus, with a much greater proportion of low-scoring students excluded from taking the reading test in 2002, Palasek concluded, North Carolina’s average score should have jumped significantly regardless of whether the average student was improving or just staying the same in reading ability.

“It is simply not valid to suggest, as a statistical matter, that the 2002 results demonstrate a real gain in achievement for North Carolina students,” Palasek said. “There may have been a small gain, there may have been no gain, or there actually may have been a small loss.”

Nor can the statistics be properly used to prove that North Carolina’s average scores (222 for 4th-graders) necessarily exceed that of neighboring states such as South Carolina (214), Tennessee (214), or Georgia (215), she said. These three states — and most others in the nation — test a far greater proportion of their low-achieving students than North Carolina does. In the 4th-grade, for example, North Carolina excluded 12 percent but South Carolina excluded only 5 percent, Tennessee 3 percent, and Georgia 4 percent.

The National Center for Education Statistics manages NAEP for the U.S. Department of Education. At the NAEP’s board meeting with NCES, project officer Arnold Goldstein noted recent dramatic movements in NAEP participation rates. From 1998 to 2002, according to Goldstein, some states had increases of as much as 7 percent in exclusion rates. Other states dropped exclusions by up to 9 percent, Goldstein said. He also said “exclusion rates are correlated with increases in NAEP reading scores at the state level.”

The National Assessment Governing Board has already reversed itself several times on a policy to ‘flag’ results when exclusion rates vary by more than 3 percent, which is one-fourth of the exclusion rate for NC 4th-graders and one-third of the exclusion rate for the state’s 8th-graders. No “precise point at which exclusion rates would have a significant impact on average test scores” has been determined, Goldstein said. Nevertheless, NAGB may reinstate the ‘flagging’ policy until it can determine whether some states are using exclusion rates to unfairly influence test scores.

“It doesn’t matter whether North Carolina’s policy is the correct one or the policies of most other states are superior,” Palasek concluded. “The point is that North Carolina’s policy has changed dramatically, and is dramatically different from the norm. This fact makes it impossible to make fair and accurate comparisons between North Carolina’s performance in 1998 and its performance last year, or between North Carolina’s schools and those of their neighbors.”

Palasek wrote an analysis in the June 2003 edition of Carolina Journal about the NAEP exclusion issue, which is posted here.