As North Carolina heads into the compliance phase of the No Child Left Behind education law, accountability experts must decide which students can legitimately be excluded from testing. Under the 2001 law, all students in grades four and eight will be required to participate in the National Assessment of Educational Progress in math and reading. NAEP is required to conduct the assessments at least once every two years under the law.
Before the No Child Left Behind law, state participation in NAEP was entirely voluntary. The National Assessment Governing Board, which determines policy for the NAEP, may offer tests in additional subjects and additional grades “to the extent that time and money allow.” What concerns the Governing Board is the effect that testing exclusions, or large changes in testing exclusions, have on the integrity of the NAEP’s results.
Exclusions and reporting
NAEP uses a sample of 2,500 students in each state to generate its measurements. NAEP is designed to provide information about the general level of skills and knowledge of students, and no student takes the entire test. All NAEP test questions, data, and assessment instruments are available for public inspection in a secure setting.
State education officials know that test scores, particularly in reading, tend to rise with testing exclusions. In an interview with Education Week, Darvin M. Winnick, chairman of the National Assessment Governing Board, noted the potential for gaming the results with student exclusions. “This is a longtime discussion,” he said. “Do you help your scores by excluding certain kids? It’s as simple as that.”
The National Center for Education Statistics manages NAEP for the U.S. Department of Education. At the NAEP’s board meeting with NCES, project officer Arnold Goldstein noted recent dramatic movements in NAEP participation rates. From 1998 to 2002, according to Goldstein, some states had increases of as much as 7 percent in exclusion rates. Other states dropped exclusions by up to 9 percent, Goldstein said. He also said “exclusion rates are correlated with increases in NAEP reading scores at the state level.”
While NAGB has the authority to deal with questions surrounding the NAEP, so far they have not set a specific cutoff for exclusion rates. The history of NAGB has been to focus on inclusion rates instead. According to the National Center for Educational Statistics, the intent of NAEP has been to include as many special-needs students as possible. Expectations under the federal law are that 95 percent of all students, regardless of disability status, be assessed by some means.
Testing accommodations
Special-needs students were not generally offered testing accommodations on NAEP assessments before 1998. NAEP assessed 31,398 public and nonpublic-school students in grades four, eight, and 12 for the 1998 NAEP reading report card. Of the 12th-grade students, 7 percent were classified as SD and/or LEP — students with disabilities or limited English proficiency. Twelve percent of eighth-graders and 16 percent of fourth-graders were SD, LEP, or both. About half of each group of SD/LEP students were excluded because they could not “participate meaningfully in the assessment.”
The guidelines for exclusion were identical in 1992 and 1994. Students with disabilities who were mainstreamed for less than 50 percent of school time, or judged incapable of meaningful participation, were exempt. Children who had spent less than two years in an English-speaking school, and could not participate meaningfully in English, were also excluded.
Under the new criteria, less than three years of English instruction, or an inability to participate in English, will remove a student from the testing pool. The revised criteria also will disqualify a student with an individual education plan that prohibits the NAEP test format, and a student with severely low cognitive function.
Field tests with the new criteria began with the 1995 NAEP reading test. Test accommodations were offered, on an experimental basis, in 1996. A split-sample design for the 1998 NAEP reading assessment gave the first statistical measures of how changes in special needs participation, and test accommodations, would play out.
According to NCES, “For the 1998 NAEP reading assessment, national and state NAEP school random samples were divided into two equivalent halves.” One- half of the sample took the test without any accommodations. Special-needs students in the other half-sample got extended time, one-on-one testing, or other provisions typically allowed in the schools. Translating dictionaries were prohibited. Questions and reading passages could not be read aloud.
The first 1998 NAEP report used data from student samples without any accommodation. In North Carolina, 38 percent of fourth-grade students were below basic in reading, 34 percent were at the basic level, 22 percent were proficient, and 6 percent were advanced, the report said. A ‘basic ’level means the student understands overall meaning, but may not be able to draw inferences, and cannot generalize to concepts outside the text.
The revised NCES report, Including Special-Needs Students in the NAEP 1998 Reading Assessment, showed that the modified assessment framework tended to reduce exclusions. Students who used accommodations made up 1 percent to 2 percent of the total number for each grade. By drawing in more low-performing students, the NAEP had succeeded in making the test accessible to a wider population. Average reading scores on the more-inclusive NAEP declined by one scale point in each grade. Accomplishing one goal raised new issues.
North Carolina’s fourth-grade reading results declined significantly using the revised procedures. Forty-two percent of students scored below basic, 31 percent demonstrated basic skills, and 21 percent were proficient. The percentage of advanced students was unchanged. Eighth- and 12th- grade scores also declined, but were not statistically significant.
States, and the NAGB, understand the implications of new NAEP procedures. The need to meet proficiency standards under No Child Left Behind law drives exclusions up. Because of the mandate to test all students, and the availability of a wider choice of options, low-performing students can now be included. The revised scores are probably a “truer picture,” NCES said. But recent volatility in participation rates, seen in the 2002 NAEP assessments, signal the depth of the dilemma that states are facing. The National Assessment Governing Board has already reversed itself several times on a policy to ‘flag’ results when exclusion rates vary by more than 3 percent. No “precise point at which exclusion rates would have a significant impact on average test scores” has been determined, Goldstein said. Nevertheless, NAGB may reinstate the ‘flagging’ policy until it can determine whether some states are using exclusion rates to unfairly influence test scores.
Palasek is a policy analyst at the North Carolina Education Alliance and assistant editor at Carolina Journal.