Students in North Carolina’s public schools face end-of-grade and end-of-course tests to determine how well they’ve mastered the material taught during the school year. But who tests the tests? Until 2009, the content and design of North Carolina’s tests were secret. When the state released some information about its tests last year, observers such as Dr. Terry Stoops, John Locke Foundation director of education studies, pored over test questions. Stoops discussed the results of his research with Donna Martinez for Carolina Journal Radio. (Click here to find a station near you or to learn about the weekly CJ Radio podcast.)
Martinez: Who creates the state’s tests for high school students?
Stoops: Well, there’s a 22-step process to …
Martinez: Oh, my.
Stoops: … creating tests for the students, and that basically comes from the state. The Department of Public Instruction spearheads the effort with assistance from teachers that they hire to write the test questions and to refine them. It takes a very long time for these tests to be developed — approximately three years. So this is not a, sort of, fly-by-night process; it’s a very long process. It’s mostly headed up in Raleigh.
Martinez: I thought, Terry, that those tests were kept private so that they would not get out to the public so that no high school students could figure out what’s on them. But you managed to get your hands on a couple of tests. How did you do that?
Stoops: Well, I wouldn’t have been able to do it had the Department of Public Instruction not decided to create new assessments, new tests, for the curriculum revision that’s happening now and will be happening for the next few years. Previously, individuals who took a look at the state’s tests had to sign a confidentiality agreement, could not reveal any of the test questions that were on there. But we had a unique opportunity, when the Department of Public Instruction released the 2008-09 tests, to look at what kind of things are on these tests. Do they make sense sequentially? Are the test questions covering a variety of areas? It’s actually a very exciting opportunity to see what our students are expected to know.
Martinez: You have honed in on the economics test and the civics test for high school students. Why those two subjects?
Stoops: Well, these are the subjects where there are a lot of subjective points of view [as] to whether certain questions or points and interpretations are correct or not correct. We also took a few questions from the U.S. history test that dealt with economics. These are subjects where there’s disagreement in the academic community whether certain phenomenon happen because of “X” or because of “Y.” So we wanted to see what the experts thought about these test questions and about whether students are expected to know, to really think about them critically, or expected to know an answer that doesn’t make a whole lot of sense.
Martinez: So not only did you analyze them, but you went out into the North Carolina education community and asked some other people to take a look at these tests. Who participated in this analysis for you?
Stoops: We sent 500 surveys to professors all across the state — professors of political science and of economics and of business — and we sent these out to public universities [and] private universities throughout the state and asked them to evaluate these test questions. Now, if they all came back to me and said, “These test questions are excellent, the testing program is doing a great job,” then that’s what my report would say. I wasn’t looking for a certain answer. I simply wanted to see what the scholarly community in North Carolina thought about these test questions, whether we should praise or whether we should criticize the state tests.
Martinez: And what does the scholarly community think?
Stoops: Well, the overwhelming majority of the responses that I received express concern over these test questions. There were disagreements whether the answers were right. There were disagreements about whether the test questions lent themselves to a specific answer. There was a lot of disagreement about possible answers on these tests. One of the things I do in my report is I document all of the comments that were left on the surveys, because I gave the professors an opportunity to write comments about these test questions. So every one of those comments is in the report, and they’re very, very revealing.
Martinez: Did you get responses from them, Terry, that were more, “Hey, this is wrong,” or was it more about interpretation? Were their comments about the answers or the questions themselves?
Stoops: We got truly a mix. It’s really down the middle. There were professors that believed that none of the answers provided were correct. There was one question on the economics test that professors answered [and] thought that different answers were correct.
Martinez: So even the professors disagreed in that case.
Stoops: That’s right, that’s right. There were professors that thought “B” was the answer when “A” was actually the answer, and professors that thought that “A” was the answer. There was one question where the professors thought that all of them were possible answers. But they also targeted the questions — does the question make sense? And I think a lot of the professors found that these questions don’t make a whole lot of sense, can easily trip up a student who knows the material, has thought through the subject, has thought through the answers, but because the question is so poorly worded or phrased or uses a term that doesn’t really correspond to the answers, that a student can answer the question wrong but have a real justification for their answer.
Martinez: That sounds like something a parent would want to know because, as parents, we all, of course, are watching for the grades that our kids bring home, even when they’re in high school. It would be interesting to know that there are all these different reactions to the questions themselves.
Stoops: That’s absolutely right. And my biggest concern was that you have students taking these tests, and they got questions wrong on the test, and some of them didn’t pass by a few questions, a handful of questions. Now they may have known the material, they may have thought through these questions, understood what the correct answer was, and answered something that the state did not believe was the correct answer, and, therefore, they didn’t get the level III, which is the at-grade-level score. And that should be our biggest concern — that this makes students look like they know less than they really do.
Martinez: And, in fact, this is part of this ongoing discussion over whether or not testing in this manner is the way to evaluate students.
Stoops: That’s absolutely right. And you know, it really lends itself, too, to the question of whether this process is correct. Should we be shutting out these professors, these experts in the area, or should we just leave it to the middle and high school teachers to write these test questions? I think we need to bring the expertise in. I think we need to look to our university professors, especially at our public universities, to look at these test questions and really see if they make sense. Because they are going to be teaching those students that come in from high school that take these tests, that have the complete wrong conception about economic concepts or political science concepts.
Martinez: Do you think that the state should be in the business, at all, of putting together these tests? Or, should we do something different?
Stoops: I think we need to do something different. I think we need a test that is nationally normed — a test [so] that, basically, we can compare North Carolina students to students from other states. The Iowa Test of Basic Skills, the Stanford 10, the California Achievement Test — these kind of tests are taken by students in states across the nation. North Carolina’s tests are taken by students in North Carolina; we can’t compare them to anyone in any other state. So I think it’s important that we move to a testing program that allows us to have a more accurate gauge of where students are, and our testing program does not do that. We shouldn’t be in the testing business, which is very expensive. We should be looking for an off-the-shelf test that we give to our students and compare them to students nationwide.