Better testing through “data forensics”?

The re-drawn chart comparing the various gradi...

Image via Wikipedia

With standardized testing occupying a more and more prominent place in American academic life, it’s only natural that cottage industries of all sorts should spring up around it. For example, there’s Caveon Test Security, which is the subject of this NY Times article. Snippets:

As tests are increasingly important in education — used to determine graduation, graduate school admission and, the latest, merit pay and tenure for teachers — business has been good for Caveon, a company that uses “data forensics” to catch cheats, billing itself as the only independent test security outfit in the country.

[…] Caveon says its analysis of answer sheets is the most sophisticated to date. In addition to looking for copying, its computers, which occupy an office in American Fork, Utah, and can crunch up to one million records, hunt for illogical patterns, like test-takers who did better on harder questions than easy ones. That can be a sign of advance knowledge of part of a test.

The computers also look for unusually large score gains from a previous test by a student or class. They also count the number of erasures on answer sheets, which in some cases can be evidence that teachers or administrators tampered with a test.

If you’re going to have this kind of testing at the kind of significance level we give it, then you have to have some security measures in place to make sure the credentialing that comes from the test is actually meaningful. With that in mind, it’s a little surprising we haven’t heard of more of these private data forensics firms popping up. Who’s been taking care of test security on the big-name tests up to this point? Locally appointed proctors? (Here in Indiana that hasn’t worked so well: this, this, this.) The testing companies themselves? Or nobody? (Related question: Who did the University of South Florida’s data forensics, if indeed the threat of data forensics wasn’t just a bluff?)

Possibly more interesting than the existence of data forensics firms like Caveon are the thoughts of John Fremer, Caveon’s founder, about standardized testing. In the NYT article he states:

Fundamentally…testing is a way of ascertaining what you know and don’t know and developing ranks, and the critics go right to the ranks. Well, it does rank, but on the basis of knowledge of the subject, and if you think that’s not important, there’s something improper about the way you think.

I’m going to assume that Dr. Fremer realizes that “knowledge” is only the bottom-most layer of human cognition, and what he’s saying is that knowing whether this layer is sound or not is important, and that testing is a way (not the way) of determining that soundness — and that he’s not saying that standardized tests are the best way to assess subject mastery. But surely there are those who believe this, and the rise of multimillion-dollar industries to ensure the soundness of a very narrow kind of assessment says something about our collective approach to education as well as the level of trust one can place in these kinds of assessments in the first place. When’s the last time we heard of  private firms being contracted to make sure our assessment of application, analysis, synthesis, and evaluation tasks are working well?


Enhanced by Zemanta


Filed under Education, Life in academia, Student culture, Teaching, Technology

2 responses to “Better testing through “data forensics”?

  1. You’re right–there are some people who think the best way to determine subject mastery is through a standardized test. At our school, we were recently told our teaching evaluations would be tied to standardized tests.

  2. Kaleberg

    “When’s the last time we heard of private firms being contracted to make sure our assessment of application, analysis, synthesis, and evaluation tasks are working well?”

    The reason you don’t hear much about it is that the process is historically wide open to corruption and bias. Standardized tests are a god awful way of assessing anything, but they are less corruptible, or, if you wish, subject to a different kind of corruption.