« Back to Volume 13 (2004), Issue 3
Pojdi na slovensko stran članka / Go to the article page in Slovene
Standard error of rating process in knowledge tests
Gašper Cankar
Full text (pdf) | Views: 14 | Written in Slovene. | Published: September 28, 2004
Abstract: Whenever tests are used for making decisions (so-called 'high-stake' tests), different aspects of reliability always become important. If tests are composed of complex items scored by human raters, measurement error increases due to raters' subjectivity. Error variance due to subjectivity of rating process can be estimated through means of generalizability theory which sometimes demand substantial computational strain. Square root of error variance due to subjectivity of rating process is called standard error of rating process (SNOP). The author suggests an alternate computational approach in the special case of two raters for each subject which is faster and easier to compute and should therefore facilitate the use of proposed statistic. Computation of SNOP is shown identical by both approaches on simulated data. The use and interpretation are illustrated by examples of Slovene Matura exams.
Keywords: Matura, knowledge tests, reliability, raters' error, generalizability theory
