Abstract: When tests or portions of tests are scored subjectively by raters, a rescoring will yield a change in the ratings of some examinees. In a test with a fixed passing score a rescoring will result in the change of some pass/fail decisions. The number of changes depends on: the reliability of the rating system, the number of raters, the variability in examinee abilities, the proportion of examinees that initially pass, and the policy used to incorporate the rescore into the pass/fail decision. In this study, we provide a model that facilitates the evaluation of various rescoring strategies. We consider and compare the efficiency of three rescoring strategies: (1) rescore everyone, (2) rescore failures only, and (3) rescore within some range of the passing cut off. These rescoring strategies are evaluated by direct simulation. Additionally we consider the optimal allocation of resc ores where the probability someone asks to be rescored is inversely proportional to the distance from their initial score to the cutscore. This allocation is evaluated via numerical integration. A further generalization of the basic model is also considered in which a test is comprised of a mixture of objectively and subjectively scored items.
Key words and phrases: Constructed responses, normal linear model, rater reliability, rescoring.