Statistica Sinica 23 (2013), 1743-1759

SEQUENTIAL TESTING OF MEASUREMENT ERRORS

IN INTER-RATER RELIABILITY STUDIES

Mei Jin$^{1,3}$, Aiyi Liu$^{2}$, Zhen Chen$^{2}$ and Zhaohai Li$^{1}$

$^1$George Washington University,

$^2$National Institute of Child Health and Human Development and $^3$Capital One

Abstract: Inter-rater reliability is usually assessed by means of the intraclass correlation coefficient. Using two-way analysis of variance to model raters and subjects as random effects, we derive group sequential testing procedures for the design and analysis of reliability studies in which multiple raters evaluate multiple subjects. Compared with the conventional fixed sample procedures, the group sequential test has smaller average sample number. The performance of the proposed technique is examined using simulation studies and critical values are tabulated for a range of two-stage design parameters. The methods are exemplified using data from the Physician Reliability Study for diagnosis of endometriosis.

Key words and phrases: Interim analysis, inter-rater reliability, intraclass correlation coefficient, measurement errors, sample size and power, two-way ANOVA.