Application of Judgment Analysis to Interrater Comparisons

Abstract

A multiple regression method is presented for comparing the bases of two raters' judgments. This technique, which has been referred to as judgment analysis or policy capturing is described for judgments of two nurses. In the example presented, judgments of future infant performance were derived from the nurse's scoring of infants' behavior on the Brazelton Neonatal Behavioral Assessment Scale. Brazelton dimension scores served as predictors of future performance in a test of differences between the policies (criteria) of the two nurse-raters. Sample data illustrate the technique but do not constitute a direct test of the data since the two nurse's ratings were actually on two different sets of infants. If the ratings had been on the same babies or identical samples of babies, the technique
would have revealed, first, that the two nurses based their judgments primarily on one Brazelton dimension, interactive processes; and second , that one nurse consistently rated the babies' future performance at a higher level than did the other nurse. This technique has potential application to evaluation of rating criteria for training of observers or judges and in other problem solving areas such as conflict resolution.