Effect of Trainee Performance Data on Standard-Setting Judgments Using the Mastery Angoff Method

ABSTRACT Background  Mastery learning in health professions education requires learners to learn and undergo assessment until they demonstrate a high level of competence. Setting defensible standards is key to accurately assessing educational outcomes in mastery learning. The Mastery Angoff method was proposed recently to set a minimum passing standard (MPS) for mastery learning curricula. However, it is unknown whether prior knowledge of trainee performance affects judges' decisions about setting an MPS using the Mastery Angoff method. Objective  We sought to determine the effect of introducing baseline data about trainee performance on faculty judges' decisions about the Mastery Angoff MPS for a written examination. Methods  We developed a mastery learning curriculum to train internal medicine residents and cardiology fellows about the correct interpretation of inpatient telemetry monitoring. All learners were required to meet or exceed an MPS on a 35-item written examination at the end of training. The MPS was set in 2017 by judges who used the item-based Mastery Angoff method without prior examinee performance information. The judges subsequently reevaluated the test items after receiving baseline data about trainee performance collected during pilot testing. Mastery Angoff MPSs with and without baseline performance data were compared. Results  Twelve judges participated in the standard-setting exercise. The initial MPS was similar to the repeat MPS set after judges received trainee performance data (86.2% versus 86.9%, P = .23). Conclusions  Prior knowledge about medical trainee performance data did not affect MPS as determined by the Mastery Angoff procedure.