A Comparison of Approaches for Mastery Learning Standard Setting

Purpose Defensible minimum passing standards (MPSs) must be used to evaluate learner performance outcomes in health professions education. In this study, the authors compared the results of traditional Angoff and Hofstee standard-setting exercises with the Mastery Angoff and Patient-Safety approaches for central venous catheter (CVC) insertion skills examinations. The authors also evaluated how these standards affected the historical performance of residents who participated in a simulation-based mastery learning (SBML) curriculum for CVC insertion skills. Method In April and May 2015, 12 physicians with expertise in CVC insertion set MPSs for previously published internal jugular (IJ) and subclavian (SC) CVC insertion checklists using Angoff, Hofstee, Mastery Angoff, and Patient-Safety approaches. The resulting MPSs were compared using historical performance of internal medicine and emergency medicine residents who participated in CVC insertion SBML. Results The MPSs were set as follows: Angoff: IJ 91% checklist items correct, SC 90%. Hofstee: IJ 88%, SC 90%. Mastery Angoff: IJ 98%, SC 98%. Patient-Safety: IJ 98%, SC 98%. Based on the historical performance of 143 residents assessed on IJ and SC insertion, applying the 98% MPS would result in additional practice and retesting of 55/123 residents (45%) who had previously passed the IJ examination and 36/130 residents (28%) who had passed the SC examination using the Angoff and Hofstee MPSs. Conclusions The Mastery Angoff and Patient-Safety standard-setting approaches resulted in higher CVC insertion SBML MPSs compared with traditional standard-setting methods. Further study should assess the impact of these more rigorous standards on patient outcomes.