Simulation-Based Cardiopulmonary Examination Skills Checklist: Assessment of Inter-rater Reliability

Purpose: Experiential learning using high-fidelity simulation is a valuable tool for teaching Doctor of Physical Therapy (DPT) students the cardiopulmonary examination. There are currently no standardized assessment tools to evaluate student performance and provide feedback on these skills. The cardiovascular and pulmonary section of the American Physical Therapy Association recently published recommended competency levels for entry-level physical therapists including examination skills. The primary aim of this study was to create and assess inter-rater reliability of a cardiopulmonary examination skills checklist that aligns with these recommendations and can be used in the simulation environment. Methods: A skills checklist with 6 sections was developed to assess student performance on the cardiopulmonary examination within a high-fidelity simulation experience. Training videos on the cardiopulmonary examination and checklist use were created and given to 3 graders. Seventy-three DPT students performed the cardiopulmonary examination within simulation while being recorded. Three graders then assessed the same 25 randomly selected videos using the checklist. Inter-rater reliability, item % agreement and student performance were analyzed. Results: The overall interclass correlation coefficient (ICC) of the checklist was calculated to be 0.858, demonstrating strong agreement between graders. The ICC for individualized checklist sections ranged from 0.659 to 0.960, demonstrating good to near perfect agreement. The average percent correct achieved by students was 80.48 (6.91%), R = 65% to 93.33%. There were >80% agreement for 55/60 items and >90% agreement for 44/60 items. Conclusion: The checklist provided an effective tool to assess student performance on the cardiopulmonary examination and demonstrated consistent scoring between graders.