Assessing Calibration and Bias of a Deployed Machine Learning Malnutrition Prediction Model within a Large Healthcare System

Abstract Introduction Malnutrition is a frequently underdiagnosed condition leading to increased morbidity, mortality and healthcare costs. The Mount Sinai Health System (MSHS) deployed a machine learning model (MUST-Plus) to detect malnutrition upon hospital admission. However, in diverse patient groups a poorly calibrated model may lead to misdiagnosis, exacerbating health care disparities. We explored the model’s calibration across different variables and methods to improve calibration. Methods Data from adult (age > 18) patients admitted to 5 MSHS hospitals from September 20, 2020 - December 31, 2021 were analyzed. We compared MUST-Plus prediction to the registered dietitian’s formal assessment. We assessed calibration following the hierarchy of weak, moderate, and strong calibration. We tested statistical differences in intercept and slope by bootstrapping with replacement. Results We included 49,282 patients (mean age = 66.0). The overall calibration intercept was − 1.25 (95% CI: -1.28, -1.22), and slope was 1.55 (95% CI: 1.51, 1.59). Calibration was not significantly different between White and Black patients. The calibration intercept was significantly different between male and female patients. Both calibration intercepts and slopes were statistically different between 2021 and 2022. Recalibration improved calibration of the model across race, gender, and year. Discussion The calibration of MUST-Plus underestimates malnutrition in females compared to males, but demonstrates similar calibration slope, suggesting similar distributions of risk estimation. Recalibration is effective at reducing miscalibration across all patient subgroups. Continual monitoring and timely recalibration can improve model accuracy.