EXPLORE: learning interpretable rules for patient-level prediction

Abstract Objective We investigate whether a trade-off occurs between predictive performance and model interpretability in real-world health care data and illustrate how to develop clinically optimal decision rules by learning under constraints with the Exhaustive Procedure for Logic-Rule Extraction (EXPLORE) algorithm. Methods We enhanced EXPLORE’s scalability to enable its use with real-world datasets and developed an R package that generates simple decision rules. We compared EXPLORE’s performance to 7 state-of-the-art model algorithms across 5 prediction tasks using data from the Dutch Integrated Primary Care Information (IPCI) database. Additionally, we characterized EXPLORE’s space of near-optimal models (i.e. Rashomon set) and conducted experiments on incorporating domain knowledge and improving existing models. Results The prediction models developed using LASSO, RandomForest, and XGBoost consistently performed best in terms of AUROC, followed by DecisionTree and EXPLORE. However, the decision rules generated by EXPLORE are much simpler (at most 5 predictors) than the aforementioned. GOSDT-G, IHT, and RIPPER performed worse. Moreover, we demonstrated that EXPLORE’s Rashomon set is very large (1,381 − 20,320 models) with a large variability in both the generalizability and model diversity. We then showed there is a potential to find more clinically optimal decision rules using EXPLORE by incorporating domain knowledge (age/sex and task-specific features) or improving existing models (the CHADS2 score). Conclusions Our study shows that more complex models generally outperform simpler ones, confirming the expected interpretability-performance trade-off, although it varies in strength across prediction tasks. EXPLORE’s ability to learn under constraints is valuable for generating clinically optimal decision rules.