Pro-DOBIE: A Protocol for Domain and Outcome Bias Exploration in Healthcare Machine Learning Models (Preprint)

UNSTRUCTURED Understanding bias, auditing and quality control are key challenges for developing machine learning methods in healthcare.  Models that have been trained on biased data, have the potential to automate decisions that are unfair and inequitable. We propose a protocol for domain and outcome bias exploration (Pro-DOBIE) as a standard framework and reference set of techniques to evaluate performance equitably across subgroups and whether the training data are representative of novel data expected in the model application. The framework consists of three dimensions that address: (i) subgroup performance equitability; (ii) application and training data similarity; and (iii) training data grouping variable congruence.­­ We applied Pro-DOBIE to models developed on two publicly available datasets and a novel data set from MUSC describing mammogram screening uptake. These application examples demonstrate how to use Pro-DOBIE as a standard operating procedure to investigate whether and why the model, the data, or both may be biased.