Demographic bias in machine learning: measuring transference from dataset bias to model predictions

As artificial intelligence (AI) systems increasingly influence critical decisions in society, ensuring fairness and avoiding bias have become pressing challenges. This dissertation investigates demographic bias in machine learning, with a particular focus on measuring how bias transfers from datasets to model predictions. Using Facial Expression Recognition (FER) as a primary case study, we develop novel metrics and methodologies to quantify and analyze bias at both the dataset and model levels. The thesis makes several key contributions to the field of algorithmic fairness. We propose a comprehensive taxonomy of types of dataset bias and metrics available for each type. Through extensive evaluation on FER datasets, we demonstrate the effectiveness and limitations of these metrics in capturing different aspects of demographic bias. Additionally, we introduce DSAP (Demographic Similarity from Auxiliary Profiles), a novel method for comparing datasets based on their demographic properties. DSAP enables interpretable bias measurement and analysis of demographic shifts between datasets, providing valuable insights for dataset curation and model development. Our research includes in-depth experiments examining the propagation of representational and stereotypical biases from datasets to FER models. Our findings reveal that while representational bias tends to be mitigated during model training, stereotypical bias is more likely to persist in model predictions. Furthermore, we present a framework for measuring bias transference from datasets to models across various bias induction scenarios. This analysis uncovers complex relationships between dataset bias and resulting model bias, highlighting the need for nuanced approaches to bias mitigation. Throughout the dissertation, we emphasize the importance of considering both representational and stereotypical biases in AI systems. Our work demonstrates that these biases can manifest and propagate differently, necessitating tailored strategies for detection and mitigation. By providing robust methodologies for quantifying and analyzing demographic bias, this research contributes to the broader goal of developing fairer and more equitable AI systems. The insights and tools presented here have implications beyond FER, offering valuable approaches for addressing bias in various machine learning applications. This dissertation paves the way for future work in algorithmic fairness, emphasizing the need for continued research into bias measurement, mitigation strategies, and the development of more inclusive AI technologies.