Assessing Fairness Fragility in ML-Enabled Systems via Mutation-Based Dataset Stress Testing

Context. Fairness of machine learning (ML) systems is increasingly recognized as a critical non-functional requirement in software engineering. In modern ML-enabled systems, where data evolves continuously, and models are frequently retrained, current practices remain fundamentally reactive: fairness is assessed only after training, and violations caused by data drift are detected too late, triggering costly retraining cycles. Objective. We introduce FADE (Fairness Assessment under Dataset Evolution), a framework for preventive fairness stress testing of ML datasets. FADE assesses the structural robustness of a dataset by applying mutation-based perturbations to individual features and measuring their effect on data-level bias symptoms. Features whose fairness profile degrades under small changes are identified as fairness-fragile: latent vulnerabilities often residing in non-sensitive attributes that serve as proxies for discrimination. Method. We validate FADE through a two-phase empirical study on four benchmark datasets spanning socio-economic, criminal justice, healthcare, and financial domains. Results. FADE identified 30 high-risk configurations involving 21 unique fairness-fragile features, 76% of which are non-sensitive. In 93.3% of configurations, data-level fragilities translated into statistically significant fairness variations in downstream models, consistent across Random Forest and XGBoost. We also observe that perturbing a fragile feature can reduce rather than amplify model bias, revealing how deeply certain features are embedded in discriminatory behavior. Conclusion. FADE enables teams to surface fairness risks before model training, supporting a shift from reactive post-hoc assessment toward proactive data quality assurance in ML-enabled systems.