Improving Real-Time Concept Drift Detection using a Hybrid Transformer-Autoencoder Framework

Abstract In applied machine learning, concept drift, which is either gradual or abrupt changes in data distribution, can significantly reduce model performance. Typ- ical detection methods like statistical tests or reconstruction-based models are generally reactive and not very sensitive in early detection. Our study pro- poses a hybrid framework consisting of Transformers and Autoencoders to model complex temporal dynamics and provide online drift detection. We create a distinct Trust Score methodology, which includes signals on (1) statistical and reconstruction-based drift metrics (more specifically, PSI, JSD, Transformer-AE error, (2) prediction uncertainty, (3) rules violations, and (4) trend of classi- fier error) aligned with the combined metrics defined by the Trust Score. Using a time-sequenced airline passenger data set with synthetic drift, our proposed model allows for a better detection of drift using as a whole and at different detec- tions thresholds for both sensitivity and interpretability compared to baseline methods and provides a strong pipeline for drift detection in real time for applied machine learning. We evaluated performance using a time-sequenced airline pas- senger dataset having the gradually injected stimulus of drift in expectations, e.g., permuted ticket prices in later batches, broken into 10 time segments [1]. In the data, our results support that the Transformation-Autoencoder detected drift earlier and with more sensitivity than the autoencoders commonly used in the literature, and provided improved modelling above more error rates and log- ical violations. Therefore, a robust framework was developed to reliably monitor concept drift.