Task Accuracy Hides Domain Leakage: A Diagnostic Framework for EEG Domain Adaptation

Background and Objective: Cross-site variability in electroencephalography (EEG) has motivated substantial work on domain adaptation (DA), but evaluation remains task-centric: a method is called invariant if its target-site accuracy is high. A model can instead reach high target accuracy by exploiting domain identity as a feature, by encoding domain nonlinearly so surface metrics miss it, or by entangling task and domain so tightly that any post-hoc correction degrades both. Task accuracy cannot separate these regimes from genuine invariance. Our objective is a procedure that can, using representation-level measurements.Methods: Six probes run on frozen embeddings: a task probe; a multi-capacity domain probe; MMD and CORAL scored against a random-split null; an Invariance Robustness Curve from a frozen-encoder gradient-reversal sweep; and two new probes, a subspace decomposition probe measuring principal angles between task and domain class-mean subspaces, and a conditional domain probe separating label-correlation confounds from nuisance leakage. A priority-ordered decision procedure maps each probe signature to one of five failure-mode categories or to Unclassified. Probes report balanced accuracy and macro-F1 with 95% bootstrap confidence intervals; every assignment carries a CI-stability flag. We apply it to nine EEG backbones on DEAP under cross-site shift (Twente vs. Geneva), giving 18 cells, with ±5% and ±10% threshold sensitivity analysis. Results: No backbone is genuinely invariant; the closest retain linear domain accuracy 0.739. Thirteen of 18 cells are task-domain entangled; eight are CI-stable, with residual domain accuracy above 0.90 once the task subspace is projected out. Adversarial gradient-reversal training yields a CI-stable, task-domain-entangled model whose domain identity survives at the ceiling, not an invariant one, and an MMD objective raises task-domain subspace alignment to cos²_max = 1.00 instead of reducing it. Cluster scores on 2-D UMAP projections miss this leakage, averaging a domain silhouette of +0.059 across the task-domain-entangled cells. Conclusions: Task-side metrics cannot support claims of domain invariance in EEG. A multi-capacity domain probe and a residual domain accuracy after projecting out the task subspace separate three regimes task accuracy leaves indistinguishable. We recommend both as reporting defaults for cross-site EEG deployment, where site-level shortcuts would degrade downstream decisions. The accompanying Python toolkit is publicly available.