EEG-based model and antidepressant response

In a recent article, Wu et al.1 presented an electroencephalogram (EEG)-based prediction model for antidepressant treatment response1. Here, we point to limitations in the methods used to define response and to validate the prediction model; specifically that change from baseline Hamilton depression rating scale (HAMD) scores needs to take into account the nonlinearity of response, and that the validation analysis transposed the predictor and the outcome.Wu et al. trained their prediction model on data from the treatment arm of the EMBARC study (n = 109) to predict change in HAMD scores. However, the relationship between HAMD scores at follow-up when compared against baseline is typically extremely nonlinear. This nonlinear relationship is caused by patients with more severe depression (higher HAMD) having much larger improvement than those with less severe depression (lower HAMD). This implies that change from baseline for HAMD is not an appropriate measure. A preferred solution is to fit a proportional odds ordinal logistic model with nonlinear adjustment for baseline scale, using for example restricted cubic splines, with raw HAMD at a fixed time point since baseline as the dependent variable. A more detailed discussion of issues relating with analyzing change from baseline in randomized controlled trials has been presented elsewhere2. Wu et al.1 also tested the model on data from the placebo arm. Finding that the model did not predict improvement, they argued that this suggests the predicted effect is specific to selective serotonin reuptake inhibitors (SSRIs). However, an alternative explanation is that the model predictions did not generalize well due to overfitting on the training dataset. Proof of the efficacy of the model hinges on the test of the “antidepressant-predictive signature” in a second independent sample (n = 72). However, in this second dataset, the authors did not test whether the model predicted treatment response. Instead, they tested whether treatment responder status predicted the model prediction of HAMD change. We reanalysed the data to test whether the model predicted treatment response. Data points were estimated from Figure 4 of the paper by Wu et al. using WebPlotDigitizer, resulting in recovery of 71/72 data points. A receiver operating characteristic (ROC) curve was estimated using the pROC package3 in R4, yielding an area under the curve (AUC) of 0.67 (95% CI 0.53-0.81) (Figure 1). This case illustrates the general point that validation of predictive models should target the prediction of interest. The model validation by Wu et al. may be said to demonstrate a fallacy of transposition, as they used treatment responder status to predict the model predicted values. Model performance is barely above chance. Together with the uncertainty with respect to what is being predicted when change scores are the target, this suggests that the time has not yet arrived for clinically useful EEG markers for treatment selection in depression.