Large language models to infer depression in patients with neurological conditions

Objective To develop and validate a large language model (LLM) prompt capable of ascertaining a patient’s depression from their multiple sclerosis (MS) neurologist’s note and to explore its potential for earlier detection of depression in MS care. Materials and methods This single-center retrospective study analysed prospectively collected electronic health record notes. In phase I, an institutionally secure ChatGPT-4 prompt was iteratively refined to infer the presence of depression using the neurologist’s note and compared with manual annotation of the neurologist’s impression (depression: present, absent, no mention) and patient-reported outcomes (PROs): Hospital Anxiety and Depression Scale or Patient Health Questionnaire-9. In phase II, longitudinal analysis compared timing of depression detection by the prompt and the neurologist across 5 years of notes for 250 patients. Results In phase I (n=278 adults with MS), the LLM prompt detected depression in 60.4% of notes (168/278). When compared with neurologist impression in the clinical notes, the prompt achieved 97.3% sensitivity and 84.4% accuracy. Specificity was more modest (68.3%): when neurologists did not mention depression, the prompt inferred depression based on symptoms, history and medications. When PRO and neurologist impression disagreed, the prompt aligned with PROs 61.9% of the time. In phase II, the LLM inferred depression earlier than the neurologist in 18.8% of patients, at an average of 2.45 (SD 1.54) years earlier. Conclusion The prompt was highly sensitive to neurologist documentation of depression in clinical notes; it inferred both present/treated depression from other note components. Potential applications include quality improvement initiatives aiming to improve depression care on a cohort level.