Real world clinical experience of using Brainomix e-CTA software in a medium size acute National Health Service Trust

Abstract Objectives Artificial intelligence (AI) software including Brainomix “e-CTA” which detect large vessel occlusions (LVO) have clinical potential. We hypothesized that in real world use where prevalence is low, its clinical utility may be overstated. Methods In this single centre retrospective service evaluation project, data sent to Brainomix from a medium size acute National Health Service (NHS) Trust hospital between January 3, 2022 and January 3, 2023 was reviewed. 584 intracranial computed tomography angiogram (CTA) datasets were analysed for LVO by e-CTA. The e-CTA output and radiology report were compared to ground truth, defined by a consultant radiologist with fellowship neuroradiology training, with access to subsequent imaging and clinical notes. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated. Results Of 584 cases (45% female, mean age 70 ± 16 years), 9% (n = 50) had LVO. e-CTA had a sensitivity of 0.78 (95% CI 0.64-0.88), specificity of 0.93 (0.9-0.95), PPV of 0.5 (0.42-0.58), and NPV of 0.98 (0.96-0.99). e-CTA had an error rate of 9% (52/584). Erroneous cases were categorized into causes for error. Common causes for false positives included incorrect anatomy (21%, 8/39) and other pathology (13%, 5/39), with several uncategorizable cases (39%, 15/39). Common causes for false negatives included LVO within the terminal internal carotid artery (ICA) (55%, 6/11) and uncategorizable (18%, 2/11). Conclusions We demonstrated that PPV of e-CTA is poor in consecutive cases in a real-world NHS setting. We advocate for local validation of AI software prior to clinical use. Advances in knowledge Common AI errors were due to anatomical misidentification, presence of other pathology, and misidentifying LVO in the terminal ICA.