Reliability and radiologists’ concordance of artificial intelligence (AI)-calculated Alberta Stroke Program Early CT Score (ASPECTS)

Abstract Background:  ASPECTS was developed for the semi-quantitative assessment of early ischemic changes (EIC) on non-contrast computed tomography (NCCT) in acute ischemic stroke (AIS). Artificial intelligence (AI)-based automated tools for the ASPECT scoring system were developed to automate the diagnosis and improve the agreement with radiologists of AIS. The performance of the automated software compared to physicians should be tested before the software is further used in clinical practice as a tool for clinicians. Objective: To evaluate the agreement with radiologists of an AI-based automated post-processing software for detecting EIC and calculating ASPECTS on NCCT images in AIS patients using a radiologist's assessment as a reference. Materials and Methods: NCCT of AIS patients were retrospectively reviewed (Stroke Fast Track Service July 2022 - December 2023). The complete set of clinical data and imaging data from both baseline and follow-up were analyzed by a radiologist as a reference. Two additional observers provided individual ASPECTS from the baseline NCCT only (observer 1 was a radiologist who independently reviewed only the baseline NCCT with stroke window setting. Observer 2 was a radiologist on service which was from the pool of 20 radiologists onsite and online). Recon&GO Inline ASPECTS software (Somaris X, VA40A, Siemens Healthineers AG, Erlangen, Germany)  was applied. Both ASPECT score analysis and ASPECTS region analysis were evaluated. Positive percent agreement (PPA) and negative percent agreement (NPA) were calculated. Interobserver agreement was assessed using the Cohen's kappa coefficient and the intraclass correlation coefficient (ICC). Results: 111 patients with a mean age of 67.8 years (±11.9), 56 (50.5%) females, a mean National Institute of Health Stroke Scale (NIHSS) score of 14.2 (±8.8), and a mean time to baseline NCCT of 123.9 minutes (±58.7) were included. For dichotomized ASPECTS, the automated software showed lower PPA (14.6% vs. 27.1%) but higher NPA (100.0% vs. 93.7%) than observer 2. For the region-based analysis, both the automated software and observer 2 differed in terms of regional contribution. The automated software showed low PPA but rather high NPA with perfect (100%) NPA in lentiform nucleus and M2. The automated software showed higher agreement with the reference and two observers in deep/central regions than cortical regions. For total ASPECTS, the automated software showed a moderate agreement of total ASPECTS with the reference and observer 1 (ICC = 0.545 and 0.545). Observer 2 showed a poor agreement of total ASPECTS with the reference, observer 1, and the automated software (ICC = 0.349, 0.422, and 0.301, respectively). Conclusion: For total ASPECT score, the agreement of the tested AI software is lower compared to observer 1 obtained by a radiologist using the stroke window on NCCT, but better compared to a pool of radiologists on service with a time limit of 30 minutes to interpret the ASPECT score. When analyzing the ASPECTS regions, there are different advantages for the assessment of the deep regions and the cortical regions. The tested AI software shows higher agreement in deep/central regions than cortical regions. From the result, the tested AI software retains its potential for use in emergency situations, particularly for radiologists with limited experience and limited time to report.