Evaluating Large Language Models in Interpreting Cervical Cytology

Abstract Background Large language models (LLMs) have shown promise in medical imaging, but their utility in cytology remains underexplored. This study evaluates GPT-5 and Gemini 2.5 Pro for Pap smear interpretation. Methods Digital cervical Pap smear images of 100 cases were obtained from the Hologic Education Site, with Hologic diagnoses considered the gold standard. Representative images were uploaded into GPT-5 and Gemini 2.5 Pro and prompted to provide a diagnosis based on the Third Edition of the Bethesda System for Reporting Cervical Cytopathology. Cases with infectious organisms were assessed using additional images. Concordance was evaluated at exact diagnosis and clinical management groupings, wherein diagnoses with similar management implications were grouped. Sensitivity and specificity for abnormal cytology were also calculated. Results Concordance of both LLMs for exact diagnostic matches were comparable (GPT-5: 47%, Gemini: 48%) and increased to 66% for clinical management grouping. GPT-5 performed best for low-grade squamous intraepithelial lesions (75%), whereas Gemini 2.5 Pro showed the highest concordance in the high-grade squamous intraepithelial lesion (HSIL) category (82%), although this was largely attributable to its strong tendency to overcall cases as HSIL. Sensitivity for detecting abnormal cytology was 74% for GPT-5 and 84% for Gemini, with specificity of 74% and 71%, respectively. GPT-5 better identified glandular lesions, while Gemini detected organisms more accurately (71% vs. 20%). Conclusions Current LLMs demonstrate moderate ability to identify cytologic abnormalities but are not yet reliable for independent Pap smear interpretation. Targeted fine-tuning, prompt optimization, and cytology-specific training could enhance their utility as adjunctive tools in cytology workflows.