How AI Responds to Obstetric Ultrasound Questions and Analyzes and Explains Obstetric Ultrasound Reports: ChatGPT-3.5 vs. Microsoft Copilot in Bing

Abstract Objectives: To evaluate and compare the accuracy and consistency of answers to obstetric ultrasound questions and analysis of obstetric ultrasound reports using publicly available ChatGPT-3.5 and Microsoft Copilot in Bing (Copilot). Methods: Twenty questions related to obstetric ultrasound were answered and 110 obstetric ultrasound reports were analyzed by both ChatGPT-3.5 and Copilot, with each question and report being posed three times to them at different times. The accuracy and consistency of each response to twenty questions and each analysis result in the report were evaluated and compared. Results: In answering twenty questions, ChatGPT-3.5 outperformed Copilot in both accuracy (95.0% vs. 80.0%) and consistency (90.0% vs. 75.0%). When analyzing obstetric ultrasound reports, two models performed similarly in accuracy and consistency, and can provide recommendations. The overall accuracy and consistency of ChatGPT-3.5 and Copilot were 83.86%, 87.30% vs 77.51%, 90.48%, respectively. However, in detecting abnormal amniotic fluid index, ChatGPT-3.5 was superior to Copilot (accuracy 87.50% vs. 66.67%, P < 0.05). Conclusion: While ChatGPT-3.5 and Copilot can provide valuable explanations on obstetric ultrasound and interpret most obstetric ultrasound reports accurately, neither model consistently answered all questions correctly or with complete consistency, the supervision of physician is crucial in the use of these models.