PERFORMANCE OF CHAT GPT ON TURKISH BOARD OF ORTHOPAEDİC SURGERY EXAMINATION (Preprint)

UNSTRUCTURED ABSTRACT Objectives The aim of this study is to evaluate the success of Chat GPT in the Turkish Board of orthopedic surgery examination Materials and Methods Among the written exam questions prepared by TOTEK between 2021 and 2023, questions asking visual information similar to those in the literature and canceled questions were not included and all other questions were taken into consideration. The questions were divided into 19 categories according to topics. Also the questions were divided into 3 categories according to the methods of evaluating information: direct recall of information, ability to comment and ability to use information correctly. Questions were asked separately to Chat GPT 3.5 and 4.0 artificial intelligence applications. All answers given were evaluated appropriately according to this grouping. Visual questions were not asked to Chat GPT due to its inability to perceive visual questions. Only questions answered by the application with the correct choice and explanation were accepted as correct answers. Questions that answered incorrectly by Chat GPT were considered incorrect. Results We eliminated the visual questions of 300 questions in total, and asked the remaining 265 multiple choice questions to the Chat GPT application. It answered 95 (35%) of 265 questions correctly and answered 169 (63%) incorrectly. It was also seen that he could not answer 1 question. It has been observed that the exam success of the Chat GPT application is higher than the subjects, especially in the infection questions (67%). Descriptive findings are shown in table 3, showing that both artificial intelligence models can be effective at different levels on various issues, but predominantly GPT 4 performs better. Conclusion Our study showed that although Chat GPT could not reach the level of passing the Turkish Orthopedics and Traumatology Proficiency Exam but it could reach a certain level of accuracy. Software such as Chat GPT needs to be developed and studied further in order to be useful for orthopedics and traumatology physicians, where evaluation of radiological images and physical examination are very important.