assessment of the capability of chatgpt-3.5 in medical physiology examination in an indian medical school

Background: there has been increasing interest in exploring the capabilities of artificial intelligence (ai) in various fields, including education. medical education is an area where ai can potentially have a significant impact, especially in helping students answer their customized questions. in this study, we aimed to investigate the capability of chatgpt, a conversational ai model in generating answers to medical physiology exam questions in an indian medical school.methods: this cross-sectional study was conducted in march 2023 in an indian medical school, deoghar, jharkhand, india. the first mid-semester physiology examination was taken as the reference examination. there were two long essays, five short essay questions (total mark 40), and 20 multiple-choice questions (mcq) (total mark 10). we generated the response from chatgpt (in march 13 version) for both essay and mcq questions. the essay-type answer sheet was evaluated by five faculties, and the average was taken as the final score. the score of 125 students (all first-year medical students) in the examination was obtained from the departmental registery. the median score of the 125 students was compared with the score of chatgpt using mann-whitney u test.results: the median score of 125 students in essay-type questions was 20.5 (q1-q3: 18-23.5) which corresponds to a median percentage of 51.25% (q1-q3: 45-58.75) (p=0.147). the answer generated by chatgpt scored 21.5 (q1-q3: 21.5-22), which corresponds to 53.75% (q1-q3: 53.75-55) (p=0.125). hence, chatgpt scored like that of the students (p=0.4) in essay-type questions. in mcq-type questions, chatgpt answered 19 correctly in 20 questions (score=9.5), and this was higher than the median score of students (6) (q1-q3: 5-6.5) (p<0.0001).conclusion: chatgpt has the potential to generate answers to medical physiology examination questions. it has a higher capability to solve mcq questions than essay-type ones. although chatgpt was able to provide answers that had the quality to pass the examination, the capability of generating high-quality answers for educational purposes is yet to be achieved. hence, its usage in medical education for teaching and learning purposes is yet to be explored.