evaluating inter-rater reliability: transitioning to a single rater for marking modified essay questions in undergraduate medical education

Modified essay questions (meqs) are often included in high-stakes examinations to assess higher-order cognitive skills. if the marking guides for meqs are inadequate, this can lead to inconsistencies in marking. to ensure the reliability of meqs as a subjective assessment tool, candidates’ responses are typically evaluated by two or more assessors. previous studies have examined the impact of marker variance. current study explores the possibility of assigning a single assessor to mark the students' performances in meq based on statistically drawn evidence in the clinical phase of the mbbs program at a private medical school in malaysia. a robust evaluation method was employed to determine whether to continue with two raters or shift to a single-rater scheme for meqs, using the discrepancy-agreement grading (dag) system for evaluation. a low standard deviation was observed across all 11 pairs of scores, with insignificant t-statistics (p>0.05) in 2 pairs (18.18%) and significant t-statistics (p<0.05) in 9 pairs (81.81%). the intraclass correlation coefficient (icc) results were excellent, ranging from .815 to .997, all with p<0.001. regarding practical effect size (cohen’s d), 1 pair (9.09%) was categorized as having a strong effect size (>0.8), 7 pairs (63.63%) as having a moderate effect size (0.5-<0.8), and 3 pairs (27.27%) as having a weak effect size (0.2-<0.5). the data analysis suggests that it is feasible to consider marking meq items by a single assessor without negatively impacting the reliability of the meq as an assessment tool.