methodology for online reliability studies: a primer for orthopedic surgeons

In orthopedic surgery, there is an increasing number of papers about online studies on the reliability of classification systems. useful classification systems need to be reliable and valid. measurement of validity can be variable and is prone to observer bias. these online collaboratives derive adequate power to study reliability by having a large group of trained surgeons review a small number of cases instead of the “classic” reliability studies in which a small number of observers evaluate many cases. large online interobserver studies have advantages (i.e., less than 15 minutes to complete the survey, the ability to randomize, and the ability to study factors associated with reliability, accuracy, or decision-making). this ‘handbook’ paper gives an overview of current methods for online reliability studies. we discuss the study design, sample size calculation, statistical analyses of results, pitfalls, and limitations of the study design.