Assessing pictograph recognition: A comparison of crowdsourcing and traditional survey approaches

Background: compared to traditional methods of participant recruitment,online crowdsourcing platforms provide a fast and low-cost alternative. amazon mechanical turk (mturk) is a large and well-known crowdsourcing service. it has developed into the leading platform for crowdsourcing recruitment. objective: to explore the application of online crowdsourcing for health informatics research,specifically the testing of medical pictographs. methods: a set of pictographs created for cardiovascular hospital discharge instructions was tested for recognition. this set of illustrations (n=486) was first tested through an in-person survey in a hospital setting (n=150) and then using online mturk participants (n=150). we analyzed these survey results to determine their comparability. results: both the demographics and the pictograph recognition rates of online participants were different from those of the in-person participants. in the multivariable linear regression model comparing the 2 groups,the mturk group scored significantly higher than the hospital sample after adjusting for potential demographic characteristics (adjusted mean difference 0.18,95% ci 0.08-0.28,p<.001). the adjusted mean ratings were 2.95 (95% ci 2.89-3.02) for the in-person hospital sample and 3.14 (95% ci 3.07-3.20) for the online mturk sample on a 4-point likert scale (1=totally incorrect,4=totally correct). conclusions: the findings suggest that crowdsourcing is a viable complement to traditional in-person surveys,but it cannot replace them. © jinqiu kuang,lauren argo,greg stoddard,bruce e bray,qing zeng-treitler.