Rigging the deck: Selecting good problems for expert-novice card-sorting experiments

A seminal study by chi et al. firmly established the paradigm that novices categorize physics problems by surface features (e.g.,incline, pendulum, projectile motion, etc.),while experts use deep structure (e.g.,energy conservation, newton 2, etc.). yet,efforts to replicate the study frequently fail,since the ability to distinguish experts from novices turns out to be highly sensitive to the problem set being used. exactly what properties of problems are most important in problem sets that discriminate experts from novices in a measurable way? to answer this question,we studied the categorizations by known physics experts and novices using a large,diverse set of problems. this set needed to be large so that we could determine how well experts and novices could be discriminated by considering both small subsets using an exhaustive monte carlo approach and larger subsets using simulated annealing. we found that the number of questions required to accurately classify experts and novices can be surprisingly small so long as the problem set is carefully crafted to be composed of problems with particular pedagogical and contextual features. finally,we found that not only was what you ask (deep structure) important,but also how you ask it (problem context). © 2012 american physical society.