Evaluating the Relative Performance of Collaborative Filtering Recommender Systems

Past work on the evaluation of recommender systems indicates that collaborative filtering algorithms are accurate and suitable for the top-n recommendation task. further, the importance of performance beyond accuracy has been recognised in the literature. here, we present an evaluation framework based on a set of accuracy and beyond accuracy metrics, including a novel metric that captures the uniqueness of a recommendation list. we perform an in-depth evaluation of three well-known collaborative filtering algorithms using three datasets. the results show that the user-based and item-based collaborative filtering algorithms have a high inverse correlation between popularity and diversity and recommend a common set of items at large neighbourhood sizes. the study also finds that the matrix factorisation approach leads to more accurate and diverse recommendations, while being less biased toward popularity.