[Paper Review] Empirical Analysis of Predictive Algorithms for Collaborative Filtering

John S. Breese, David Heckerman+1 more|arXiv (Cornell University)|Jan 30, 2013

Data Management and Algorithms11 references4,510 citations

TL;DR

The paper compares various predictive algorithms for collaborative filtering, including correlation-based, vector similarity, and Bayesian methods, across multiple domains and evaluation metrics.

ABSTRACT

Collaborative filtering or recommender systems use a database about user preferences to predict additional topics or products a new user might like. In this paper we describe several algorithms designed for this task, including techniques based on correlation coefficients, vector-based similarity calculations, and statistical Bayesian methods. We compare the predictive accuracy of the various methods in a set of representative problem domains. We use two basic classes of evaluation metrics. The first characterizes accuracy over a set of individual predictions in terms of average absolute deviation. The second estimates the utility of a ranked list of suggested items. This metric uses an estimate of the probability that a user will see a recommendation in an ordered list. Experiments were run for datasets associated with 3 application areas, 4 experimental protocols, and the 2 evaluation metrics for the various algorithms. Results indicate that for a wide range of conditions, Bayesian networks with decision trees at each node and correlation methods outperform Bayesian-clustering and vector-similarity methods. Between correlation and Bayesian networks, the preferred method depends on the nature of the dataset, nature of the application (ranked versus one-by-one presentation), and the availability of votes with which to make predictions. Other considerations include the size of database, speed of predictions, and learning time.

Motivation & Objective

• Evaluate predictive accuracy of different collaborative filtering algorithms.
• Compare correlation, vector-based similarity, and Bayesian methods.
• Assess performance across multiple datasets, protocols, and evaluation metrics.

Proposed method

• Implement and compare correlation-based, vector similarity, and Bayesian method variants for collaborative filtering.
• Use two evaluation metrics: average absolute deviation and ranked list utility.
• Conduct experiments across three application domains, four protocols, and two metrics.

Experimental results

Research questions

• RQ1 Which predictive algorithms (correlation-based, vector similarity, Bayesian) yield higher accuracy in predictive tasks for collaborative filtering across datasets?
• RQ2 How do Bayesian networks with decision trees compare to Bayesian clustering and vector-similarity methods under different evaluation metrics and application settings?
• RQ3 What factors (dataset nature, ranking vs one-by-one presentation, voting availability) influence the preferred method for predictions?

Key findings

• Bayesian networks with decision trees at each node and correlation methods often outperform Bayesian-clustering and vector-similarity methods.
• The preferred method depends on dataset characteristics and application type (ranked vs one-by-one presentation).
• Performance depends on dataset size, prediction speed, and learning time.
• Results vary across problem domains and experimental protocols.