[Paper Review] Evaluating Recommendation Algorithms by Graph Analysis

Batul J. Mirza, Benjamin J. Keller+1 more|ArXiv.org|Apr 3, 2001

Recommender Systems and Techniques50 references3 citations

TL;DR

This paper introduces a graph-based framework for evaluating recommendation algorithms by analyzing 'jumps'—connections between users and items—focusing on reachability and structural properties rather than predictive accuracy. It demonstrates that path lengths in recommender graphs are typically short (1–2), implying that simpler algorithms can replicate complex ones, and identifies the hammock jump as a key mechanism for maintaining connectivity under varying rating thresholds.

ABSTRACT

We present a novel framework for evaluating recommendation algorithms in terms of the `jumps' that they make to connect people to artifacts. This approach emphasizes reachability via an algorithm within the implicit graph structure underlying a recommender dataset, and serves as a complement to evaluation in terms of predictive accuracy. The framework allows us to consider questions relating algorithmic parameters to properties of the datasets. For instance, given a particular algorithm `jump,' what is the average path length from a person to an artifact? Or, what choices of minimum ratings and jumps maintain a connected graph? We illustrate the approach with a common jump called the `hammock' using movie recommender datasets.

Motivation & Objective

• To develop a complementary evaluation framework for recommendation algorithms that focuses on structural connectivity rather than predictive accuracy.
• To analyze how algorithmic parameters such as minimum ratings and jump width affect graph reachability and connectivity in recommender systems.
• To understand the role of hits-buffs (popular users and items) in reducing average path lengths in user-item graphs.
• To explore whether simpler recommendation algorithms can replicate the connectivity of more complex ones by analyzing jump patterns.
• To provide a foundation for designing efficient, explainable recommendation systems by modeling user-item relationships as random graphs with preferential attachment.

Proposed method

• The framework models recommender systems as bipartite graphs and transforms them into one-mode graphs using a 'jump' operation, such as the 'hammock' jump, to simulate user-to-item connections.
• It employs random graph theory, particularly the Newman-Strogatz-Watts model, to analyze path length and connectivity in the resulting one-mode graphs.
• The hammock jump is defined by a width parameter w, connecting users who share at least w common items, enabling analysis of cluster formation and reachability.
• The paper uses two real-world datasets—MovieLens and EachMovie—to evaluate the model under varying thresholds for minimum ratings and jump width.
• It applies a rewiring model to simulate hits-buffs structures, preferentially linking popular users and items to assess their impact on average path length.
• Theoretical formulas for average path length (l_pp and l_r) are derived from induced graph parameters, enabling quantitative analysis of structural properties.

Experimental results

Research questions

• RQ1 How do algorithmic parameters such as minimum rating thresholds and jump width affect the connectivity and reachability in a recommender system's underlying graph?
• RQ2 What is the average path length between users in a recommender graph, and how does it vary with different jump mechanisms?
• RQ3 To what extent do hits-buffs structures (popular users and items) reduce average path length in the graph?
• RQ4 Can simpler recommendation algorithms replicate the connectivity of more complex ones, and under what conditions?
• RQ5 At what point does the graph become disconnected (shattered) as the minimum rating threshold or jump width is varied?

Key findings

• The average path length in the one-mode graph formed by the hammock jump is consistently between 1 and 2, indicating that most users are connected via very short paths.
• For small rewiring probabilities, preferential rewiring has little effect on path length, but for high probabilities, hits-buffs structures significantly reduce path length.
• The graph remains connected under reasonable thresholds for minimum ratings and hammock width, suggesting that even sparse datasets can maintain strong connectivity.
• The results suggest that algorithms attempting to find longer paths are often equivalent to simpler ones, implying that computational complexity may not be justified by structural gains.
• The framework reveals that most recommendation algorithms implicitly rely on hammock-like jumps, even if not explicitly designed as such.
• The study indicates that the choice of algorithm is highly dataset-dependent, and simpler mechanisms may be more efficient than complex ones with similar connectivity outcomes.