[Paper Review] UltraGCN: Ultra Simplification of Graph Convolutional Networks for Recommendation

Kelong Mao, Jieming Zhu+4 more|arXiv (Cornell University)|Oct 28, 2021

Recommender Systems and TechniquesComputer Science35 references34 citations

TL;DR

UltraGCN proposes skipping explicit multi-layer message passing in GCNs for collaborative filtering, using a constraint-based loss to approximate infinite-layer convergence, yielding strong recommendation performance with substantial training speedups.

ABSTRACT

With the recent success of graph convolutional networks (GCNs), they have been widely applied for recommendation, and achieved impressive performance gains. The core of GCNs lies in its message passing mechanism to aggregate neighborhood information. However, we observed that message passing largely slows down the convergence of GCNs during training, especially for large-scale recommender systems, which hinders their wide adoption. LightGCN makes an early attempt to simplify GCNs for collaborative filtering by omitting feature transformations and nonlinear activations. In this paper, we take one step further to propose an ultra-simplified formulation of GCNs (dubbed UltraGCN), which skips infinite layers of message passing for efficient recommendation. Instead of explicit message passing, UltraGCN resorts to directly approximate the limit of infinite-layer graph convolutions via a constraint loss. Meanwhile, UltraGCN allows for more appropriate edge weight assignments and flexible adjustment of the relative importances among different types of relationships. This finally yields a simple yet effective UltraGCN model, which is easy to implement and efficient to train. Experimental results on four benchmark datasets show that UltraGCN not only outperforms the state-of-the-art GCN models but also achieves more than 10x speedup over LightGCN. Our source code will be available at https://reczoo.github.io/UltraGCN.

Motivation & Objective

• Motivate the need for more efficient GCN-based collaborative filtering in large-scale recommender systems.
• Identify limitations of explicit message passing in LightGCN and related models.
• Propose UltraGCN to approximate infinite-layer graph convolutions via a constraint loss.
• Enable flexible weighting of different relation types while maintaining training efficiency.
• Demonstrate superior accuracy and efficiency on standard benchmark datasets.

Proposed method

• Formulate an ultra-simplified GCN, UltraGCN, that skips explicit multi-layer message passing by approximating the infinite-layer convergence state with a constraint loss.
• Derive a convergence expression e_u = sum_{i in N(u)} beta_{u,i} e_i and maximize cosine similarity via a constraint loss L_C with sigmoid-based optimization.
• Incorporate negative sampling to mitigate over-smoothing, yielding L_C with positive and negative pairs.
• Extend UltraGCN with an item-item graph constraint L_I by selecting top-K similar items per item and weighting pairs with omega_{i,j} to capture item-item relationships.
• Combine losses into L = L_O + lambda L_C + gamma L_I, enabling flexible balancing of user-item and item-item information.
• Discuss model complexity and how UltraGCN compares to MF and LightGCN in terms of efficiency and parameter count.

Experimental results

Research questions

• RQ1 Can explicit message passing in CF-oriented GCNs be replaced by a constraint-based learning objective that approximates the infinite-layer convergence state?
• RQ2 How should different relationship types (user-item, item-item, potentially user-user) be weighted to optimize recommendation performance and training efficiency?
• RQ3 Does incorporating a selectively weighted item-item constraint improve performance without sacrificing scalability?
• RQ4 What are the empirical gains in accuracy and training speed when adopting UltraGCN on standard benchmarks?

Key findings

• UltraGCN achieves state-of-the-art or competitive performance across four datasets, outperforming strongest GCN baselines (e.g., DGCF) by substantial margins (e.g., up to 61.4% Recall@20 and 71.6% NDCG@20 on Amazon-Book compared to DGCF).
• UltraGCN Base and UltraGCN with item-item learning consistently improve over baselines, demonstrating the benefit of the constraint-based approach and selective item-item learning.
• UltraGCN delivers significant training efficiency, with reported speedups up to ~14x over LightGCN and notable reductions in training epochs and time (e.g., 75 epochs to converge, ~45 minutes total on Amazon-Book).
• The model uses a unified BCE loss framework for optimization and shows favorable scalability, with complexity on par with MF when accounting for typical hyperparameters (K, R, embedding size).
• Edge weights and constraint coefficients (beta and omega) provide interpretable, more reasonable representations of user-item and item-item relationships for CF tasks.