[Paper Review] TransGNN: Harnessing the Collaborative Power of Transformers and Graph Neural Networks for Recommender Systems

Peiyan Zhang, Yuchen Yan+5 more|arXiv (Cornell University)|Aug 28, 2023

Advanced Graph Neural Networks4 citations

TL;DR

TransGNN proposes a hybrid model that alternately integrates Transformer and Graph Neural Network (GNN) layers to overcome the limited receptive field and noise sensitivity of GNNs in recommender systems. By using structured positional encoding in Transformers and sampling relevant nodes, TransGNN achieves state-of-the-art performance with 30–40% faster training and inference than NGCF, while maintaining low memory and FLOP usage.

ABSTRACT

Graph Neural Networks (GNNs) have emerged as promising solutions for collaborative filtering (CF) through the modeling of user-item interaction graphs. The nucleus of existing GNN-based recommender systems involves recursive message passing along user-item interaction edges to refine encoded embeddings. Despite their demonstrated effectiveness, current GNN-based methods encounter challenges of limited receptive fields and the presence of noisy "interest-irrelevant" connections. In contrast, Transformer-based methods excel in aggregating information adaptively and globally. Nevertheless, their application to large-scale interaction graphs is hindered by inherent complexities and challenges in capturing intricate, entangled structural information. In this paper, we propose TransGNN, a novel model that integrates Transformer and GNN layers in an alternating fashion to mutually enhance their capabilities. Specifically, TransGNN leverages Transformer layers to broaden the receptive field and disentangle information aggregation from edges, which aggregates information from more relevant nodes, thereby enhancing the message passing of GNNs. Additionally, to capture graph structure information effectively, positional encoding is meticulously designed and integrated into GNN layers to encode such structural knowledge into node attributes, thus enhancing the Transformer's performance on graphs. Efficiency considerations are also alleviated by proposing the sampling of the most relevant nodes for the Transformer, along with two efficient sample update strategies to reduce complexity. Furthermore, theoretical analysis demonstrates that TransGNN offers increased expressiveness compared to GNNs, with only a marginal increase in linear complexity. Extensive experiments on five public datasets validate the effectiveness and efficiency of TransGNN.

Motivation & Objective

• Address the limited receptive field and noise from edge-based message passing in GNN-based recommender systems.
• Overcome the over-smoothing problem in deep GNNs that restricts modeling beyond 3-hop neighborhoods.
• Leverage the global attention of Transformers to capture long-range sequential dependencies in user-item interactions.
• Integrate structural information from graphs into Transformers by designing specialized positional encodings.
• Improve computational efficiency for large-scale graphs through targeted node sampling and update strategies.

Proposed method

• Alternate between GNN and Transformer layers to jointly refine user and item embeddings.
• Introduce three types of positional encoding (structural, proximity, and attention-based) to inject graph structure into Transformer layers.
• Apply node sampling to limit self-attention computation to the most relevant nodes, reducing complexity.
• Propose two efficient sample update strategies: dynamic sampling and incremental update to maintain relevance during training.
• Use a dual-stream message passing mechanism where Transformers aggregate long-range context and GNNs refine local, edge-dependent features.
• Design a hybrid architecture where GNNs handle local structural propagation and Transformers perform global, adaptive aggregation.

Experimental results

Research questions

• RQ1 Can combining Transformers and GNNs improve the receptive field of GNNs in collaborative filtering without increasing computational cost?
• RQ2 How can graph structural information be effectively encoded into the self-attention mechanism of Transformers for recommendation tasks?
• RQ3 Can efficient sampling strategies reduce the computational burden of Transformers on large-scale interaction graphs while preserving performance?
• RQ4 Does the alternating architecture of GNN and Transformer layers lead to better representation learning than standalone models?
• RQ5 How does TransGNN compare in efficiency and expressiveness to existing SOTA models like NGCF, SHT, and HyRec?

Key findings

• TransGNN achieves state-of-the-art performance on five public datasets, with Recall@20 of 0.0927 on Yelp, outperforming NGCF (0.0294), HyRec (0.0472), and SHT (0.0651).
• Training time on Yelp is reduced by 30–40% compared to NGCF, with inference time also reduced by 30–40%.
• TransGNN uses 20–30% less GPU memory than SHT and HyRec, with only 1.79GB on the Yelp dataset.
• TransGNN has the lowest FLOPs (3.74×10⁶) among all baselines, indicating superior computational efficiency.
• t-SNE visualization shows that TransGNN produces better-separated clusters than NGCF, HyRec, and SHT, indicating more discriminative representations.
• Theoretical analysis confirms that TransGNN has higher expressiveness than GNNs, with only a marginal increase in linear complexity.