[Paper Review] Pre-training Graph Neural Network for Cross Domain Recommendation

Chen Wang, Yueqing Liang+3 more|arXiv (Cornell University)|Nov 16, 2021

Recommender Systems and Techniques4 citations

TL;DR

This paper proposes PCRec, a pre-training and fine-tuning framework for cross-domain recommendation that leverages self-supervised contrastive learning on a graph neural network (GNN) in the source domain to initialize user and item embeddings in the target domain. By transferring structural information via a pre-trained GNN and fine-tuning with a simple matrix factorization model, PCRec achieves state-of-the-art performance while minimizing bias from the source domain, outperforming existing methods on Recall@20 and MAP@20 by up to 27.6% and 26.4%, respectively.

ABSTRACT

A recommender system predicts users' potential interests in items, where the core is to learn user/item embeddings. Nevertheless, it suffers from the data-sparsity issue, which the cross-domain recommendation can alleviate. However, most prior works either jointly learn the source domain and target domain models, or require side-features. However, jointly training and side features would affect the prediction on the target domain as the learned embedding is dominated by the source domain containing bias information. Inspired by the contemporary arts in pre-training from graph representation learning, we propose a pre-training and fine-tuning diagram for cross-domain recommendation. We devise a novel Pre-training Graph Neural Network for Cross-Domain Recommendation (PCRec), which adopts the contrastive self-supervised pre-training of a graph encoder. Then, we transfer the pre-trained graph encoder to initialize the node embeddings on the target domain, which benefits the fine-tuning of the single domain recommender system on the target domain. The experimental results demonstrate the superiority of PCRec. Detailed analyses verify the superiority of PCRec in transferring information while avoiding biases from source domains.

Motivation & Objective

• Address the data sparsity and cold-start problems in single-domain recommender systems by leveraging cross-domain knowledge transfer.
• Overcome the limitation of existing cross-domain recommendation (CDR) methods that suffer from bias dominance by source domains due to joint training or side-feature reliance.
• Develop a pre-training and fine-tuning paradigm that decouples structural representation learning from target domain prediction to preserve target-specific information.
• Investigate whether a simple fine-tuning model can outperform complex models when initialized with pre-trained GNN embeddings in CDR.
• Design a self-supervised, side-feature-free pre-training scheme suitable for sparse interaction-only data in CDR.

Proposed method

• Pre-train a GNN encoder on the source domain using contrastive self-supervised learning, where positive node pairs are generated via random walk-based subgraph augmentation.
• Use the pre-trained GNN encoder to initialize node embeddings in the target domain, preserving structural inductive bias from the source.
• Fine-tune the target domain recommendation model using a simple matrix factorization (MF) model, which empirically outperforms deeper models like LightGCN.
• Apply a contrastive learning objective to maximize agreement between node representations in two augmented views of the same node’s neighborhood.
• Limit the fine-tuning stage to the target domain only, ensuring that the final predictions are not dominated by source domain biases.
• Evaluate the framework on real-world datasets, comparing variants with different hop counts and fine-tuning architectures to analyze design choices.

Figure 1: An illustration of PCRec model architecture in specific example. The model is mainly composed of three setps. In the pre-training stage, a model with the structural learning of nodes’ embedding is used to learn the source domain’s structure. The target domain is put into the pre-trained mo

Experimental results

Research questions

• RQ1 Can self-supervised pre-training on a source domain’s interaction graph improve recommendation performance on a target domain with limited data?
• RQ2 Does transferring only the pre-trained GNN encoder (without fine-tuning) lead to performance degradation due to source domain bias?
• RQ3 How does the choice of fine-tuning model architecture affect performance when initialized with pre-trained embeddings in cross-domain recommendation?
• RQ4 Does using the pre-trained GNN to generate embeddings for common users (via model transfer) outperform directly copying embeddings from the source domain?
• RQ5 How does the number of hops in neighborhood aggregation during pre-training affect downstream performance?

Key findings

• PCRec achieves the highest performance across all metrics, with a Recall@20 of 0.2756 and MAP@20 of 0.0264, significantly outperforming the next best baseline (LGCN) by 27.6% and 26.4% respectively.
• The variant PCRec-2hop (2-hop neighbors) outperforms PCRec-3hop (3-hop neighbors), indicating that increasing neighborhood distance reduces similarity between positive samples and hinders representation learning.
• Using a simple matrix factorization (MF) model for fine-tuning (PCRec) significantly outperforms more complex models like 1-layer and 3-layer LightGCN, suggesting that complex models may overfit to target data and diminish transferred knowledge.
• The CU-PM method, which uses the pre-trained GNN to generate embeddings for common users, outperforms CU-PE (direct embedding copy) and even the standard LGCN baseline, proving the value of model-based transfer over direct embedding transfer.
• The Pre-Only baseline, which applies the pre-trained GNN without fine-tuning, performs the worst (Recall@20 = 0.0329), confirming that fine-tuning is essential to mitigate source domain bias.
• The ablation study confirms that including non-common users in the fine-tuning stage, while initialized via the pre-trained encoder, improves performance and prevents the target domain from being dominated by source domain information.