[Paper Review] CoLLM: Integrating Collaborative Embeddings into Large Language Models for Recommendation

Yang Zhang, Fuli Feng+4 more|arXiv (Cornell University)|Oct 30, 2023

Recommender Systems and Techniques4 citations

TL;DR

CoLLM proposes a novel framework that integrates collaborative information from user-item interactions into large language models (LLMs) for recommendation by mapping external collaborative embeddings into the LLM's input token space. By decoupling collaborative modeling from the LLM via a learnable mapping module, CoLLM enhances performance in both warm and cold start scenarios without modifying the LLM, achieving state-of-the-art results on benchmark datasets.

ABSTRACT

Leveraging Large Language Models as Recommenders (LLMRec) has gained significant attention and introduced fresh perspectives in user preference modeling. Existing LLMRec approaches prioritize text semantics, usually neglecting the valuable collaborative information from user-item interactions in recommendations. While these text-emphasizing approaches excel in cold-start scenarios, they may yield sub-optimal performance in warm-start situations. In pursuit of superior recommendations for both cold and warm start scenarios, we introduce CoLLM, an innovative LLMRec methodology that seamlessly incorporates collaborative information into LLMs for recommendation. CoLLM captures collaborative information through an external traditional model and maps it to the input token embedding space of LLM, forming collaborative embeddings for LLM usage. Through this external integration of collaborative information, CoLLM ensures effective modeling of collaborative information without modifying the LLM itself, providing the flexibility to employ various collaborative information modeling techniques. Extensive experiments validate that CoLLM adeptly integrates collaborative information into LLMs, resulting in enhanced recommendation performance. We release the code and data at https://github.com/zyang1580/CoLLM.

Motivation & Objective

• To address the limitation of existing LLM-based recommendation methods that underutilize collaborative information from user-item interactions.
• To improve recommendation performance in warm-start scenarios where collaborative patterns are strong, without compromising cold-start capability.
• To enable flexible integration of various collaborative modeling techniques into LLMs without altering the LLM architecture.
• To maintain high information compression efficiency by avoiding tokenization redundancy from direct user/item token injection.
• To develop a scalable and effective method for incorporating low-rank collaborative signals into LLMs via external modeling and embedding mapping.

Proposed method

• CoLLM uses a pre-trained collaborative model (e.g., matrix factorization or LightGCN) to extract collaborative embeddings from user-item interaction data.
• These collaborative embeddings are projected into the LLM’s input token embedding space via a learnable mapping layer (CIE module), enabling the LLM to attend to collaborative signals.
• The method preserves the LLM’s original architecture by externalizing collaborative modeling, avoiding direct tokenization of user/item IDs.
• A two-stage fine-tuning strategy is employed: first, LoRA is fine-tuned on text-only data to learn recommendation tasks; second, the CIE mapping layer is fine-tuned while keeping the collaborative model fixed.
• The framework supports various collaborative modeling techniques and maintains low-rank representation to reduce redundancy and improve compression efficiency.
• The CIE module enables end-to-end training where collaborative signals are dynamically aligned with textual inputs in the LLM’s attention mechanism.

Experimental results

Research questions

• RQ1 Can collaborative information from user-item interactions be effectively integrated into LLMs without modifying their architecture?
• RQ2 Does externalizing collaborative modeling improve recommendation performance in warm-start scenarios while preserving cold-start capability?
• RQ3 How does the proposed mapping mechanism compare to directly injecting user/item tokens into the LLM’s input space?
• RQ4 What is the impact of different fine-tuning strategies on CoLLM’s performance across warm and cold user scenarios?
• RQ5 Can CoLLM maintain high performance across diverse LLMs and recommendation datasets with minimal architectural changes?

Key findings

• CoLLM outperforms existing LLMRec methods on both warm and cold user scenarios, achieving SOTA results on ML-1M and Amazon-Book datasets.
• The ablation study shows that removing the CIE module (w/o CIE) leads to a significant performance drop, confirming its critical role in collaborative signal integration.
• Directly injecting user and item tokens (w/ UI-token) degrades performance compared to both the baseline and CoLLM, due to increased tokenization redundancy and reduced compression efficiency.
• The two-step tuning strategy (first text-only, then CIE fine-tuning) yields better performance than one-step tuning, especially in cold-start scenarios, highlighting the importance of initial task-specific learning.
• Tuning the collaborative model in the CIE module (T1, T2) further improves performance but at the cost of increased training time and computational overhead.
• On ML-1M, CoLLM achieves an AUC of 0.7295 and UAUC of 0.6875, with T2 achieving the highest AUC (0.7418), demonstrating the effectiveness of full CIE fine-tuning.