[论文解读] Leveraging Large Language Models for Pre-trained Recommender Systems

Zhixuan Chu, Hongyan Hao+10 more|arXiv (Cornell University)|Aug 21, 2023

Topic Modeling被引用 11

一句话总结

tldr: RecSysLLM 是一个基于大型语言模型的预训练推荐系统，通过数据、训练和推理阶段设计注入领域特定的推荐知识，同时保留 LLM 的推理能力。

ABSTRACT

Recent advancements in recommendation systems have shifted towards more comprehensive and personalized recommendations by utilizing large language models (LLM). However, effectively integrating LLM's commonsense knowledge and reasoning abilities into recommendation systems remains a challenging problem. In this paper, we propose RecSysLLM, a novel pre-trained recommendation model based on LLMs. RecSysLLM retains LLM reasoning and knowledge while integrating recommendation domain knowledge through unique designs of data, training, and inference. This allows RecSysLLM to leverage LLMs' capabilities for recommendation tasks in an efficient, unified framework. We demonstrate the effectiveness of RecSysLLM on benchmarks and real-world scenarios. RecSysLLM provides a promising approach to developing unified recommendation systems by fully exploiting the power of pre-trained language models.

研究动机与目标

• Motivate the integration of large language models (LLMs) into unified, pre-trained recommender systems to leverage reasoning and external knowledge.
• Propose RecSysLLM, a pre-trained recommendation model that preserves LLM capabilities while incorporating recommendation-domain knowledge.
• Develop data, training, and inference phases that align tabular recommendation data with LLM pretraining concepts.
• Demonstrate multitask performance across rating, sequential recommendation, explanations, reviews, and direct recommendations.
• Provide evidence from real-world and benchmark datasets that RecSysLLM improves recommendation quality and generalization.

提出的方法

• Textualize tabular user/item data into natural language to fit LLM pretraining.
• Introduce a masking mechanism that preserves entity integrity and supports entity-, sentence-, and document-level masking.
• Adopt an autoregressive blank infilling objective with spans kept in chronological order to retain inter-entity relationships.
• Extend 2D positional encodings to include inter- and intra-entity positions and employ a dynamic position mechanism during inference.
• Use a parameter-efficient fine-tuning approach (LoRA) on a GLM-based foundation (e.g., GLM-10B/ChatGLM-6B) to inject recommendation-specific knowledge.
• Train with multitask prompts spanning rating, sequential recommendation, explanations, reviews, and direct recommendations to enable zero-shot generalization.

实验结果

研究问题

• RQ1 Can RecSysLLM leverage LLMs' reasoning and background knowledge to improve recommendation performance across multiple tasks?
• RQ2 How does textualizing tabular data and introducing entity-aware masking affect the learning of user-item interactions?
• RQ3 Does parameter-efficient fine-tuning of a pre-trained LLM enable effective adaptation to downstream recommendation tasks with competitive or superior performance?
• RQ4 To what extent can RecSysLLM achieve zero-shot generalization to unseen prompts across various recommendation tasks?
• RQ5 What are the practical benefits of representing items as text rather than IDs in enhancing semantic understanding for recommendations?

主要发现

• RecSysLLM achieves competitive or superior performance on rating, sequential recommendation, explanations, reviews, and direct recommendations, particularly outperforming baselines on unseen prompts in zero-shot settings.
• Textual representation of items and entities enables richer semantic understanding than numeric IDs, contributing to improved recommendations.
• The model demonstrates strong zero-shot generalization, surpassing prompt-based baselines like P5 on unseen prompts across several tasks.
• Prompt design and the generative power of LLMs substantially boost explanation generation and review summarization performance.
• Experiments on real-world data (Alibaba/Alipay Chinese dataset) and Amazon-English datasets show RecSysLLM’s effectiveness and practical applicability.