[Paper Review] Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender System

Yunfan Gao, Tao Sheng+4 more|arXiv (Cornell University)|Mar 25, 2023

Topic ModelingComputer Science64 citations

TL;DR

Chat-Rec augments recommender systems with LLMs via in-context prompts to enable multi-round, interactive, and explainable recommendations, including cross-domain and cold-start handling.

ABSTRACT

Large language models (LLMs) have demonstrated their significant potential to be applied for addressing various application tasks. However, traditional recommender systems continue to face great challenges such as poor interactivity and explainability, which actually also hinder their broad deployment in real-world systems. To address these limitations, this paper proposes a novel paradigm called Chat-Rec (ChatGPT Augmented Recommender System) that innovatively augments LLMs for building conversational recommender systems by converting user profiles and historical interactions into prompts. Chat-Rec is demonstrated to be effective in learning user preferences and establishing connections between users and products through in-context learning, which also makes the recommendation process more interactive and explainable. What's more, within the Chat-Rec framework, user's preferences can transfer to different products for cross-domain recommendations, and prompt-based injection of information into LLMs can also handle the cold-start scenarios with new items. In our experiments, Chat-Rec effectively improve the results of top-k recommendations and performs better in zero-shot rating prediction task. Chat-Rec offers a novel approach to improving recommender systems and presents new practical scenarios for the implementation of AIGC (AI generated content) in recommender system studies.

Motivation & Objective

• Motivate addressing interactivity and explainability gaps in traditional recommender systems.
• Propose a prompt-based interface that leverages LLMs to learn user preferences from context.
• Enable cross-domain recommendations by transferring user preferences across item domains.
• Address cold-start challenges for new items using external information integration.
• Demonstrate effectiveness on real-world data for top-k recommendations and zero-shot rating prediction.

Proposed method

• Bridge traditional recommenders and LLMs by converting user profiles and history into prompts that guide ChatGPT-based recommendations.
• Use an input-driven prompt constructor that aggregates user history, profile, query, and dialogue history to produce tailored prompts.
• Allow LLMs to filter and re-rank a candidate set through in-context learning and reasoning.
• Handle cold-start by incorporating external item information to generate embeddings for new items and retrieve similar items.
• Demonstrate cross-domain transfer by enabling the LLM to recommend non-movie items based on movie preferences.
• Evaluate top-k recommendation and zero-shot rating prediction on MovieLens 100K with multiple GPT-3.5 family models.

Figure 1 : Overview of Chat-Rec . The left side shows a dialogue between a user and ChatGPT. The middle side shows the flowchart to how Chat-Rec links traditional recommender systems with conversational AI such as ChatGPT. The right side describes the specific judgment in the process.

Experimental results

Research questions

• RQ1 Can LLMs augmented with in-context learning improve top-k recommendation performance without additional training?
• RQ2 How does prompt design impact the quality of interactive, explainable recommendations?
• RQ3 Can the system support cross-domain and cold-start recommendations effectively?
• RQ4 What is the role of the candidate set and its ordering in the final ranking produced by the LLM?
• RQ5 Do different LLM backbones (GPT-3.5 variants) differ significantly in recommendation and rating-prediction tasks?

Key findings

• Chat-Rec improves top-k recommendation metrics over LightGCN on MovieLens 100K across tested GPT-3.5 models.
• text-davinci-003 achieves the best top-k results with precision 0.3240, recall 0.1404, and NDCG 0.3802.
• In rating prediction, text-davinci-003 yields the best RMSE 0.785 and MAE 0.593 among tested models.
• Ablation shows prompt design and the inclusion of the top-1 baseline background significantly affect NDCG, with temperature and prompt order also impacting performance.
• Chat-Rec can significantly re-rank and refine a large candidate set, improving relevance without explicit recommender training.

Figure 2 : Case study of interactive recommendation. It shows two conversations between different users and LLM . Where the user profile and historical users are converted into corresponding prompts for personalized recommendations, but the input of this part of the prompts is not visible to the use