[Paper Review] On Generative Agents in Recommendation

An Zhang, Yuxin Chen+3 more|arXiv (Cornell University)|Oct 16, 2023

Topic ModelingComputer Science3 citations

TL;DR

This paper introduces Agent4Rec, a recommendation simulator that uses LLM-powered generative agents to emulate realistic, autonomous user behavior in personalized recommendation systems. By integrating user profiles, memory modules with emotion-driven reflection, and diverse action modules, Agent4Rec enables high-fidelity simulation of user interactions, demonstrating strong alignment with real user preferences while revealing insights into filter bubbles and causal dynamics in recommendations.

ABSTRACT

Recommender systems are the cornerstone of today's information dissemination, yet a disconnect between offline metrics and online performance greatly hinders their development. Addressing this challenge, we envision a recommendation simulator, capitalizing on recent breakthroughs in human-level intelligence exhibited by Large Language Models (LLMs). We propose Agent4Rec, a user simulator in recommendation, leveraging LLM-empowered generative agents equipped with user profile, memory, and actions modules specifically tailored for the recommender system. In particular, these agents' profile modules are initialized using real-world datasets (e.g. MovieLens, Steam, Amazon-Book), capturing users' unique tastes and social traits; memory modules log both factual and emotional memories and are integrated with an emotion-driven reflection mechanism; action modules support a wide variety of behaviors, spanning both taste-driven and emotion-driven actions. Each agent interacts with personalized recommender models in a page-by-page manner, relying on a pre-implemented collaborative filtering-based recommendation algorithm. We delve into both the capabilities and limitations of Agent4Rec, aiming to explore an essential research question: ``To what extent can LLM-empowered generative agents faithfully simulate the behavior of real, autonomous humans in recommender systems?'' Extensive and multi-faceted evaluations of Agent4Rec highlight both the alignment and deviation between agents and user-personalized preferences. Beyond mere performance comparison, we explore insightful experiments, such as emulating the filter bubble effect and discovering the underlying causal relationships in recommendation tasks. Our codes are available at https://github.com/LehengTHU/Agent4Rec.

Motivation & Objective

• To address the persistent gap between offline evaluation metrics and online performance in recommender systems.
• To develop a scalable, configurable simulation platform that faithfully replicates human-like decision-making in recommendation scenarios.
• To explore whether LLM-empowered generative agents can realistically simulate autonomous, personalized user behavior in recommendation environments.
• To enable new forms of research, including causal discovery and filter bubble simulation, within a controlled and reproducible framework.

Proposed method

• Agent4Rec employs 1,000 LLM-empowered generative agents per simulation scenario, each initialized with real-world user data from datasets like MovieLens-1M.
• Each agent is structured with three core modules: a profile module capturing social traits and historical preferences, a memory module logging factual and emotional memories in natural language, and an action module supporting taste-driven and emotion-driven behaviors.
• The memory module incorporates an emotion-driven reflection mechanism to simulate fatigue, satisfaction, and emotional shifts during interactions.
• Agents interact with a collaborative filtering-based recommender system in a page-by-page manner, simulating real-time user actions such as watching, rating, exiting, and evaluating items.
• The simulator supports both rule-based and algorithmic recommenders, enabling evaluation across diverse recommendation paradigms.
• Agent behavior is grounded in real user data, with LLMs generating responses and actions based on profile, memory, and contextual cues, while maintaining consistency through prompt engineering and output validation.

Experimental results

Research questions

• RQ1 To what extent can LLM-empowered generative agents faithfully simulate the behavior of real, autonomous humans in recommendation systems?
• RQ2 How well do the simulated agents align with real user preferences in terms of taste, diversity, and conformity?
• RQ3 Can Agent4Rec effectively emulate complex phenomena such as the filter bubble effect in recommendation systems?
• RQ4 What causal relationships underlie user behavior and recommendation outcomes, and can they be discovered through simulation?

Key findings

• Agent4Rec demonstrates strong alignment between simulated agent behavior and real user preferences, particularly in taste-driven and emotionally informed actions.
• The simulator successfully reproduces the filter bubble effect, showing how repeated exposure to similar content reinforces narrow preferences.
• Agents exhibit measurable diversity and conformity patterns consistent with real-world user behavior, validating the simulator’s representativeness.
• The memory module with emotion-driven reflection improves behavioral coherence and realism, especially in long-term interaction sequences.
• Despite limitations, Agent4Rec enables the discovery of causal relationships in recommendation dynamics through controlled simulation experiments.
• Occasional hallucinations in LLM outputs—such as fabricating non-existent items or inconsistent ratings—were observed, highlighting the need for fine-tuning to improve simulation stability.