[Paper Review] User Behavior Simulation with Large Language Model based Agents

Lei Wang, Jingsen Zhang+12 more|arXiv (Cornell University)|Jun 5, 2023

Topic Modeling4 citations

TL;DR

This paper proposes a large language model (LLM)-based agent framework that simulates realistic user behaviors across multiple interactive environments, such as recommender systems and social networks, by leveraging LLMs' world knowledge and reasoning capabilities. The method achieves near-human-level behavioral fidelity in zero-shot settings, enabling the study of social phenomena like information cocoons and conformity with high realism.

ABSTRACT

Simulating high quality user behavior data has always been a fundamental problem in human-centered applications, where the major difficulty originates from the intricate mechanism of human decision process. Recently, substantial evidences have suggested that by learning huge amounts of web knowledge, large language models (LLMs) can achieve human-like intelligence. We believe these models can provide significant opportunities to more believable user behavior simulation. To inspire such direction, we propose an LLM-based agent framework and design a sandbox environment to simulate real user behaviors. Based on extensive experiments, we find that the simulated behaviors of our method are very close to the ones of real humans. Concerning potential applications, we simulate and study two social phenomenons including (1) information cocoons and (2) user conformity behaviors. This research provides novel simulation paradigms for human-centered applications.

Motivation & Objective

• To address the limitations of traditional user behavior simulation methods that rely on simplified decision models and real-world data.
• To overcome the 'chicken and egg' problem in simulation by enabling zero-shot user behavior generation using LLMs.
• To create a multi-environment simulation framework that captures interdependencies between user behaviors in diverse contexts.
• To study emergent social phenomena such as information cocoons and conformity through realistic, LLM-driven user agents.
• To establish a novel simulation paradigm for human-centered AI applications using LLM-based agents.

Proposed method

• Design a sandbox environment that emulates real-world user interaction scenarios, including recommender systems and social media platforms.
• Train and deploy LLM-based agents that simulate user decisions using natural language reasoning and world knowledge.
• Implement behavior templates for key actions: buying a movie, navigating pages, searching, leaving, generating feelings, initiating dialogue, and posting on social media.
• Use prompt engineering to guide agents to act in contextually appropriate ways, ensuring consistency with observed user behavior patterns.
• Integrate multi-turn interaction logic to simulate dynamic, evolving user interactions across environments.
• Leverage pre-trained LLMs to enable zero-shot simulation without requiring fine-tuning on real user data.

Figure 1 : a , A brief running process of the simulator. b , The agent framework, which includes a profile module, a memory module, and an action module. c , Key characters of the simulator. Different agents behave in a round-by-round manner based on Pareto distribution, where, in each round, only a

Experimental results

Research questions

• RQ1 Can LLM-based agents generate user behaviors that are indistinguishable from real human behaviors in multi-environment settings?
• RQ2 To what extent can LLM agents simulate complex social phenomena such as information cocoons and conformity without access to real-world training data?
• RQ3 How do LLM-based agents perform in zero-shot simulation compared to traditional simulation methods relying on real data?
• RQ4 Can LLM agents maintain behavioral consistency and contextual relevance across multiple interaction modalities (e.g., watching, chatting, posting)?
• RQ5 What emergent social dynamics can be observed when deploying multiple LLM agents in a shared simulation environment?

Key findings

• The simulated behaviors of LLM-based agents are highly realistic and closely resemble actual human behaviors in both recommender systems and social networks.
• The framework enables zero-shot simulation, meaning it can generate plausible user behaviors without requiring any real-world training data.
• The agents successfully simulate complex social phenomena such as information cocoons and user conformity, demonstrating the emergence of realistic social dynamics.
• Agents exhibit context-aware decision-making, including appropriate responses to social cues, emotional reflections, and targeted content searches.
• The simulation framework supports multi-modal and multi-environment interactions, enabling realistic modeling of cross-platform user behavior.
• The method outperforms traditional simulation techniques that rely on simplified models and real-data dependency, offering broader applicability in human-centered AI research.

Figure 2 : Evaluation on the believability of the simulated user behaviors. a , Evaluation on the recommendation behaviors based on different $(a,b)$ ’s (discrimination capability). b , Evaluation on the recommendation behaviors based on different $N$ ’s (generation capability). c , Evaluation on th