[Paper Review] Reflexion: Language Agents with Verbal Reinforcement Learning

Noah Shinn, Cassano, Federico+4 more|arXiv (Cornell University)|Mar 20, 2023

Topic Modeling256 citations

TL;DR

Reflexion enables language agents to learn from trial-and-error by generating verbal reflections based on environment feedback, stored in episodic memory to guide future trials, achieving state-of-the-art results on coding, reasoning, and decision-making benchmarks without weight updates.

ABSTRACT

Large language models (LLMs) have been increasingly used to interact with external environments (e.g., games, compilers, APIs) as goal-driven agents. However, it remains challenging for these language agents to quickly and efficiently learn from trial-and-error as traditional reinforcement learning methods require extensive training samples and expensive model fine-tuning. We propose Reflexion, a novel framework to reinforce language agents not by updating weights, but instead through linguistic feedback. Concretely, Reflexion agents verbally reflect on task feedback signals, then maintain their own reflective text in an episodic memory buffer to induce better decision-making in subsequent trials. Reflexion is flexible enough to incorporate various types (scalar values or free-form language) and sources (external or internally simulated) of feedback signals, and obtains significant improvements over a baseline agent across diverse tasks (sequential decision-making, coding, language reasoning). For example, Reflexion achieves a 91% pass@1 accuracy on the HumanEval coding benchmark, surpassing the previous state-of-the-art GPT-4 that achieves 80%. We also conduct ablation and analysis studies using different feedback signals, feedback incorporation methods, and agent types, and provide insights into how they affect performance.

Motivation & Objective

• Motivate a lightweight alternative to traditional RL that leverages verbal feedback rather than weight updates.
• Show that self-reflection and episodic memory improve task performance across decision-making, reasoning, and programming tasks.
• Demonstrate scalability across multiple environments and languages, including a new LeetcodeHardGym benchmark.
• Provide ablations to understand how feedback type and memory influence performance.

Proposed method

• Propose a modular Reflexion framework with three models: Actor (LLM-based generator of text and actions), Evaluator (scores Actor outputs), and Self-Reflection (generates verbal feedback to augment memory).
• Translate environment feedback into verbal reflections and store them in a memory buffer to influence future episodes.
• Use short-term (trajectory) and long-term (self-reflection) memory to condition the Actor's decisions.
• Experiment with various feedback signals (binary, heuristic, self-evaluation) and different task types (decision-making, reasoning, programming).
• Describe the Reflexion algorithm as an iterative loop where each trial yields a trajectory, an Evaluator score, and a self-reflection that is appended to memory for subsequent trials.

Figure 1 : Reflexion works on decision-making 4.1 , programming 4.3 , and reasoning 4.2 tasks.

Experimental results

Research questions

• RQ1 Can verbal self-reflection and episodic memory enable LLM-based agents to learn from sparse/trial-and-error feedback without gradient-based fine-tuning?
• RQ2 How do different feedback signals and memory configurations impact performance across decision-making, reasoning, and programming tasks?
• RQ3 What are the gains from Reflexion on established benchmarks and a new code-generation environment?
• RQ4 Is Reflexion robust across languages and tool usage in programming tasks?

Key findings

• Reflexion improves performance across decision-making, reasoning, and programming tasks compared to strong baselines.
• In AlfWorld decision-making, Reflexion with simple heuristic self-evaluation achieves substantial gains and reaches near-perfect performance over 12 trials.
• On HotPotQA reasoning tasks, Reflexion significantly outperforms baselines, with notable gains over CoT and ReAct variants.
• In HumanEval programming (Python and Rust), Reflexion achieves state-of-the-art pass@1 scores, notably 91.0 on Python HumanEval using GPT-4, surpassing prior SOTA.
• LeetcodeHardGym demonstrates Reflexion’s ability to handle hard coding problems with a self-generated unit-test approach, achieving strong pass@1 results.
• Ablation studies show the importance of self-reflection and memory, with performance drops when either component is removed.

Algorithm 1 Reinforcement via self-reflection