[Paper Review] Self-Refine: Iterative Refinement with Self-Feedback

Aman Madaan, Niket Tandon+14 more|arXiv (Cornell University)|Mar 30, 2023

Topic ModelingComputer Science207 citations

TL;DR

Self-Refine uses a single large language model to generate an initial output, then provide feedback and refine iteratively, achieving improvements across tasks without extra training.

ABSTRACT

Like humans, large language models (LLMs) do not always generate the best output on their first try. Motivated by how humans refine their written text, we introduce Self-Refine, an approach for improving initial outputs from LLMs through iterative feedback and refinement. The main idea is to generate an initial output using an LLMs; then, the same LLMs provides feedback for its output and uses it to refine itself, iteratively. Self-Refine does not require any supervised training data, additional training, or reinforcement learning, and instead uses a single LLM as the generator, refiner, and feedback provider. We evaluate Self-Refine across 7 diverse tasks, ranging from dialog response generation to mathematical reasoning, using state-of-the-art (GPT-3.5, ChatGPT, and GPT-4) LLMs. Across all evaluated tasks, outputs generated with Self-Refine are preferred by humans and automatic metrics over those generated with the same LLM using conventional one-step generation, improving by ~20% absolute on average in task performance. Our work demonstrates that even state-of-the-art LLMs like GPT-4 can be further improved at test time using our simple, standalone approach.

Motivation & Objective

• Motivate improving LLM outputs via iterative self-feedback inspired by human writing and problem-solving.
• Propose a training-free method that uses the same LLM for generation, feedback, and refinement.
• Demonstrate effectiveness across diverse tasks and analyze the impact of feedback quality and iteration depth.

Proposed method

• Generate initial output with a base LLM.
• Prompt the same LLM to produce actionable feedback on its output.
• Refine the output using the feedback via a refined prompt to the same LLM.
• Iterate feedback and refinement up to a task-specific stopping condition (up to 4 iterations).
• Use few-shot prompts to guide generation, feedback, and refinement without external training.

Figure 1 : Given an input ( \raisebox{-1pt} {\footnotesize{0}\vphantom{a}}⃝ ), \ours starts by generating an output and passing it back to the same model $\mathcal{M}$ to get feedback ( \raisebox{-1pt} {\footnotesize{1}\vphantom{a}}⃝ ). The feedback is passed back to $\mathcal{M}$ , which refines th

Experimental results

Research questions

• RQ1 Can a single LLM improve its own outputs through iterative self-feedback and refinement without additional training?
• RQ2 How does the quality of self-generated feedback affect refinement outcomes?
• RQ3 What is the impact of multiple feedback-refinement iterations on various tasks?
• RQ4 Does self-refinement outperform single-pass generation across diverse domains?

Key findings

• Across 7 tasks, self-refinement yields higher human and automatic preferences versus one-shot generation.
• GPT-4 with self-refine shows notable absolute gains (e.g., Code Optimization from 27.3% to 36.0%; +8.7).
• In preference-based tasks, gains are especially large (e.g., Dialogue Response: GPT-4 from 25.4 to 74.6).
• Constrained Generation benefits significantly due to exploring more outputs after iterative feedback.
• Code-based tasks also improve, with up to 13% absolute gains when using Codex.
• Actionable, specific feedback is crucial for performance; generic or no feedback degrades results.

Figure 10 : Initial web layout generated by our model for a fictional ice cream parlor.