[Paper Review] When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs

Ryo Kamoi, Yusen Zhang+3 more|arXiv (Cornell University)|Jun 3, 2024

Legal Education and Practice Innovations4 citations

TL;DR

This paper critically surveys self-correction in large language models (LLMs), identifying conditions under which LLMs can reliably correct their own mistakes. It finds that self-correction fails in general tasks with in-context learning but succeeds when using reliable external feedback or large-scale fine-tuning, highlighting feedback generation as the key bottleneck.

ABSTRACT

Self-correction is an approach to improving responses from large language models (LLMs) by refining the responses using LLMs during inference. Prior work has proposed various self-correction frameworks using different sources of feedback, including self-evaluation and external feedback. However, there is still no consensus on the question of when LLMs can correct their own mistakes, as recent studies also report negative results. In this work, we critically survey broad papers and discuss the conditions required for successful self-correction. We first find that prior studies often do not define their research questions in detail and involve impractical frameworks or unfair evaluations that over-evaluate self-correction. To tackle these issues, we categorize research questions in self-correction research and provide a checklist for designing appropriate experiments. Our critical survey based on the newly categorized research questions shows that (1) no prior work demonstrates successful self-correction with feedback from prompted LLMs, except for studies in tasks that are exceptionally suited for self-correction, (2) self-correction works well in tasks that can use reliable external feedback, and (3) large-scale fine-tuning enables self-correction.

Motivation & Objective

• To address the lack of consensus on whether LLMs can self-correct their own mistakes.
• To identify and categorize research questions in self-correction research to improve experimental design.
• To evaluate the effectiveness of self-correction under different feedback sources: in-context learning, external tools/knowledge, and fine-tuning.
• To provide a checklist for designing fair and valid self-correction experiments.
• To clarify when self-correction is effective and when it fails, based on empirical analysis of prior work.

Proposed method

• Categorizes research questions in self-correction into three types: intrinsic self-correction (in-context learning), self-correction with external feedback, and self-correction via fine-tuning.
• Analyzes 28 prior studies across reasoning, coding, and question-answering tasks to evaluate self-correction performance under different feedback sources.
• Identifies flawed evaluation practices in prior work, such as unfair comparisons and lack of strong baselines.
• Proposes a checklist for designing valid experiments, including proper baseline comparisons and reporting of negative results.
• Evaluates feedback quality as the primary bottleneck in self-correction, especially for in-context learning.
• Examines the role of task properties, such as decomposability, in enabling effective self-correction.

Figure 1 : Self-correction in three stages: initial response generation, feedback, and refinement.

Experimental results

Research questions

• RQ1 Under what conditions can LLMs self-correct using only in-context learning without external feedback?
• RQ2 How effective is self-correction when reliable external feedback (e.g., from tools or search) is available?
• RQ3 Can large-scale fine-tuning enable robust self-correction, and how does it compare to other methods?
• RQ4 Why do some self-correction frameworks fail despite theoretical promise?
• RQ5 What are the key design flaws in prior self-correction evaluations that lead to overestimation of performance?

Key findings

• No prior work demonstrates successful self-correction with feedback from prompted LLMs in general tasks, indicating a fundamental limitation of in-context learning for self-correction.
• Self-correction works well in tasks where reliable external feedback is available, such as code execution or web search, due to verifiable outputs.
• Large-scale fine-tuning significantly improves self-correction performance, suggesting that model capabilities can be learned rather than inferred.
• Tasks with decomposable responses (e.g., step-by-step reasoning) are more amenable to self-correction, even with in-context learning.
• Many prior studies use unfair evaluation settings, such as overly strong baselines or lack of comparison with strong alternatives, leading to overestimation of self-correction gains.
• Feedback generation is the primary bottleneck in self-correction, especially when relying solely on in-context learning without external supervision.

Figure 2 : LLM self-correction frameworks, categorized by information used for generating feedback and whether they use best-possible initial responses (§ 3.2 ). This figure illustrates representative architectures.