[Paper Review] LLMs cannot find reasoning errors, but can correct them given the error location

Gladys Tyen, Hassan Mansoor+3 more|arXiv (Cornell University)|Nov 14, 2023

Software Engineering Research6 citations

TL;DR

This paper introduces BIG-Bench Mistake, a dataset of logical errors in Chain-of-Thought reasoning traces, and proposes backtracking—a lightweight correction method that leverages mistake location to fix reasoning errors. While LLMs fail at detecting errors independently, backtracking achieves significant accuracy gains (up to +22.59% on held-out tasks) even with reward models as low as 60–70% accuracy, demonstrating that error correction is feasible when mistake locations are provided.

ABSTRACT

While self-correction has shown promise in improving LLM outputs in terms of style and quality (e.g. Chen et al., 2023b; Madaan et al., 2023), recent attempts to self-correct logical or reasoning errors often cause correct answers to become incorrect, resulting in worse performances overall (Huang et al., 2023). In this paper, we show that poor self-correction performance stems from LLMs' inability to find logical mistakes, rather than their ability to correct a known mistake. Firstly, we benchmark several state-of-the-art LLMs on their mistake-finding ability and demonstrate that they generally struggle with the task, even in highly objective, unambiguous cases. Secondly, we test the correction abilities of LLMs -- separately from mistake finding -- using a backtracking setup that feeds ground truth mistake location information to the model. We show that this boosts downstream task performance across our 5 reasoning tasks, indicating that LLMs' correction abilities are robust. Finally, we show that it is possible to obtain mistake location information without ground truth labels or in-domain training data. We train a small classifier with out-of-domain data, which exhibits stronger mistake-finding performance than prompting a large model. We release our dataset of LLM-generated logical mistakes, BIG-Bench Mistake, to enable further research into locating LLM reasoning mistakes.

Motivation & Objective

• To investigate whether LLMs can reliably detect logical errors in their own Chain-of-Thought reasoning traces.
• To address the failure of self-correction in LLMs when correcting reasoning errors, which often degrades performance.
• To propose a backtracking method that improves output correction when mistake locations are provided.
• To evaluate the effectiveness of backtracking using both gold-standard mistake labels and simulated reward models.
• To explore whether small, fine-tuned reward models can assist in mistake finding and transfer to out-of-distribution tasks.

Proposed method

• The authors release BIG-Bench Mistake, a dataset of 2,186 CoT-style reasoning traces generated by PaLM 2-L-Unicorn and annotated with the location of the first logical mistake.
• They decompose self-correction into two components: mistake finding (identifying errors) and output correction (fixing them), isolating the core failure point as mistake finding.
• They propose backtracking as a post-hoc correction technique that re-uses the model’s own reasoning path, modifying only the steps from the mistake location onward.
• Backtracking is framed as a form of 'verbal reinforcement learning' that enables iterative improvement without weight updates.
• They train a small PaLM 2-XS-Otter reward model on four tasks and evaluate its ability to improve mistake-finding accuracy on a held-out task.
• They evaluate backtracking performance using both gold-standard mistake labels and simulated reward models at varying accuracy levels (e.g., 60–70%).

Figure 1: Graph of mistake location accuracies for each prompting method (excluding GPT-4-Turbo which we do not have CoT step-level results for). Blue bars show accuracies on traces with no mistakes, so the model must predict that the trace has no mistake to be considered correct; orange bars show a

Experimental results

Research questions

• RQ1 Can state-of-the-art LLMs reliably detect logical errors in their own Chain-of-Thought reasoning traces without external feedback?
• RQ2 Does providing the location of a reasoning error significantly improve the ability of LLMs to correct their outputs?
• RQ3 Can a small, fine-tuned reward model outperform zero-shot LLMs in mistake-finding tasks, and does it generalize to out-of-distribution tasks?
• RQ4 How effective is backtracking when the mistake location is provided by a low-accuracy reward model rather than a gold standard?
• RQ5 Can backtracking be iteratively applied to correct multiple errors in a single reasoning trace?

Key findings

• LLMs, including state-of-the-art models like PaLM 2-L-Unicorn, struggle significantly with mistake finding, achieving low accuracy even on objective, unambiguous logical errors.
• Backtracking with gold-standard mistake locations improves performance on the original task, with gains of up to +22.59% on the Dyck languages task.
• Backtracking remains effective even with reward models as low as 60–70% accuracy, where the improvement in accuracy delta is already larger than with lower-accuracy models.
• A fine-tuned small reward model achieves absolute accuracy gains of +11.66 to +22.59 on four out of five held-out tasks compared to the zero-shot baseline.
• The reward model shows transferability to out-of-distribution tasks, suggesting potential for generalization, though performance gains are inconsistent across tasks.
• The study identifies mistake finding as the primary bottleneck in self-correction, not output correction, and demonstrates that correction is feasible when error locations are provided.

Figure 2: $\Delta$ accuracy ${}_{\text{\char 51}}$ and $\Delta$ accuracy ${}_{\text{\char 55}}$ on each dataset as accuracy RM increases.