[Paper Review] REFINER: Reasoning Feedback on Intermediate Representations

Debjit Paul, Mete Ismayilzada+5 more|arXiv (Cornell University)|Apr 4, 2023

Topic Modeling59 references31 citations

TL;DR

REFINER trains a generator to produce intermediate reasoning steps and a critic to give fine-grained feedback, improving reasoning tasks across math word problems, synthetic natural language reasoning, and moral story generation.

ABSTRACT

Language models (LMs) have recently shown remarkable performance on reasoning tasks by explicitly generating intermediate inferences, e.g., chain-of-thought prompting. However, these intermediate inference steps may be inappropriate deductions from the initial context and lead to incorrect final predictions. Here we introduce REFINER, a framework for finetuning LMs to explicitly generate intermediate reasoning steps while interacting with a critic model that provides automated feedback on the reasoning. Specifically, the critic provides structured feedback that the reasoning LM uses to iteratively improve its intermediate arguments. Empirical evaluations of REFINER on three diverse reasoning tasks show significant improvements over baseline LMs of comparable scale. Furthermore, when using GPT-3.5 or ChatGPT as the reasoner, the trained critic significantly improves reasoning without finetuning the reasoner. Finally, our critic model is trained without expensive human-in-the-loop data but can be substituted with humans at inference time.

Motivation & Objective

• Motivate and enable explicit generation of intermediate reasoning steps in language models while mitigating incorrect deductions.
• Propose an interaction loop between a generator and a critic that provides fine-grained feedback on intermediate representations.
• Show that structured feedback improves reasoning across multiple tasks and model scales.
• Demonstrate that a trained critic can enhance reasoning even for external LLMs like GPT-3.5 and ChatGPT.
• Provide ablations and analyses on the roles of feedback, critic quality, and inference-time use.

Proposed method

• Two-model REFINER framework with a generator and a critic.
• Critic trained on automatically constructed, fine-grained feedback data describing reasoning errors.
• Generator fine-tuned to generate intermediate representations and refined via critic feedback.
• Feedback is semi-structured text derived from error-type templates, converted to natural language for the generator.
• Exploration during training via nucleus sampling to expose the generator to diverse feedback.
• Inference uses the trained critic to guide or correct the generator’s intermediate steps.

Figure 1: REFINER example. The critic model provides the generator model with feedback on its reasoning errors after evaluating the generated intermediate steps. The feedback, alongside the original question and previous intermediate equation, are fed back to the generator model.

Experimental results

Research questions

• RQ1 Can fine-grained, structured feedback on intermediate reasoning steps improve final task performance beyond scalar rewards?
• RQ2 Does a task-specialist critic improve intermediate representations and final answers across diverse reasoning tasks?
• RQ3 Can a trained critic enhance performance of external LLMs (e.g., GPT-3.5, ChatGPT) without finetuning them?
• RQ4 How does REFINER perform relative to other refinement approaches like self-refine or self-consistency across tasks?
• RQ5 How robust are REFINER’s gains to imperfect (noisy) critics and to inference-time use?

Key findings

• REFINER yields significant gains over comparably sized baselines on math word problems, synthetic natural language reasoning, and moral story generation.
• On MWP, IR (z) improves from 34.1 to 47.2 with REFINER base and to 53.8 with REFINER large; GPT-3.5 + CoT + REFINER critic reaches 67.3 IR and 70.6 final accuracy.
• On sNLR, REFINER improves IR to 53.8 (from 46.7 baseline) and final answer accuracy to 53.8? (table shows IR=53.8, Con=?) with large? and shows gains like +2.9 EM over UQA-base and +6.8 EM over GPT-3.5.
• On MS, REFINER improves moral norm and action relevance vs UQA-large baseline by about 20 points with Krippendorff’s alpha indicating higher agreement.
• REFINER’s trained critic alone can improve GPT-3.5’s few-shot reasoning by notable margins (+3.5, +6.8) on MWP and sNLR respectively.
• REFINER outperforms self-refinement approaches and can improve CoT-based methods when combined (Self-Consistency, ReACT).
• Ablations show the critical role of inference-time critic feedback and the exploration phase; imperfect critics still provide benefits, while highly noisy critics can harm performance.

Figure 2: Overview of REFINER interaction loop. Left side: Training the critic model. Right side: In each iteration, the generator generates multiple hypotheses. The critic randomly selects one hypothesis and provides feedback based on reasoning errors.