[Paper Review] Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoning

Debjit Paul, Robert West+2 more|arXiv (Cornell University)|Feb 21, 2024

Complex Systems and Decision MakingDecision Sciences3 citations

TL;DR

This paper introduces Frodo, a framework that improves the faithfulness of chain-of-thought (CoT) reasoning in small language models by training them to generate correct reasoning steps and to robustly reason over them using implicit feedback and causal preference objectives. Frodo achieves 2–3% absolute accuracy gains over strong baselines and demonstrates superior robustness and generalization on out-of-distribution reasoning tasks.

ABSTRACT

Large language models (LLMs) have been shown to perform better when asked to reason step-by-step before answering a question. However, it is unclear to what degree the model's final answer is faithful to the stated reasoning steps. In this paper, we perform a causal mediation analysis on twelve LLMs to examine how intermediate reasoning steps generated by the LLM influence the final outcome and find that LLMs do not reliably use their intermediate reasoning steps when generating an answer. To address this issue, we introduce FRODO, a framework to tailor small-sized LMs to generate correct reasoning steps and robustly reason over these steps. FRODO consists of an inference module that learns to generate correct reasoning steps using an implicit causal reward function and a reasoning module that learns to faithfully reason over these intermediate inferences using a counterfactual and causal preference objective. Our experiments show that FRODO significantly outperforms four competitive baselines. Furthermore, FRODO improves the robustness and generalization ability of the reasoning LM, yielding higher performance on out-of-distribution test sets. Finally, we find that FRODO's rationales are more faithful to its final answer predictions than standard supervised fine-tuning.

Motivation & Objective

• To investigate whether large language models (LLMs) reliably use their intermediate reasoning steps to produce final answers.
• To diagnose the lack of faithfulness in chain-of-thought (CoT) reasoning, where reasoning steps do not causally influence the final output.
• To develop a method that ensures reasoning steps are both correct and faithfully used in generating answers.
• To improve the robustness and generalization of distilled reasoning models beyond standard supervised fine-tuning.

Proposed method

• Conduct causal mediation analysis to measure the causal effect of reasoning steps (mediator) on final answers (outcome) across 12 LLMs on three reasoning tasks.
• Generate preference data using LLMs to create factual and counterfactual reasoning chains for implicit feedback during training.
• Train an inference module using Direct Preference Optimization (DPO) to favor correct reasoning chains over counterfactual ones.
• Design a reasoning module trained with a counterfactual and causal preference objective to ensure faithful use of reasoning steps.
• Use small-sized language models fine-tuned via DPO and causal preference ranking to distill reliable CoT reasoning from larger models.
• Evaluate Frodo across four reasoning benchmarks (Quarel, StrategyQA, OpenBookQA, QASC) with multiple model scales and ablation studies.

Figure 1: An example of our proposed causal analysis to measure the faithfulness of the final output with respect to the CoT generated by the model. We perturbed CoT rationales and studied the causal impact on the model’s behaviour.

Experimental results

Research questions

• RQ1 To what extent do LLMs causally rely on their own generated reasoning steps when producing final answers?
• RQ2 Why do many LLMs generate plausible but unfaithful reasoning traces that do not influence their final predictions?
• RQ3 Can implicit feedback from LLMs be used to train smaller models to generate correct and causally relevant reasoning steps?
• RQ4 How can reasoning modules be trained to faithfully use intermediate reasoning steps to improve answer accuracy?
• RQ5 Does Frodo improve generalization and robustness on out-of-distribution reasoning examples compared to standard distillation?

Key findings

• Causal mediation analysis reveals that instruction-tuned models like GPT-3.5-Instruct show stronger causal influence of reasoning steps on final answers than RLHF-tuned models such as ChatGPT.
• GPT-4 changes its final answer only 30% of the time when given perturbed counterfactual reasoning chains, indicating weak faithfulness to reasoning steps.
• Frodo achieves 2–3% absolute accuracy improvements over strong baselines, including supervised fine-tuning and CoT distillation methods.
• Frodo demonstrates superior robustness and generalization on out-of-distribution test sets, outperforming standard distillation in OOD settings.
• The rationales generated by Frodo are significantly more faithful to the final predictions than those from standard supervised fine-tuning.
• The framework’s performance is sensitive to the quality of LLM-generated counterfactual rationales, highlighting a dependency on external model outputs.

Figure 2: Causal graph for natural language reasoning, where we model $P(Y|do(x))$ . $X_{0}$ = original reasoning problem, $X_{1}$ = intervened reasoning problem, $R_{0}$ = reasoning steps for $X_{0}$ reasoning problem, $R_{1}$ = reasoning steps for $X_{1}$ , $Y$ = output, and $M$ = model parameters