[Paper Review] How Likely Do LLMs with CoT Mimic Human Reasoning?

Guangsheng Bao, Hongbo Zhang+3 more|arXiv (Cornell University)|Feb 25, 2024

Auction Theory and ApplicationsDecision Sciences3 citations

TL;DR

This paper investigates whether Chain-of-Thought (CoT) in LLMs genuinely mimics human causal reasoning by analyzing causal structures through Structural Causal Models (SCMs). It reveals that LLMs often produce correct answers via incorrect CoTs, indicating non-causal reasoning; interventions show that in-context learning and fine-tuning significantly alter causal relations between instructions and answers, while CoT's causal influence is weak, exposing a fundamental gap between LLM and human reasoning processes.

ABSTRACT

Chain-of-thought emerges as a promising technique for eliciting reasoning capabilities from Large Language Models (LLMs). However, it does not always improve task performance or accurately represent reasoning processes, leaving unresolved questions about its usage. In this paper, we diagnose the underlying mechanism by comparing the reasoning process of LLMs with humans, using causal analysis to understand the relationships between the problem instruction, reasoning, and the answer in LLMs. Our empirical study reveals that LLMs often deviate from the ideal causal chain, resulting in spurious correlations and potential consistency errors (inconsistent reasoning and answers). We also examine various factors influencing the causal structure, finding that in-context learning with examples strengthens it, while post-training techniques like supervised fine-tuning and reinforcement learning on human feedback weaken it. To our surprise, the causal structure cannot be strengthened by enlarging the model size only, urging research on new techniques. We hope that this preliminary study will shed light on understanding and improving the reasoning process in LLM.

Motivation & Objective

• To investigate whether Chain-of-Thought (CoT) in LLMs reflects genuine causal reasoning akin to humans.
• To identify the causal structure between CoT, instructions, and answers in LLMs using Structural Causal Models (SCMs).
• To compare the implied SCMs in LLMs with those in human reasoning, highlighting discrepancies.
• To examine how training techniques—such as in-context learning, supervised fine-tuning, and RLHF—affect the causal structure of LLM reasoning.
• To develop a causal analysis framework to anticipate and explain LLM reasoning errors.

Proposed method

• The authors use intervention-based causal analysis to test whether CoT or instruction directly causes the answer in LLMs, applying interventions to CoT and instruction while holding the other constant.
• They construct and compare Structural Causal Models (SCMs) inferred from LLM outputs across multiple tasks and models, identifying implied causal structures.
• The study employs controlled interventions: replacing CoT with golden or random reasoning, and altering instruction roles or biases to measure treatment effects on answer accuracy.
• The analysis is applied across six reasoning tasks (e.g., GSM8K, LogiQA, FOLIO) and four LLMs (Llama2, GPT-3.5-Turbo, GPT-4, Mistral-7B), with results aggregated via statistical significance testing.
• The authors classify implied SCM types based on causal influence patterns, using treatment effect shifts as indicators of causal direction.
• They evaluate the impact of in-context learning (ICL), supervised fine-tuning (SFT), and RLHF on causal structure, using intervention-based comparisons across model variants.

Figure 1: CoT and Answer do not fully align. Refer to Figure 4 in Appendix C for instances and Section 4 for a thorough explanation.

Experimental results

Research questions

• RQ1 To what extent does a correct Chain-of-Thought (CoT) reliably lead to a correct answer in LLMs?
• RQ2 Is there a direct causal relationship between CoT and the final answer in LLMs, as in human reasoning?
• RQ3 How do training techniques like in-context learning, SFT, and RLHF affect the causal structure between instructions and answers in LLMs?
• RQ4 How do the implied SCMs in LLMs compare to those in human reasoning, and where do they diverge?
• RQ5 Can causal structure analysis predict or explain LLM reasoning errors?

Key findings

• LLMs frequently produce correct answers following incorrect CoTs, and incorrect answers following correct CoTs, indicating that CoT is not a reliable indicator of reasoning quality.
• Intervention analysis shows that in-context learning (ICL) significantly alters the causal structure between instruction and answer, especially in tasks like Addition and Multiplication.
• Supervised fine-tuning (SFT) and RLHF have strong effects on the causal relation between instruction and answer, but minimal impact on the CoT-to-answer causal link.
• The implied SCM in LLMs often lacks the causal consistency seen in human reasoning, with CoT frequently not being a true cause of the answer.
• Larger models (e.g., Llama2-70B-Chat) tend to approximate more optimal SCM types, suggesting scale may improve causal alignment.
• The study identifies that spurious correlations from pre-training data can lead to non-causal reasoning, where CoT appears predictive but is not causally responsible for the answer.

Figure 2: LLMs with CoT exhibit inconsistent effects, where the CoT shows inferior performance to Direct in the tasks on the left but superior performance in the tasks on the right in a zero-shot setting. A more comprehensive set of results can be found in Table 6 in Appendix C . (The Addition resul