[Paper Review] Is ChatGPT a Good Causal Reasoner? A Comprehensive Evaluation

Jinglong Gao, Xiao Ding+2 more|arXiv (Cornell University)|May 12, 2023

Topic Modeling4 citations

TL;DR

This paper evaluates ChatGPT's causal reasoning abilities across multiple benchmarks, revealing it is a strong causal explainer but a weak causal reasoner due to severe hallucination. Despite using in-context learning and chain-of-thought prompting, performance degrades on implicit causality and complex sentences, with open-ended prompts worsening results—highlighting fundamental limitations in reasoning over causal relationships without supervision.

ABSTRACT

Causal reasoning ability is crucial for numerous NLP applications. Despite the impressive emerging ability of ChatGPT in various NLP tasks, it is unclear how well ChatGPT performs in causal reasoning. In this paper, we conduct the first comprehensive evaluation of the ChatGPT's causal reasoning capabilities. Experiments show that ChatGPT is not a good causal reasoner, but a good causal explainer. Besides, ChatGPT has a serious hallucination on causal reasoning, possibly due to the reporting biases between causal and non-causal relationships in natural language, as well as ChatGPT's upgrading processes, such as RLHF. The In-Context Learning (ICL) and Chain-of-Thought (CoT) techniques can further exacerbate such causal hallucination. Additionally, the causal reasoning ability of ChatGPT is sensitive to the words used to express the causal concept in prompts, and close-ended prompts perform better than open-ended prompts. For events in sentences, ChatGPT excels at capturing explicit causality rather than implicit causality, and performs better in sentences with lower event density and smaller lexical distance between events. The code is available on https://github.com/ArrogantL/ChatGPT4CausalReasoning .

Motivation & Objective

• To comprehensively evaluate ChatGPT’s causal reasoning capabilities across multiple task formats and versions.
• To identify the root causes of ChatGPT’s causal hallucinations, particularly in relation to training data biases and RLHF.
• To assess the impact of prompting strategies—such as in-context learning and chain-of-thought—on causal reasoning performance.
• To examine how linguistic cues, event density, and lexical distance affect causal reasoning accuracy.
• To determine whether open-ended prompts improve or degrade causal reasoning performance compared to closed-ended ones.

Proposed method

• Conducted evaluations on four GPT versions: text-davinci-002, text-davinci-003, gpt-3.5-turbo, and gpt-4.
• Used three benchmark tasks: Event Causality Identification (ECI), Causal Discovery (CD) in multiple-choice and binary classification formats, and Causal Explanation Generation (CEG).
• Designed prompts with varying formats: closed-ended (Yes/No), open-ended (free-form generation), and those using causal cue words.
• Applied in-context learning (ICL) and chain-of-thought (CoT) prompting to assess their impact on hallucination and accuracy.
• Evaluated performance using strict metrics: precision, recall, and F1, with relaxed matching for open-ended outputs.
• Analyzed sensitivity to linguistic expression of causality, event density, and lexical distance between events.

Figure 1: The forms of three causal reasoning tasks and the prompts we use. The content that requires ChatGPT to reply is marked in red . The multiple-choice CD task also involves samples that ask for selecting the result of the input event. For such samples, we modify the “cause” in the question to

Experimental results

Research questions

• RQ1 How does ChatGPT perform on causal reasoning tasks compared to its ability to generate causal explanations?
• RQ2 To what extent do in-context learning and chain-of-thought prompting exacerbate causal hallucinations in ChatGPT?
• RQ3 How does the choice of causal cue words in prompts affect the accuracy of causal reasoning?
• RQ4 Does ChatGPT perform better on explicit or implicit causality, and how does event density or lexical distance influence performance?
• RQ5 Can open-ended generation prompts improve causal reasoning, or do they degrade performance due to combined event extraction and reasoning demands?

Key findings

• ChatGPT is not a good causal reasoner but performs well in generating causal explanations, indicating it understands causal logic in a descriptive, not inferential, way.
• ChatGPT exhibits severe causal hallucination, with performance dropping significantly on open-ended prompts and when using ICL or CoT techniques.
• The model performs best on explicit causality (e.g., 'because', 'so') and struggles with implicit causality requiring commonsense reasoning.
• Performance decreases as event density increases and lexical distance between events grows, indicating difficulty in long-range dependency reasoning.
• Open-ended prompts reduce F1 scores to as low as 7.4, showing they are ineffective for causal reasoning compared to closed-ended prompts (F1 = 41.0).
• Binary classification format for causal discovery is more reliable than multiple-choice, which overestimates performance due to limited options.

Figure 2: Prompts that express causal concepts in various ways. The content that requires ChatGPT to reply is marked in red .