[Paper Review] Causal Reasoning and Large Language Models: Opening a New Frontier for Causality

Emre Kıcıman, Robert A. Van Ness+2 more|arXiv (Cornell University)|Apr 28, 2023

Topic ModelingComputer Science90 citations

TL;DR

The paper shows that GPT-3.5/4-based methods achieve state-of-the-art accuracy on causal discovery and counterfactual tasks, while highlighting robustness concerns and proposing a framework to integrate LLMs with traditional causal methods.

ABSTRACT

The causal capabilities of large language models (LLMs) are a matter of significant debate, with critical implications for the use of LLMs in societally impactful domains such as medicine, science, law, and policy. We conduct a "behavorial" study of LLMs to benchmark their capability in generating causal arguments. Across a wide range of tasks, we find that LLMs can generate text corresponding to correct causal arguments with high probability, surpassing the best-performing existing methods. Algorithms based on GPT-3.5 and 4 outperform existing algorithms on a pairwise causal discovery task (97%, 13 points gain), counterfactual reasoning task (92%, 20 points gain) and event causality (86% accuracy in determining necessary and sufficient causes in vignettes). We perform robustness checks across tasks and show that the capabilities cannot be explained by dataset memorization alone, especially since LLMs generalize to novel datasets that were created after the training cutoff date. That said, LLMs exhibit unpredictable failure modes, and we discuss the kinds of errors that may be improved and what are the fundamental limits of LLM-based answers. Overall, by operating on the text metadata, LLMs bring capabilities so far understood to be restricted to humans, such as using collected knowledge to generate causal graphs or identifying background causal context from natural language. As a result, LLMs may be used by human domain experts to save effort in setting up a causal analysis, one of the biggest impediments to the widespread adoption of causal methods. Given that LLMs ignore the actual data, our results also point to a fruitful research direction of developing algorithms that combine LLMs with existing causal techniques. Code and datasets are available at https://github.com/py-why/pywhy-llm.

Motivation & Objective

• Clarify the spectrum of causal reasoning tasks (covariance-based, logic-based, type vs actual causality) and how LLMs fit into them.
• Assess LLMs on causal discovery benchmarks and actual causality tasks to understand capabilities and limitations.
• Propose a knowledge-based approach where LLMs leverage metadata to inform causal graphs and reasoning.
• Highlight how LLMs can augment, not replace, traditional causal methods in practice, especially in high-stakes domains.

Proposed method

• Evaluate LLMs (GPT-3.5 and GPT-4) on pairwise causal discovery using the Tübingen cause-effect pairs benchmark and other medical/climate datasets.
• Prompt engineering to elicit directional causal judgments and counterfactual reasoning.
• Memorization and redaction tests to probe robustness and ensure results are not due to data memorization.
• Compare LLM performance to state-of-the-art covariance-based causal discovery methods.
• Analyze the role of LLMs in generating causal graphs from variable metadata rather than data values.

Experimental results

Research questions

• RQ1 Can LLMs identify causal direction between two variables using only variable metadata?
• RQ2 How do LLMs perform on pairwise causal discovery and full graph discovery compared to covariance-based methods?
• RQ3 To what extent can LLMs answer counterfactual questions and identify necessary/sufficient causes?
• RQ4 What are the robustness and memorization concerns when using prompts to probe LLM causal reasoning?
• RQ5 How can LLMs complement traditional causal methods in practice?

Key findings

• LLMs achieve up to 97% accuracy on the Tübingen pairwise causal discovery task, surpassing prior bests by ~13 points.
• GPT-3.5/4 show strong performance on medical pain diagnosis and climate datasets for causal discovery.
• GPT-4 attains 92% accuracy on counterfactual reasoning benchmarks, with an 86% accuracy on identifying necessary and sufficient causes in vignettes.
• Performance is highly dependent on prompt design and can exhibit unpredictable failure modes.
• LLMs provide complementary knowledge-based signals by leveraging domain knowledge to generate causal graphs from natural language context.
• LLMs can augment existing causal methods by bootstrapping, critiquing, and translating between natural language and formal causal representations.