[Paper Review] Multi-Agent Causal Discovery Using Large Language Models

Hao Duong Le, Xin Xia+1 more|arXiv (Cornell University)|Jul 21, 2024

Data Quality and Management5 citations

TL;DR

The paper proposes MAC, a three-model framework using multi-agent LLMs to perform causal discovery, including Meta Agents, Coding Agents, and Hybrid variants, evaluated on multiple datasets with favorable results compared to single-agent and classical methods.

ABSTRACT

Causal discovery aims to identify causal relationships between variables and is a critical research area in machine learning. Traditional methods focus on statistical or machine learning algorithms to uncover causal links from structured data, often overlooking the valuable contextual information provided by metadata. Large language models (LLMs) have shown promise in creating unified causal discovery frameworks by incorporating both structured data and metadata. However, their potential in multi-agent settings remains largely unexplored. To address this gap, we introduce the Multi-Agent Causal Discovery Framework (MAC), which consists of two key modules: the Debate-Coding Module (DCM) and the Meta-Debate Module (MDM). The DCM begins with a multi-agent debating and coding process, where agents use both structured data and metadata to collaboratively select the most suitable statistical causal discovery (SCD) method. The selected SCD is then applied to the structured data to generate an initial causal graph. This causal graph is transformed into causal metadata through the Meta Fusion mechanism. With all the metadata, MDM then refines the causal structure by leveraging a multi-agent debating framework. Extensive experiments across five datasets demonstrate that MAC outperforms both traditional statistical causal discovery methods and existing LLM-based approaches, achieving state-of-the-art performance.

Motivation & Objective

• Motivate data-driven causal discovery when domain knowledge is scarce or costly.
• Leverage multi-agent LLM capabilities to enhance causal graph learning.
• Integrate reasoning (debate) with statistical causal discovery algorithms.
• Compare three MAC configurations (Meta, Coding, Hybrid) to baseline methods.

Proposed method

• Introduce three MAC models: Meta Agents Model (debate-based reasoning to output a causal graph).
• Introduce Debate-Coding Module (debaters select and implement a causal discovery algorithm via coding execution).
• Define Hybrid Models (Coding-Meta Hybrid and Debating-Coding Hybrid) combining reasoning and statistics.
• Use GPT-3.5-turbo with ReAct prompting and zero-shot/few-shot prompting strategies for agent reasoning and action.
• Evaluate using adjacency matrix metrics (SHD, FPR, FNR, Precision, Recall, F1) and NHD on continuous-variable datasets.

Figure 1: Meta-Debate Module

Experimental results

Research questions

• RQ1 Can multi-agent LLM systems improve causal graph discovery over single-agent LLMs and traditional algorithms?
• RQ2 What are the relative strengths of purely reasoning-based, purely coding-based, and hybrid MAC configurations for causal discovery?
• RQ3 How do MAC methods perform across diverse datasets (cars, climate, proteins) compared to PC, Exact Search, DirectLiNGAM, and LLM-based baselines?

Key findings

• Coding Agents Model achieves best SHD (4) and high precision (1) on Auto MPG.
• Meta Agents Model achieves strong performance with SHD 5 and balanced metrics on Auto MPG.
• On DWD climate data, Meta Agents Model yields SHD 5 and NHD 0.194, outperforming classical methods.
• On Sachs protein data, Single-agent GPT-4o shows low SHD (18) and strong FNR (0.098), while multi-agent variants (Coding-Debating Hybrid, Coding-Meta Hybrid) show competitive results.
• Across datasets, classical SCD methods (PC, Exact Search, DirectLiNGAM) generally have higher SHD values than MAC variants.
• Overall, the proposed multi-agent framework often achieves better structural accuracy and competitive or superior F1 scores compared to baselines.

Figure 2: Debate-Coding Module