[Paper Review] CausalBench: A Large-scale Benchmark for Network Inference from Single-cell Perturbation Data

Mathieu Chevalley, Yusuf Roohani+3 more|arXiv (Cornell University)|Oct 31, 2022

Advanced Causal Inference Techniques7 citations

TL;DR

CausalBench introduces a large-scale, open benchmark for evaluating causal network inference methods on real-world single-cell perturbation data, using over 200,000 interventional samples from CRISPR-based scRNA-seq experiments. It reveals that state-of-the-art methods often fail to scale and do not outperform observational-only approaches, challenging assumptions from synthetic benchmark results.

ABSTRACT

Causal inference is a vital aspect of multiple scientific disciplines and is routinely applied to high-impact applications such as medicine. However, evaluating the performance of causal inference methods in real-world environments is challenging due to the need for observations under both interventional and control conditions. Traditional evaluations conducted on synthetic datasets do not reflect the performance in real-world systems. To address this, we introduce CausalBench, a benchmark suite for evaluating network inference methods on real-world interventional data from large-scale single-cell perturbation experiments. CausalBench incorporates biologically-motivated performance metrics, including new distribution-based interventional metrics. A systematic evaluation of state-of-the-art causal inference methods using our CausalBench suite highlights how poor scalability of current methods limits performance. Moreover, methods that use interventional information do not outperform those that only use observational data, contrary to what is observed on synthetic benchmarks. Thus, CausalBench opens new avenues in causal network inference research and provides a principled and reliable way to track progress in leveraging real-world interventional data.

Motivation & Objective

• To address the lack of reliable, real-world benchmarks for evaluating causal inference methods in single-cell genomics.
• To provide a standardized, scalable evaluation framework using large-scale interventional single-cell data.
• To identify performance gaps in existing causal inference methods when applied to real biological systems.
• To develop biologically meaningful metrics that reflect the recovery of true causal relationships in gene regulatory networks.
• To challenge assumptions from synthetic benchmarks by testing methods on real interventional data.

Proposed method

• CausalBench integrates two large-scale, publicly available single-cell CRISPR perturbation datasets with over 200,000 interventional samples.
• It introduces novel distribution-based interventional metrics to assess the recovery of strong interventional effects and minimize omission of causal edges.
• The benchmark includes curated implementations of 15 state-of-the-art causal and non-causal network inference methods for direct comparison.
• Performance is evaluated under varying sample and intervention set sizes to assess scalability and robustness.
• A standardized evaluation protocol ensures consistent computational resources and hyperparameter tuning across all methods.
• Metrics are validated against existing biological knowledge bases to ensure biological relevance.

Experimental results

Research questions

• RQ1 How do state-of-the-art causal inference methods perform on real-world single-cell perturbation data compared to synthetic benchmarks?
• RQ2 To what extent do interventional data improve causal network inference performance in real biological systems?
• RQ3 What are the scalability limitations of current causal inference methods when applied to large-scale single-cell data?
• RQ4 Do methods that explicitly model interventions outperform those relying only on observational data in real-world settings?
• RQ5 How well do existing metrics capture biologically meaningful causal relationships in gene regulatory networks?

Key findings

• No state-of-the-art method achieved both sample and intervention scaling, with performance improving by less than 10% when using 100% of samples or perturbations.
• Methods that incorporate interventional data did not outperform those using only observational data, contradicting results from synthetic benchmarks.
• Causal inference methods did not consistently outperform non-causal baselines on real biological data, indicating a gap in methodological robustness.
• The benchmark revealed poor scalability in existing algorithms, especially under increasing numbers of interventions and samples.
• The performance of all methods was limited by computational constraints and suboptimal utilization of interventional information.
• CausalBench enables more accurate, biologically grounded evaluation of causal discovery methods, highlighting the need for new algorithmic developments.