[Paper Review] The CausalBench challenge: A machine learning contest for gene network inference from single-cell perturbation data

Mathieu Chevalley, Jacob Sackett-Sanders+16 more|arXiv (Cornell University)|Aug 29, 2023

Bioinformatics and Genomic Networks4 citations

TL;DR

The CausalBench Challenge invited machine learning researchers to develop methods for inferring causal gene regulatory networks from single-cell perturbation data, using large-scale CRISPR-based datasets. The winning approaches significantly improved performance by effectively leveraging interventional data—demonstrating scalable, data-efficient network inference and establishing a new state of the art in causal gene network reconstruction.

ABSTRACT

In drug discovery, mapping interactions between genes within cellular systems is a crucial early step. Such maps are not only foundational for understanding the molecular mechanisms underlying disease biology but also pivotal for formulating hypotheses about potential targets for new medicines. Recognizing the need to elevate the construction of these gene-gene interaction networks, especially from large-scale, real-world datasets of perturbed single cells, the CausalBench Challenge was initiated. This challenge aimed to inspire the machine learning community to enhance state-of-the-art methods, emphasizing better utilization of expansive genetic perturbation data. Using the framework provided by the CausalBench benchmark, participants were tasked with refining the current methodologies or proposing new ones. This report provides an analysis and summary of the methods submitted during the challenge to give a partial image of the state of the art at the time of the challenge. Notably, the winning solutions significantly improved performance compared to previous baselines, establishing a new state of the art for this critical task in biology and medicine.

Motivation & Objective

• To advance the state of the art in causal gene network inference from single-cell perturbation data, particularly using interventional (CRISPR) datasets.
• To address the gap where existing methods failed to scale with increasing interventional data, despite theoretical advantages.
• To stimulate innovation in machine learning by organizing a community challenge focused on scalable, biologically relevant network inference.
• To establish a benchmark with biologically meaningful metrics that reflect both statistical precision and biological relevance.
• To promote reproducibility and transparency through standardized data, evaluation pipelines, and public submission of detailed method reports.

Proposed method

• Participants were tasked with developing models that infer directed gene regulatory networks from single-cell RNA-seq data under genetic perturbations.
• The challenge used two real-world CRISPR perturbation datasets from RPE-1 and K562 cell lines, with varying fractions of interventional data (25%, 50%, 75%, 100%) to test scalability.
• Evaluation was based on the area under the curve (AUC) of mean Wasserstein distance against interventional data ratio, emphasizing effective use of perturbation data.
• A dual evaluation framework combined statistical precision (Wasserstein distance) and biological recall (false omission rate) to assess method performance.
• Submissions were evaluated via a secure, reproducible pipeline using Docker containers on Google Cloud Platform and Slurm HPC, ensuring consistent and auditable execution.
• A comprehensive ranking was computed as the average rank across four metrics (two datasets × two evaluation modes), enabling fair comparison of method robustness.

Experimental results

Research questions

• RQ1 Can machine learning models effectively scale their performance with increasing amounts of interventional single-cell data in gene network inference?
• RQ2 Why do existing state-of-the-art methods fail to leverage interventional data despite its theoretical advantages in causal discovery?
• RQ3 What methodological innovations enable improved utilization of perturbation data in reconstructing causal gene regulatory networks?
• RQ4 How can evaluation metrics better reflect both statistical accuracy and biologically relevant network structure in gene network inference?
• RQ5 To what extent can community-driven challenges like CausalBench accelerate progress in causal inference for systems biology and drug discovery?

Key findings

• The winning methods demonstrated a strong, upward trend in performance as more interventional data was provided, indicating effective utilization of perturbation signals—unlike prior baselines that showed no scaling.
• Methods such as Betterboost, Guanlab, MeanDifference, and SparseRC significantly improved performance with increasing interventional data, establishing a new state of the art.
• The best-performing models achieved a substantial improvement in mean Wasserstein distance, with performance gains observed across both RPE-1 and K562 cell line datasets.
• The evaluation framework successfully identified methods that balanced statistical precision and biological recall, highlighting the importance of biologically grounded metrics.
• The challenge revealed that prior methods underutilized interventional data, suggesting a large untapped potential for improvement in causal discovery from single-cell perturbation data.
• The use of Dockerized, reproducible evaluation pipelines ensured consistent and auditable results, supporting transparency and future reproducibility in network inference research.