[Paper Review] PerturBench: Benchmarking Machine Learning Models for Cellular Perturbation Analysis

Yan Wu, Esther Wershof+8 more|arXiv (Cornell University)|Aug 20, 2024

Cell Image Analysis Techniques11 citations

TL;DR

PerturBench provides a modular benchmarking framework, diverse datasets, and a standardized set of metrics to evaluate perturbation response models in single-cell data, revealing that simple models can outperform some sophisticated architectures and highlighting the importance of rank-based metrics.

ABSTRACT

We introduce a comprehensive framework for modeling single cell transcriptomic responses to perturbations, aimed at standardizing benchmarking in this rapidly evolving field. Our approach includes a modular and user-friendly model development and evaluation platform, a collection of diverse perturbational datasets, and a set of metrics designed to fairly compare models and dissect their performance. Through extensive evaluation of both published and baseline models across diverse datasets, we highlight the limitations of widely used models, such as mode collapse. We also demonstrate the importance of rank metrics which complement traditional model fit measures, such as RMSE, for validating model effectiveness. Notably, our results show that while no single model architecture clearly outperforms others, simpler architectures are generally competitive and scale well with larger datasets. Overall, this benchmarking exercise sets new standards for model evaluation, supports robust model development, and furthers the use of these models to simulate genetic and chemical screens for therapeutic discovery.

Motivation & Objective

• Motivate standard benchmarking in perturbation response modelling and address inconsistencies across datasets and metrics.
• Provide a modular codebase for model development and evaluation of perturbation effects in single-cell data.
• Curate diverse datasets and define biologically relevant tasks to test models under realistic constraints (scaling, imbalance).
• Define a comprehensive metric suite, including rank-based metrics, to fairly compare models.
• Reproduce and evaluate core components of existing models to identify their strengths and limitations.

Proposed method

• Introduce a modular PerturBench codebase with datasets, models, and evaluation APIs (GitHub repository).
• Curate three datasets with covariate transfer and combo prediction tasks to reflect real-world challenges.
• Implement and benchmark multiple models (CPA*, Biolord, SAMS-VAE) and baselines (Linear, Latent Additive, Decoder, Decoder with Covariates).
• Use counterfactual perturbation prediction framed via matching and disentanglement strategies to compare core components.
• Define evaluation metrics including RMSE, cosine similarity of LogFC, and novel rank-based metrics to assess perturbation ordering.
• Perform hyperparameter optimization with RMSE and rank-based loss (RMSE + 0.1 × rank) to select robust configurations.

Experimental results

Research questions

• RQ1 How do different perturbation response models perform on covariate transfer and combo prediction tasks across diverse datasets?
• RQ2 Do simple models and training setups outperform complex disentanglement or adversarial approaches in perturbation prediction?
• RQ3 How do data scale and data imbalance affect model performance and ranking ability in perturbation prediction?
• RQ4 What evaluation metrics best capture practical utility for in-silico perturbation screening (including perturbation ranking)?

Key findings

• Simple models, particularly latent additive with scGPT embeddings, often outperform more complex models on several metrics.
• Some models exhibit mode or posterior collapse; rank metrics reveal ordering failures not captured by traditional metrics.
• Decoder-only models using covariates and perturbations can achieve competitive performance, underscoring the role of architecture and inductive biases.
• Model performance generally improves with more training data, though CPA* shows high variance and limited rank gains on certain tasks.
• Data imbalance reduces performance for several models, with scGPT embeddings helping buffer against imbalance.
• Across combo prediction, linear models can match or exceed performance in some settings, while other architectures show non-linear gains.