[Paper Review] Benchmarking Transcriptomics Foundation Models for Perturbation Analysis : one PCA still rules them all

Ihab Bendidi, Shawn Whitfield+5 more|arXiv (Cornell University)|Oct 17, 2024

RNA Research and Splicing8 citations

TL;DR

The paper benchmarks transcriptomics foundation models for perturbation analysis across public datasets, finding scVI and PCA generally outperform foundation-models, and introduces Structural Integrity as a new evaluation metric.

ABSTRACT

Understanding the relationships among genes, compounds, and their interactions in living organisms remains limited due to technological constraints and the complexity of biological data. Deep learning has shown promise in exploring these relationships using various data types. However, transcriptomics, which provides detailed insights into cellular states, is still underused due to its high noise levels and limited data availability. Recent advancements in transcriptomics sequencing provide new opportunities to uncover valuable insights, especially with the rise of many new foundation models for transcriptomics, yet no benchmark has been made to robustly evaluate the effectiveness of these rising models for perturbation analysis. This article presents a novel biologically motivated evaluation framework and a hierarchy of perturbation analysis tasks for comparing the performance of pretrained foundation models to each other and to more classical techniques of learning from transcriptomics data. We compile diverse public datasets from different sequencing techniques and cell lines to assess models performance. Our approach identifies scVI and PCA to be far better suited models for understanding biological perturbations in comparison to existing foundation models, especially in their application in real-world scenarios.

Motivation & Objective

• Motivate a biologically grounded benchmark for perturbation analysis in transcriptomics.
• Compare pretrained transcriptomics foundation models to classical methods on perturbation tasks.
• Identify which models best capture perturbation signals across datasets and techniques.
• Introduce Structural Integrity as a new evaluation metric for preserving gene activity structure.

Proposed method

• Curate diverse public perturbation datasets from three sequencing techniques and multiple cell lines.
• Define a hierarchical evaluation framework with metrics: iLISI batch integration, latent separability (linear probing), perturbation consistency, local latent structure (kNN), zero-shot known relationships recall, and reconstruction interpretability.
• Propose Structural Integrity, a normalization-based metric using centered log-expression and Frobenius distance to quantify preservation of perturbation structure within batches.
• Apply post-processing (control-based centering, TVN, or raw embeddings) to model outputs and select best-performing method per model-task.
• Benchmark baseline PCA and scVI against foundation models like Geneformer, scGPT, CellPLM, and UCE across tasks.
• Provide full results and code (Tx-Evaluation) for reproducibility.

Figure 1: Known biological relationship recall scores for (Replogle et al., 2022 ) and L1000 Assay for scVI, trained using different gene distributions. Different datasets and sequencing approaches benefit from different gene distributions.

Experimental results

Research questions

• RQ1 Can transcriptomics foundation models generalize to perturbation analysis tasks beyond batch correction or classification?
• RQ2 Which models and post-processing strategies best capture perturbation effects across diverse datasets and sequencing modalities?
• RQ3 How does gene distribution assumption affect model performance on perturbation tasks?
• RQ4 Does a simple model like PCA or scVI outperform complex foundation models on perturbation-centric benchmarks?
• RQ5 What is the utility of the Structural Integrity metric for evaluating perturbation representations?

Key findings

• Foundation models do not generalize well to perturbation tasks compared to PCA and scVI, except for batch effect reduction.
• scVI (trained from scratch or via zero-shot transfer) often achieves strong performance and scalable results, outperforming many foundation models.
• Gene distribution (ZINB, NB, Poisson) materially impacts scVI performance and is dataset-dependent (Replogle vs L1000).
• scVI shows robust learning even with small training data and scales with more data in single-cell perturbation contexts.
• Foundation models like Geneformer and scGPT mainly excel in batch effect reduction and struggle on biologically meaningful perturbation tasks.
• Structural Integrity is a new metric indicating how well perturbation relationships are preserved in the latent gene activity space.

Figure 2: Training scVI on different amounts of samples, using distinct batches and perturbations, before evaluating known biological relationship retrieval. scVI is robust to very low data regime, and shows strong data scaling laws on single cell data.