[Paper Review] Season combinatorial intervention predictions with Salt & Peper

Thomas Gaudelet, Alice Del Vecchio+5 more|arXiv (Cornell University)|Apr 25, 2024

Intelligent Tutoring Systems and Adaptive Learning4 citations

TL;DR

This paper introduces Salt, a biologically-motivated, non-parametric baseline assuming additive effects of pairwise genetic interventions, and Peper, a deep learning model that extends Salt's additive inductive bias to achieve state-of-the-art performance in predicting transcriptomic responses to combinatorial CRISPR perturbations. Despite high in-distribution accuracy, all models, including Peper, show significant performance drops in out-of-distribution settings, highlighting critical limitations in generalization and the need for better priors and data acquisition strategies.

ABSTRACT

Interventions play a pivotal role in the study of complex biological systems. In drug discovery, genetic interventions (such as CRISPR base editing) have become central to both identifying potential therapeutic targets and understanding a drug's mechanism of action. With the advancement of CRISPR and the proliferation of genome-scale analyses such as transcriptomics, a new challenge is to navigate the vast combinatorial space of concurrent genetic interventions. Addressing this, our work concentrates on estimating the effects of pairwise genetic combinations on the cellular transcriptome. We introduce two novel contributions: Salt, a biologically-inspired baseline that posits the mostly additive nature of combination effects, and Peper, a deep learning model that extends Salt's additive assumption to achieve unprecedented accuracy. Our comprehensive comparison against existing state-of-the-art methods, grounded in diverse metrics, and our out-of-distribution analysis highlight the limitations of current models in realistic settings. This analysis underscores the necessity for improved modelling techniques and data acquisition strategies, paving the way for more effective exploration of genetic intervention effects.

Motivation & Objective

• To address the challenge of predicting transcriptomic responses to pairwise genetic interventions in complex biological systems.
• To develop a biologically grounded baseline, Salt, assuming mostly additive effects of gene combinations.
• To design Peper, a deep learning model that leverages Salt’s additive inductive bias for improved prediction accuracy.
• To evaluate current state-of-the-art models across diverse metrics and identify limitations in real-world, out-of-distribution scenarios.
• To highlight the urgent need for better inductive biases, data acquisition strategies, and model architectures to enable reliable prediction of novel combinatorial intervention effects.

Proposed method

• Salt is a non-parametric, biologically-inspired baseline model that assumes the combined effect of two gene perturbations is approximately the sum of their individual effects.
• Peper is a deep learning model trained to predict transcriptomic responses to pairwise genetic interventions, incorporating Salt’s additive prior as an inductive bias.
• The model uses single-cell RNA-seq data from genome-scale perturbation screens, with predictions evaluated on gene expression levels across individual cells.
• Performance is evaluated using multiple metrics, including RMSE and correlation, with subgroup analysis based on interaction types (e.g., potentiation, suppression) as defined by Norman et al. (2019).
• An out-of-distribution evaluation strategy is employed, splitting data by intervention subgroups to simulate real-world prediction scenarios where models face previously unseen combinations.
• The study compares Peper and Salt against state-of-the-art models (GEARS, CPA) using both in-distribution and out-of-distribution splits to assess generalization capacity.

Experimental results

Research questions

• RQ1 Can a biologically-motivated, non-parametric baseline like Salt provide a reliable reference point for evaluating combinatorial intervention prediction models?
• RQ2 To what extent does incorporating an additive inductive bias improve prediction accuracy in transcriptomic response modeling for pairwise genetic perturbations?
• RQ3 How do current state-of-the-art models perform when evaluated on out-of-distribution combinations not seen during training?
• RQ4 Why do existing models, including Peper, show significant performance degradation in out-of-distribution settings despite strong in-distribution performance?
• RQ5 What are the implications of modeling higher moments of single-cell distributions, and does this improve or hinder performance on key prediction metrics?

Key findings

• Peper achieves state-of-the-art performance on the primary evaluation metric, outperforming existing models including GEARS and CPA across all intervention subgroups.
• Salt demonstrates strong baseline performance, particularly in subgroups with mostly additive effects, validating its biological plausibility as a non-parametric reference.
• All models, including Peper, suffer significant performance drops in out-of-distribution settings, with the largest degradation observed in the 'potentiation' interaction type.
• The out-of-distribution analysis reveals that current models are ill-suited for predicting novel, previously unseen combinatorial intervention effects—precisely the scenario where they are most needed in drug discovery.
• The study suggests that current inductive biases, such as those in Salt, are insufficient to capture complex biological interactions like potentiation, indicating a need for more dynamic or context-aware priors.
• The results imply that modeling higher moments of single-cell distributions may not improve performance on key metrics, and that (pseudo-)bulk data may be more effective for certain applications.