[Paper Review] Cell-JEPA: Latent Representation Learning for Single-Cell Transcriptomics

Ali ElSheikh, Rui-Xi Wang+10 more|arXiv (Cornell University)|Feb 2, 2026

Single-cell and spatial transcriptomics0 citations

TL;DR

Cell-JEPA introduces a joint-embedding predictive architecture that learns robust latent cell representations by predicting teacher embeddings from masked student inputs, improving zero-shot cell-type clustering and downstream tasks beyond reconstruction-based baselines.

ABSTRACT

Single-cell foundation models learn by reconstructing masked gene expression, implicitly treating technical noise as signal. With dropout rates exceeding 90%, reconstruction objectives encourage models to encode measurement artifacts rather than stable cellular programs. We introduce Cell-JEPA, a joint-embedding predictive architecture that shifts learning from reconstructing sparse counts to predicting in latent space. The key insight is that cell identity is redundantly encoded across genes. We show predicting cell-level embeddings from partial observations forces the model to learn dropout-robust features. On cell-type clustering, Cell-JEPA achieves 0.72 AvgBIO in zero-shot transfer versus 0.53 for scGPT, a 36% relative improvement. On perturbation prediction within a single cell line, Cell-JEPA improves absolute-state reconstruction but not effect-size estimation, suggesting that representation learning and perturbation modeling address complementary aspects of cellular prediction.

Motivation & Objective

• Motivate learning robust cellular representations from highly sparse and noisy single-cell data.
• Propose a JEPA-style latent-space prediction objective added to a reconstruction baseline.
• Pre-train on a large human kidney single-cell corpus to learn a generalizable foundation model.
• Evaluate on cell-type clustering (finetuned and zero-shot) and perturbation-response prediction.

Proposed method

• Adopt a student–teacher Transformer architecture built on scGPT tiles.
• Mask expression values and train the student to predict latent representations produced by a slow EMA teacher.
• Train with a joint pre-training objective: gene-level reconstruction loss and JEPA latent-space prediction loss.
• JEPA loss aligns the student’s masked embedding with the teacher’s embedding via a cosine similarity objective.
• Downstream finetuning combines GEP, GEPC, ECS, and JEPA losses for robust cell-state modeling.
• Optionally extend finetuning with perturbation prediction using perturbation embeddings and a perturbed JEPA objective.

Figure 1 : Overview of the Cell-JEPA Pipeline. Raw biological cells undergo single-cell RNA sequencing to generate high-dimensional, sparse gene expression profiles. The profiles are passed through a joint embedding architecture, where a student encoder receives masked inputs and predicts the stable

Experimental results

Research questions

• RQ1 Can latent-space JEPA objectives improve transferability of single-cell representations beyond reconstruction losses?
• RQ2 How does Cell-JEPA perform on cell-type clustering in finite-sample and zero-shot settings compared with scGPT?
• RQ3 Do latent-space representations improve perturbation-response prediction within a fixed cellular context?
• RQ4 What is the impact of the JEPA objective on embedding robustness to dropout and sparsity in scRNA-seq data?

Key findings

• Cell-JEPA achieves higher clustering quality than scGPT after finetuning on PBMC-10K across AvgBIO, NMI_cell, ASW_cell, and ARI_cell (0.7830 vs 0.7531; 0.7761 vs 0.7652; 0.7256 vs 0.7100; 0.8472 vs 0.7842).
• In zero-shot PBMC evaluation, Cell-JEPA outperforms scGPT on AvgBIO, NMI_cell, ASW_cell, and ARI_cell (0.7194 vs 0.5288; 0.7690 vs 0.5585; 0.5811 vs 0.5329; 0.8081 vs 0.4951).
• Cell-JEPA yields substantial relative improvement (~36%) in zero-shot cell-type clustering over scGPT.
• For perturbation prediction in K562 Perturb-seq datasets, Cell-JEPA improves absolute post-perturbation state prediction over scGPT (e.g., Norman pear 0.787 vs 0.631; Adamson pear 0.937 vs 0.905).
• Delta-based metrics show limited gains and in some cases decreased performance for unseen perturbations, indicating a trade-off between absolute-state fidelity and perturbation effect-size estimation.

Figure 2 : Cell-JEPA Model Architecture and Training Pipeline. Cell-JEPA builds on scGPT with a student–teacher transformer architecture. Gene identities and discretized expression values are embedded and summed to form input token embeddings. The student encoder processes masked inputs, while the t