[Paper Review] CryoBench: Diverse and challenging datasets for the heterogeneity problem in cryo-EM

Minkyu Jeon, Rishwanth Raghu+7 more|arXiv (Cornell University)|Aug 10, 2024

Advanced Electron Microscopy Techniques and Applications11 citations

TL;DR

CryoBench presents five synthetic, ground-truth cryo-EM datasets capturing conformational and compositional heterogeneity, along with metrics and benchmarks to evaluate heterogeneous reconstruction methods. It analyzes state-of-the-art methods and introduces new evaluation criteria.

ABSTRACT

Cryo-electron microscopy (cryo-EM) is a powerful technique for determining high-resolution 3D biomolecular structures from imaging data. Its unique ability to capture structural variability has spurred the development of heterogeneous reconstruction algorithms that can infer distributions of 3D structures from noisy, unlabeled imaging data. Despite the growing number of advanced methods, progress in the field is hindered by the lack of standardized benchmarks with ground truth information and reliable validation metrics. Here, we introduce CryoBench, a suite of datasets, metrics, and benchmarks for heterogeneous reconstruction in cryo-EM. CryoBench includes five datasets representing different sources of heterogeneity and degrees of difficulty. These include conformational heterogeneity generated from designed motions of antibody complexes or sampled from a molecular dynamics simulation, as well as compositional heterogeneity from mixtures of ribosome assembly states or 100 common complexes present in cells. We then analyze state-of-the-art heterogeneous reconstruction tools, including neural and non-neural methods, assess their sensitivity to noise, and propose new metrics for quantitative evaluation. We hope that CryoBench will be a foundational resource for accelerating algorithmic development and evaluation in the cryo-EM and machine learning communities. Project page: https://cryobench.cs.princeton.edu.

Motivation & Objective

• Provide standardized, ground-truth benchmarks for conformational and compositional heterogeneity in cryo-EM.
• Enable quantitative and qualitative comparison of heterogeneous reconstruction methods.
• Assess robustness to noise and pose/CTF variations across diverse data sources.
• Propose new metrics for embedding quality, clustering, and volume reconstruction performance.
• Offer dataset generation tools to foster future method development in cryo-EM and ML.

Proposed method

• Design five synthetic datasets with ground-truth conformational and compositional heterogeneity (IgG-1D, IgG-RL, Spike-MD, Ribosembly, Tomotwin-100).
• Simulate cryo-EM forward model to generate images with known poses, conformations, and imaging parameters.
• Evaluate seven fixed-pose and three ab initio heterogeneous reconstruction methods on CryoBench datasets.
• Introduce embedding and volume evaluation metrics (Neighborhood Similarity, Information Imbalance, ARI/AMI, Per-Conformation FSC, AUC-FSC).
• Provide dataset generation and evaluation software and public data releases for reproducibility and extension.
• Compare methods across datasets and noise levels to reveal strengths/limits of current approaches.

Figure 1: Overview of CryoBench. (a) In cryo-EM, each image $X_{i}$ captures a molecule $V_{i}$ projected at an unknown pose $\phi_{i}$ . A latent variable $z_{i}$ models the conformational space $\mathcal{V}$ that describes the heterogeneity between molecules $\{V_{i}\}$ . (b) CryoBench includes 5

Experimental results

Research questions

• RQ1 How do contemporary heterogeneous cryo-EM reconstruction methods perform across diverse sources of heterogeneity (conformational vs. compositional) and varying noise levels?
• RQ2 What metrics best capture latent space organization, disentanglement, and reconstruction quality for heterogeneous cryo-EM data?
• RQ3 Can ground-truth synthetic benchmarks enable fair, cross-method comparisons and drive improvements in ab initio reconstruction?
• RQ4 Which methods scale to large, complex assemblies and molecular dynamics-derived motions?
• RQ5 What are the limitations of current benchmarks and how can they be extended to reflect real-data challenges?

Key findings

• CryoBench datasets expose differences in method performance across heterogeneity types and noise regimes.
• Embedding-based metrics (Neighborhood Similarity, Information Imbalance) reveal how well methods capture latent structure and disentangle heterogeneity.
• Per-Conformation FSC provides a joint measure of conformation estimation and reconstruction quality, aligning with qualitative observations.
• RECOVAR often yields latent-space structures most consistent with ground truth manifold and strong FSC performance on several datasets.
• Ab initio methods generally struggle on large, diverse or high-complexity datasets (e.g., Tomotwin-100, Spike-MD) while fixed-pose methods show varying success depending on dataset.
• CryoBench offers a scalable, ground-truth benchmark suite and tools that enable fair comparison and guide future method development in cryo-EM and ML.

Figure 2: IgG-1D results. (a) Dataset design with 100 ground truth (G.T.) conformations producing a circular motion. (b) A representative volume for each method with the G.T. volume. (c) Latent embeddings visualized by UMAP and colored by the G.T. dihedral angle parameterizing the circular motion. (