[Paper Review] HEST-1k: A Dataset for Spatial Transcriptomics and Histology Image Analysis

Guillaume Jaume, Paul Doucet+9 more|arXiv (Cornell University)|Jun 23, 2024

Cancer-related molecular mechanisms research21 citations

TL;DR

HEST-1k presents a large paired dataset of spatial transcriptomics with H&E WSIs and metadata, plus the HEST-Library and HEST-Benchmark for multimodal tissue analysis and foundation-model evaluation.

ABSTRACT

Spatial transcriptomics enables interrogating the molecular composition of tissue with ever-increasing resolution and sensitivity. However, costs, rapidly evolving technology, and lack of standards have constrained computational methods in ST to narrow tasks and small cohorts. In addition, the underlying tissue morphology, as reflected by H&E-stained whole slide images (WSIs), encodes rich information often overlooked in ST studies. Here, we introduce HEST-1k, a collection of 1,229 spatial transcriptomic profiles, each linked to a WSI and extensive metadata. HEST-1k was assembled from 153 public and internal cohorts encompassing 26 organs, two species (Homo Sapiens and Mus Musculus), and 367 cancer samples from 25 cancer types. HEST-1k processing enabled the identification of 2.1 million expression--morphology pairs and over 76 million nuclei. To support its development, we additionally introduce the HEST-Library, a Python package designed to perform a range of actions with HEST samples. We test HEST-1k and Library on three use cases: (1) benchmarking foundation models for pathology (HEST-Benchmark), (2) biomarker exploration, and (3) multimodal representation learning. HEST-1k, HEST-Library, and HEST-Benchmark can be freely accessed at https://github.com/mahmoodlab/hest.

Motivation & Objective

• Provide a large, standardized, multimodal resource linking spatial transcriptomics with H&E-stained WSIs across diverse organs and species.
• Enable reproducible benchmarking and development of foundation models for histology and multimodal tissue analysis.
• Facilitate biomarker discovery and expression-guided multimodal representation learning through curated tasks and tools.

Proposed method

• Assembled 1,108 paired ST and WSI samples from 131 cohorts across 25 organs and two species.
• Unified metadata schema including generic, expression, and histology descriptors.
• Processed histology via tissue segmentation and 224x224 patches around ST spots at 20x magnification; generated 1.5M patches.
• Provided automatic tissue detection and alignment to link ST spots with WSIs.
• Performed nuclear segmentation/classification with CellViT to obtain ~60M nuclei across slides.
• Unified expression data into Anndata/Scanpy-compatible objects with raw counts and alignment to WSIs.
• Introduced HEST-Library to assemble/query HEST-1k and enable HEST-Benchmark execution.
• Implemented automatic alignment and resolution-inference pipelines to standardize cross-dataset mappings.

Experimental results

Research questions

• RQ1 Can a large, diverse paired ST and WSI dataset improve multimodal tissue representation learning and biomarker discovery?
• RQ2 How well do state-of-the-art patch encoders predict gene expression from histology across multiple organs and cancer types?
• RQ3 What is the utility of fine-tuning histology encoders on disease-specific data to improve molecular-status prediction?
• RQ4 Can HEST-1k enable robust benchmarking of foundation models for histology on gene-expression prediction tasks?
• RQ5 How can morphology-derived features correlate with gene expression in tumor regions to aid discovery?

Key findings

• HEST-1k comprises 1,108 samples with 1.5M expression–morphology pairs and 60M nuclei across 25 organs and 2 species.
• HEST-Benchmark reveals varied performance across 10 patch-encoder models for gene-expression prediction from histology, with language-aligned and transformer-based models achieving strong results in several tasks.
• Fine-tuning a patch encoder on disease-specific data (CONCH-FT) improves downstream molecular-status prediction (ER/PR/HER2) on an independent breast cancer cohort.
• Nuclear-size and other morphology features show significant correlations with certain gene expressions (e.g., GATA3 with nuclear area in IDC), illustrating morphomolecular links.
• HS-Collection and HEST-Library provide automated alignment, patching, and data standardization, enabling scalable, reproducible analyses across legacy ST datasets.
• HEST for multimodal learning demonstrates that modality-aligned patch encoders can be further optimized for tissue-specific morphologies and molecular landscapes.