[Paper Review] HoloHisto: End-to-end Gigapixel WSI Segmentation with 4K Resolution Sequential Tokenization

Yucheng Tang, Yufan He+12 more|arXiv (Cornell University)|Jul 3, 2024

CCD and CMOS Imaging Sensors4 citations

TL;DR

HoloHisto proposes an end-to-end framework for gigapixel whole slide image (WSI) segmentation using 4K-resolution sequential tokenization, enabling direct input/output of full WSIs without patching. It achieves state-of-the-art performance with a 4K patch sampler and a two-stage ViT with ReLU linear attention, outperforming prior models by up to 8.8% in Dice score on a new WSI-level kidney pathology benchmark.

ABSTRACT

In digital pathology, the traditional method for deep learning-based image segmentation typically involves a two-stage process: initially segmenting high-resolution whole slide images (WSI) into smaller patches (e.g., 256x256, 512x512, 1024x1024) and subsequently reconstructing them to their original scale. This method often struggles to capture the complex details and vast scope of WSIs. In this paper, we propose the holistic histopathology (HoloHisto) segmentation method to achieve end-to-end segmentation on gigapixel WSIs, whose maximum resolution is above 80,000$ imes$70,000 pixels. HoloHisto fundamentally shifts the paradigm of WSI segmentation to an end-to-end learning fashion with 1) a large (4K) resolution base patch for elevated visual information inclusion and efficient processing, and 2) a novel sequential tokenization mechanism to properly model the contextual relationships and efficiently model the rich information from the 4K input. To our best knowledge, HoloHisto presents the first holistic approach for gigapixel resolution WSI segmentation, supporting direct I/O of complete WSI and their corresponding gigapixel masks. Under the HoloHisto platform, we unveil a random 4K sampler that transcends ultra-high resolution, delivering 31 and 10 times more pixels than standard 2D and 3D patches, respectively, for advancing computational capabilities. To facilitate efficient 4K resolution dense prediction, we leverage sequential tokenization, utilizing a pre-trained image tokenizer to group image features into a discrete token grid. To assess the performance, our team curated a new kidney pathology image segmentation (KPIs) dataset with WSI-level glomeruli segmentation from whole mouse kidneys. From the results, HoloHisto-4K delivers remarkable performance gains over previous state-of-the-art models.

Motivation & Objective

• To address the limitation of patch-based WSI segmentation that restricts context and fails to capture fine-scale tissue relationships.
• To enable end-to-end training and inference on full gigapixel WSIs without pre-defined tiling or cropping strategies.
• To develop a scalable, efficient architecture capable of modeling ultra-high-resolution (4K) images with rich contextual and local detail.
• To create a new gold-standard WSI-level segmentation dataset (KPIS) for benchmarking histopathology models.
• To demonstrate the superiority of 4K-resolution sequential tokenization and ReLU-based attention for dense prediction in pathology imaging.

Proposed method

• HoloHisto employs a 4K-resolution patch sampler that captures 31× more pixels than standard 512×512 patches, enabling richer visual context.
• It uses a pre-trained image tokenizer to convert 4K image features into a discrete token grid, reducing dimensionality while preserving perceptual structure.
• A two-stage Vision Transformer (ViT) architecture with multi-scale attention and ReLU-based linear attention replaces standard softmax attention to reduce computational cost.
• The framework integrates cuCIM and MONAI to enable real-time, dynamic WSI loading and online data augmentation at various magnification levels.
• The model is trained and inferred end-to-end on full WSIs, directly producing gigapixel segmentation masks without reconstruction.
• A novel random 4K sampler enables efficient data loading and training by sampling from any region of interest in the WSI.

Experimental results

Research questions

• RQ1 Can a holistic, end-to-end approach achieve superior segmentation performance on gigapixel WSIs compared to patch-based methods?
• RQ2 Does 4K-resolution sequential tokenization significantly improve context modeling and feature representation over standard 256×256 or 512×512 patches?
• RQ3 Can a ReLU-based linear attention mechanism effectively replace standard softmax attention in ultra-high-resolution image modeling with reduced computational cost?
• RQ4 To what extent does direct WSI input/output with a 4K patch strategy improve segmentation accuracy on dense, small-object tasks like glomeruli detection?
• RQ5 Can a new WSI-level benchmark dataset (KPIS) support the development and evaluation of next-generation histopathology segmentation models?

Key findings

• HoloHisto-4K achieves a 4K patch Dice score of 88.85, outperforming the next-best method (SAM ViT-H multi-scale) by 0.87 points.
• On the full WSI level, HoloHisto-4K achieves a Dice score of 86.11, significantly outperforming baseline models that rely on tiling and reconstruction.
• The ablation study shows that combining the tokenizer with ReLU attention increases Dice score by 1.85 points over the baseline (Linear Proj + MHSA) on 4K patches.
• Using a random 4K sampler instead of tiling improves WSI-level Dice by 3.67 points, demonstrating the advantage of end-to-end learning.
• On the NEPTUNE dataset, HoloHisto achieves higher Dice scores than baselines at all tissue scales, with consistent gains at 3000×3000 resolution.
• The model successfully generates complete gigapixel segmentation masks in a single forward pass, demonstrating true end-to-end capability.