[Paper Review] GAN-based Virtual Re-Staining: A Promising Solution for Whole Slide Image Analysis

Zhaoyang Xu, Huang, Xingru+3 more|arXiv (Cornell University)|Jan 13, 2019

Cell Image Analysis Techniques20 references75 citations

TL;DR

The paper proposes a conditional CycleGAN (cCGAN) to translate H&E whole slide images to IHC-like images for virtual staining, incorporating patch-level labels and photorealism/SSIM losses to preserve structure.

ABSTRACT

Histopathological cancer diagnosis is based on visual examination of stained tissue slides. Hematoxylin and eosin (H\&E) is a standard stain routinely employed worldwide. It is easy to acquire and cost effective, but cells and tissue components show low-contrast with varying tones of dark blue and pink, which makes difficult visual assessments, digital image analysis, and quantifications. These limitations can be overcome by IHC staining of target proteins of the tissue slide. IHC provides a selective, high-contrast imaging of cells and tissue components, but their use is largely limited by a significantly more complex laboratory processing and high cost. We proposed a conditional CycleGAN (cCGAN) network to transform the H\&E stained images into IHC stained images, facilitating virtual IHC staining on the same slide. This data-driven method requires only a limited amount of labelled data but will generate pixel level segmentation results. The proposed cCGAN model improves the original network \cite{zhu_unpaired_2017} by adding category conditions and introducing two structural loss functions, which realize a multi-subdomain translation and improve the translation accuracy as well. % need to give reasons here. Experiments demonstrate that the proposed model outperforms the original method in unpaired image translation with multi-subdomains. We also explore the potential of unpaired images to image translation method applied on other histology images related tasks with different staining techniques.

Motivation & Objective

• Motivate the need for virtual staining to overcome low-contrast H&E limitations and reduce reliance on expensive IHC.
• Develop a multi-class unpaired image-to-image translation model to generate virtual IHC from H&E.
• Ensure structural fidelity of the original tissue during translation via specialized loss functions.
• Enable improved tissue segmentation and downstream AI analysis through virtual staining.

Proposed method

• Extend CycleGAN with conditional inputs for tissue classes (multi-subdomain translation).
• Attach patch-wise classifiers to enforce accurate sub-domain representation.
• Introduce photorealism loss and SSIM loss to preserve structure and texture during translation.
• Use a combination of conditional adversarial loss, cycle loss, and classification losses to train generators, discriminators, and classifiers.
• Optimize a weighted loss (L) balancing cGAN, cycle, identity, class, clcyc, SSIM, and photorealism terms (Eq. 11).

Experimental results

Research questions

• RQ1 Can unpaired image-to-image translation produce accurate multi-class virtual staining from H&E to IHC without paired data?
• RQ2 Does adding patch-level class information improve translation quality and sub-domain consistency?
• RQ3 Do photorealism and SSIM losses preserve tissue structure during virtual staining?
• RQ4 What is the impact of the proposed losses on preserving original morphology while achieving correct staining style?

Key findings

• The cCGAN with proposed losses improves virtual staining quality versus CycleGAN, especially for mixed components with limited training data.
• Best-performing settings found: gamma=0.5, alpha=0.5, beta=1.
• Quantitative results show varying accuracy across classes, with TF, HB, HF, TN showing differing challenges due to data balance and complexity.
• Pathologists’ evaluations indicate better content preservation and reduced imaginary artifacts compared to CycleGAN.
• Zero-shot or unsupervised extensions are discussed as future work to reduce reliance on labels.