[Paper Review] Adaptive Supervised PatchNCE Loss for Learning H&E-to-IHC Stain Translation with Inconsistent Groundtruth Image Pairs

Fangda Li, Zhiqiang Hu+2 more|arXiv (Cornell University)|Mar 10, 2023

AI in cancer detection4 citations

TL;DR

This paper proposes Adaptive Supervised PatchNCE (ASP), a novel contrastive loss for H&E-to-IHC stain translation that mitigates noise from inconsistent ground-truth image pairs by adaptively down-weighting dissimilar patch pairs. The method improves performance across multiple IHC stains using a new public dataset, MIST, and achieves state-of-the-art results on HER2, Ki67, ER, and PR stains with significant gains in SSIM, FID, and perceptual quality.

ABSTRACT

Immunohistochemical (IHC) staining highlights the molecular information critical to diagnostics in tissue samples. However, compared to H&E staining, IHC staining can be much more expensive in terms of both labor and the laboratory equipment required. This motivates recent research that demonstrates that the correlations between the morphological information present in the H&E-stained slides and the molecular information in the IHC-stained slides can be used for H&E-to-IHC stain translation. However, due to a lack of pixel-perfect H&E-IHC groundtruth pairs, most existing methods have resorted to relying on expert annotations. To remedy this situation, we present a new loss function, Adaptive Supervised PatchNCE (ASP), to directly deal with the input to target inconsistencies in a proposed H&E-to-IHC image-to-image translation framework. The ASP loss is built upon a patch-based contrastive learning criterion, named Supervised PatchNCE (SP), and augments it further with weight scheduling to mitigate the negative impact of noisy supervision. Lastly, we introduce the Multi-IHC Stain Translation (MIST) dataset, which contains aligned H&E-IHC patches for 4 different IHC stains critical to breast cancer diagnosis. In our experiment, we demonstrate that our proposed method outperforms existing image-to-image translation methods for stain translation to multiple IHC stains. All of our code and datasets are available at https://github.com/lifangda01/AdaptiveSupervisedPatchNCE.

Motivation & Objective

• To address the challenge of inconsistent H&E-IHC ground-truth image pairs due to tissue sectioning and staining variability in histopathology.
• To develop a robust learning framework that leverages imperfect but diagnostically correlated H&E-IHC pairs for accurate stain translation.
• To improve image-to-image translation performance despite noisy supervision from non-pixel-perfect image pairs.
• To introduce a new public dataset, MIST, with aligned H&E-IHC patches for four key breast cancer IHC stains.

Proposed method

• Proposes Adaptive Supervised PatchNCE (ASP), a patch-based contrastive loss built on Supervised PatchNCE (SP), which learns to align feature embeddings between H&E and IHC patches.
• Introduces adaptive weighting to the contrastive loss based on local similarity between generated and ground-truth IHC patch embeddings, reducing influence of inconsistent regions.
• Employs a weight scheduling strategy that dynamically adjusts loss contribution based on patch-level similarity, improving robustness to noise.
• Uses a ResNet-6Blocks generator and PatchGAN discriminator in a GAN-based image-to-image translation framework.
• Applies random 512×512 crops during training with batch size one and Adam optimizer with linear decay learning rate scheduling.
• Leverages perceptual and structural metrics (SSIM, PHV, FID, KID) for comprehensive evaluation.

Figure 1 : Illustration of the PatchNCE loss from [ 11 ] and the Supervised PatchNCE (SP) loss. The patch embeddings $\boldsymbol{z}$ are extracted by a shared network $F$ .

Experimental results

Research questions

• RQ1 Can a contrastive learning-based loss effectively handle inconsistent H&E-IHC ground-truth pairs in stain translation?
• RQ2 How does adaptive weighting based on patch similarity improve model robustness to noisy supervision?
• RQ3 To what extent can the proposed ASP loss outperform existing image-to-image translation methods on multiple IHC stains?
• RQ4 Does the proposed method generalize across different IHC stains, including HER2, Ki67, ER, and PR?
• RQ5 Can a new public dataset with aligned H&E-IHC patches for multiple IHC stains enable better benchmarking and reproducibility?

Key findings

• The proposed ASP loss achieves the best performance across all four IHC stains (HER2, Ki67, ER, PR), with the lowest FID and KID scores and highest SSIM and PHV values.
• On the MIST_{HER2} dataset, ASP with linear-top scheduling achieved a FID of 51.2 and KID of 16.7, outperforming the SP baseline (FID 61.9, KID 9.8).
• For MIST_{ER}, the ASP loss reduced FID from 46.4 (SP baseline) to 41.4 with linear-λ scheduling, indicating improved distributional alignment.
• The ASP loss with sigmoid-top scheduling achieved the lowest FID of 45.2 on MIST_{HER2}, demonstrating the effectiveness of adaptive weighting.
• The method significantly improves visual consistency and pathological representation fidelity in generated IHC images, as shown in qualitative comparisons.
• The MIST dataset, containing over 4,000 training and 1,000 test patches per stain, enables reproducible benchmarking for future H&E-to-IHC translation research.

Figure 2 : (a) Input H&E image $\boldsymbol{x}$ , (b) generated IHC image $\boldsymbol{\hat{y}}$ , (c) groundtruth IHC image $\boldsymbol{y}$ , and (d) heat map of the anchor-positive cosine similarities produced by a trained network at corresponding locations: $C_{s}=\boldsymbol{z}_{\boldsymbol{\ha