[Paper Review] Unsupervised Instance Segmentation in Microscopy Images via Panoptic Domain Adaptation and Task Re-weighting

Dongnan Liu, Donghao Zhang+6 more|arXiv (Cornell University)|May 5, 2020

AI in cancer detection49 references4 citations

TL;DR

This paper proposes CyC-PDAM, a novel unsupervised domain adaptation framework for nuclei instance segmentation in histopathology images using fluorescence microscopy as the source domain. By integrating nuclei inpainting, panoptic-level domain adaptation via a semantic branch, and dynamic task re-weighting, the method achieves state-of-the-art performance, matching fully supervised methods on benchmark datasets without requiring target domain annotations.

ABSTRACT

Unsupervised domain adaptation (UDA) for nuclei instance segmentation is important for digital pathology, as it alleviates the burden of labor-intensive annotation and domain shift across datasets. In this work, we propose a Cycle Consistency Panoptic Domain Adaptive Mask R-CNN (CyC-PDAM) architecture for unsupervised nuclei segmentation in histopathology images, by learning from fluorescence microscopy images. More specifically, we first propose a nuclei inpainting mechanism to remove the auxiliary generated objects in the synthesized images. Secondly, a semantic branch with a domain discriminator is designed to achieve panoptic-level domain adaptation. Thirdly, in order to avoid the influence of the source-biased features, we propose a task re-weighting mechanism to dynamically add trade-off weights for the task-specific loss functions. Experimental results on three datasets indicate that our proposed method outperforms state-of-the-art UDA methods significantly, and demonstrates a similar performance as fully supervised methods.

Motivation & Objective

• To address the high annotation cost and domain shift in histopathology image analysis by enabling unsupervised instance segmentation without target domain annotations.
• To reduce domain bias in nuclei instance segmentation by jointly adapting features at the semantic and instance levels.
• To mitigate source-domain bias in training by dynamically re-weighting task-specific loss functions.
• To improve the realism of synthesized target-domain images by removing auxiliary nuclei via a novel inpainting mechanism.
• To achieve panoptic-level domain adaptation that preserves spatial relationships between foreground and background.

Proposed method

• Proposes a nuclei inpainting mechanism to remove spurious nuclei artifacts in synthesized histopathology images, improving segmentation fidelity.
• Introduces a semantic branch with a domain discriminator to learn domain-invariant features at the panoptic level, capturing foreground-background relationships.
• Adapts the CyCADA framework to Mask R-CNN for instance segmentation, enabling cycle-consistent domain adaptation across appearance, image, and instance levels.
• Employs a task re-weighting mechanism that dynamically adjusts loss weights based on feature domain invariance, reducing source bias during training.
• Uses a multi-branch architecture combining instance-level detection and semantic-level segmentation heads for joint optimization.
• Leverages cycle-consistent translation to generate realistic target-like images from source fluorescence microscopy data.

Experimental results

Research questions

• RQ1 Can unsupervised domain adaptation effectively close the domain gap between fluorescence microscopy and histopathology images for nuclei instance segmentation?
• RQ2 How does panoptic-level domain adaptation—simultaneously addressing semantic and instance-level domain shifts—improve segmentation performance?
• RQ3 To what extent does dynamic task re-weighting reduce source-domain bias in model optimization?
• RQ4 Can a method trained without target annotations achieve performance comparable to fully supervised models?
• RQ5 How do individual components (inpainting, semantic branch, task re-weighting) contribute to overall performance gains?

Key findings

• The proposed CyC-PDAM method outperforms state-of-the-art unsupervised domain adaptation methods on three public datasets, demonstrating significant performance gains.
• On the Kumar dataset, CyC-PDAM achieves a pixel-level F1-score of 0.887, matching the performance of the fully supervised Panoptic FPN model.
• The method achieves a mean AJI of 0.728 on the test set, slightly below Panoptic FPN but significantly outperforming fully supervised CNN3 and DIST.
• Ablation studies show that removing any of the three core components (inpainting, semantic branch, task re-weighting) reduces performance by 4–6% across metrics, with p-values < 0.05, confirming their statistical significance.
• The model generalizes better to unseen organs in the Kumar dataset, indicating strong domain-invariant feature learning.
• Visualizations confirm that the task re-weighting mechanism reduces source-biased predictions, such as irregularly sized nuclei and false positives, compared to the baseline without re-weighting.