[Paper Review] Instance-aware Self-supervised Learning for Nuclei Segmentation

Xinpeng Xie, Jiawei Chen+4 more|arXiv (Cornell University)|Jul 22, 2020

AI in cancer detection17 references4 citations

TL;DR

This paper proposes an instance-aware self-supervised learning framework for nuclei instance segmentation that leverages scale-wise triplet learning and count ranking to implicitly learn nuclei size and quantity priors from unlabeled histopathological images. The method achieves a new state-of-the-art average AJI of 70.63% on the MoNuSeg dataset using a self-supervised ResUNet-101, significantly outperforming both training-from-scratch and prior self-supervised approaches.

ABSTRACT

Due to the wide existence and large morphological variances of nuclei, accurate nuclei instance segmentation is still one of the most challenging tasks in computational pathology. The annotating of nuclei instances, requiring experienced pathologists to manually draw the contours, is extremely laborious and expensive, which often results in the deficiency of annotated data. The deep learning based segmentation approaches, which highly rely on the quantity of training data, are difficult to fully demonstrate their capacity in this area. In this paper, we propose a novel self-supervised learning framework to deeply exploit the capacity of widely-used convolutional neural networks (CNNs) on the nuclei instance segmentation task. The proposed approach involves two sub-tasks (i.e., scale-wise triplet learning and count ranking), which enable neural networks to implicitly leverage the prior-knowledge of nuclei size and quantity, and accordingly mine the instance-aware feature representations from the raw data. Experimental results on the publicly available MoNuSeg dataset show that the proposed self-supervised learning approach can remarkably boost the segmentation accuracy of nuclei instance---a new state-of-the-art average Aggregated Jaccard Index (AJI) of 70.63%, is achieved by our self-supervised ResUNet-101. To our best knowledge, this is the first work focusing on the self-supervised learning for instance segmentation.

Motivation & Objective

• Address the challenge of limited annotated histopathological data for nuclei instance segmentation, which hinders deep learning performance.
• Overcome the data scarcity problem by developing a self-supervised pre-training framework that reduces dependency on expensive manual annotations.
• Enable convolutional neural networks to implicitly learn instance-aware features—specifically nuclei size and count—through proxy tasks on raw, unlabeled data.
• Achieve state-of-the-art performance in nuclei instance segmentation without relying on instance-level annotations during pre-training.
• Demonstrate generalization and robustness by validating on both MoNuSeg and CPM datasets, showing consistent improvements over strong baselines.

Proposed method

• Design a self-supervised proxy task composed of two sub-tasks: scale-wise triplet learning and count ranking, to instill instance-aware inductive biases into the network.
• Generate triplet samples (anchor, positive, negative) from patches of histopathological images, where the positive has similar nuclei size and the negative has larger nuclei size than the anchor.
• Formulate the scale-wise triplet loss ($\mathcal{L}_{ST}$) to enforce feature embeddings to be closer for similar-sized nuclei and farther for dissimilar-sized nuclei.
• Introduce the count ranking loss ($\mathcal{L}_{CR}$) to encourage the network to learn relative nucleus count differences between image patches by ranking them based on nucleus density.
• Pre-train a ResUNet-101 backbone on the combined proxy task using the MoNuSeg dataset without any instance-level annotations.
• Fine-tune the pre-trained model on the downstream nuclei instance segmentation task using only a small fraction of annotated data.

Experimental results

Research questions

• RQ1 Can self-supervised proxy tasks that encode nuclei size and count priors improve instance segmentation performance with limited annotations?
• RQ2 How does the proposed instance-aware self-supervised learning framework compare to standard self-supervised methods (e.g., Jigsaw, RotNet, ColorMe) in nuclei instance segmentation?
• RQ3 To what extent does pre-training on the proposed proxy tasks reduce the need for large-scale annotated data in nuclei segmentation?
• RQ4 Does the joint optimization of scale-wise triplet and count ranking losses lead to better feature representations than using either loss alone?
• RQ5 Can the proposed method generalize to other histopathological datasets beyond MoNuSeg?

Key findings

• The proposed self-supervised pre-training strategy achieves a new state-of-the-art average Aggregated Jaccard Index (AJI) of 70.63% on the MoNuSeg 2018 test set, surpassing the previous best method (CIA-Net) by 1.56 percentage points.
• The method improves AJI by +5.34% compared to training from scratch, demonstrating the effectiveness of learning instance-aware features without annotations.
• The ablation study confirms that combining both scale-wise triplet learning and count ranking losses yields the highest performance gain (+5.34% over training from scratch), with individual losses contributing +4.35% and +4.80%, respectively.
• On the CPM dataset, the self-supervised ResUNet-101 achieves an average Dice score of 86.36%, which is comparable to the CPM 2018 competition winner (87.00%).
• The method achieves a +11.43% improvement in AJI when fine-tuned with only 10% of the training annotations, highlighting its data efficiency.
• The framework achieves superior performance with lower computational complexity and fewer parameters than state-of-the-art models like CIA-Net, which uses a DenseNet encoder and dual decoders.