[Paper Review] Unsupervised Dense Nuclei Detection and Segmentation with Prior Self-activation Map For Histology Images

Pingyi Chen, Chenglu Zhu+5 more|arXiv (Cornell University)|Oct 14, 2022

AI in cancer detection4 citations

TL;DR

This paper proposes an unsupervised deep learning framework for dense nuclei detection and segmentation in histology images using a Prior Self-activation Module (PSM) that generates pseudo masks from self-supervised features without any manual annotations. By leveraging gradient-based activation maps from shallow network layers and a semantic-guided generator, the method achieves competitive performance on two public pathological datasets, outperforming fully- and weakly-supervised baselines in the absence of labels.

ABSTRACT

The success of supervised deep learning models in medical image segmentation relies on detailed annotations. However, labor-intensive manual labeling is costly and inefficient, especially in dense object segmentation. To this end, we propose a self-supervised learning based approach with a Prior Self-activation Module (PSM) that generates self-activation maps from the input images to avoid labeling costs and further produce pseudo masks for the downstream task. To be specific, we firstly train a neural network using self-supervised learning and utilize the gradient information in the shallow layers of the network to generate self-activation maps. Afterwards, a semantic-guided generator is then introduced as a pipeline to transform visual representations from PSM to pixel-level semantic pseudo masks for downstream tasks. Furthermore, a two-stage training module, consisting of a nuclei detection network and a nuclei segmentation network, is adopted to achieve the final segmentation. Experimental results show the effectiveness on two public pathological datasets. Compared with other fully-supervised and weakly-supervised methods, our method can achieve competitive performance without any manual annotations.

Motivation & Objective

• To address the high cost and inefficiency of manual annotation in dense nuclei segmentation of histopathology images.
• To develop a self-supervised framework that eliminates the need for pixel-level or instance-level annotations.
• To generate reliable pseudo masks for downstream segmentation using low-level features from a self-supervised network.
• To enable end-to-end unsupervised detection and segmentation of densely distributed nuclei in whole-slide images.

Proposed method

• Proposes a Prior Self-activation Module (PSM) that generates self-activation maps using gradient information from shallow layers of a self-supervised network, focusing on cell-level structures without annotations.
• Introduces a semantic-guided generator (SGG) to transform PSM-derived features into pixel-level pseudo masks for downstream tasks.
• Employs a two-stage training pipeline: a nuclei detection network (NDN) for instance localization and a nuclei segmentation network (NSN) for dense pixel-wise segmentation.
• Utilizes self-supervised pre-training via contrastive learning (e.g., SimCLR-style) on unlabeled data to learn robust representations before pseudo mask generation.
• Applies joint training with both pixel-level and object-level supervision via pseudo masks, with a hyperparameter λ controlling the balance between supervision types.
• Leverages class activation map (CAM)-like attention from shallow layers to enhance localization of nuclei boundaries despite lacking ground-truth labels.

Experimental results

Research questions

• RQ1 Can self-activation maps from a self-supervised network’s shallow layers serve as effective pseudo supervision for unsupervised nuclei segmentation?
• RQ2 How effective is the proposed Prior Self-activation Module (PSM) in generating meaningful activation patterns for dense, small objects like nuclei without annotations?
• RQ3 To what extent does joint training with pixel-level and object-level pseudo masks improve segmentation performance in the absence of real annotations?
• RQ4 Can the proposed framework generalize to other biomedical image types beyond histology, such as fundus or cytology images?

Key findings

• The proposed unsupervised method achieves competitive performance on two public pathological datasets—comparable to fully-supervised and weakly-supervised methods—despite using no manual annotations.
• The joint training mechanism with both pixel-level and object-level pseudo masks significantly improves segmentation metrics, with IoU increasing from 0.501 (w/o object-level) to 0.610 (with λ=0.5).
• The model’s performance is robust to the hyperparameter λ, with minimal metric changes across λ values from 0.5 to 10, indicating stability in pseudo mask weighting.
• The Prior Self-activation Module (PSM) generates meaningful activation maps even in complex scenarios, such as fundus images, where it clearly highlights optic disc and cup regions.
• The method generalizes to other image types, including chest X-rays and cytology images, though performance degrades in natural images with multiple salient objects due to ambiguous attention allocation.
• Visualization results confirm that PSM effectively highlights nuclei boundaries and subcellular structures, demonstrating its potential for unsupervised biomedical image analysis.