[Paper Review] Multi-stream Cell Segmentation with Low-level Cues for Multi-modality Images

Wei Lou, Xinyi Yu+5 more|arXiv (Cornell University)|Oct 22, 2023

Image Processing Techniques and Applications7 citations

TL;DR

This paper presents a class-wise, multi-stream segmentation framework that first classifies multi-modal cell images by low-level cues, then applies category-specific segmentation models (Stardist for roundish cells and HoverNet for irregular shapes) with an efficient ConvNeXt backbone.

ABSTRACT

Cell segmentation for multi-modal microscopy images remains a challenge due to the complex textures, patterns, and cell shapes in these images. To tackle the problem, we first develop an automatic cell classification pipeline to label the microscopy images based on their low-level image characteristics, and then train a classification model based on the category labels. Afterward, we train a separate segmentation model for each category using the images in the corresponding category. Besides, we further deploy two types of segmentation models to segment cells with roundish and irregular shapes respectively. Moreover, an efficient and powerful backbone model is utilized to enhance the efficiency of our segmentation model. Evaluated on the Tuning Set of NeurIPS 2022 Cell Segmentation Challenge, our method achieves an F1-score of 0.8795 and the running time for all cases is within the time tolerance.

Motivation & Objective

• Classify multi-modal microscopy images into four low-level feature-based categories automatically.
• Train category-specific segmentation models to exploit strengths of different segmentation approaches.
• Enhance efficiency with a ConvNeXt backbone and evaluate end-to-end performance on NeurIPS 2022 Cell Segmentation Challenge data.
• Investigate whether unlabeled data via pseudo-labeling improves segmentation performance.

Proposed method

• Automatic cell image classification via pseudo-label synthesis with a ConvNeXt-Stardist segmentation model to generate labels.
• Unsupervised classification of images into four categories based on low-level cues (binary, gray, large-cell color, small-cell color).
• Train a ResNet18-based classifier to assign testing images to one of four categories.
• Class-wise multi-stream segmentation using ConvNeXt-Stardist for roundish cells (Classes 0, 2, 3) and ConvNeXt-Hover for irregular cells (Class 1).
• Pretrain on all labeled data; fine-tune per-class on corresponding category images.
• Losses for Stardist: CE, Dice, MAE; for Hover: CE, Dice, MSE, MSGE; post-processing with NMS and marker-controlled watershed.
• Inference uses classifier to route test images to the corresponding per-class segmentation model.

Experimental results

Research questions

• RQ1 Can automatic low-level-feature-based clustering of multi-modal microscopy images improve segmentation generalization across modalities?
• RQ2 Does a class-wise, multi-stream segmentation framework outperform single-model approaches on multi-modal cell images?
• RQ3 Is a ConvNeXt backbone with Stardist or HoverNet heads effective for efficient, accurate cell segmentation across shapes?
• RQ4 Does incorporating unlabeled data via pseudo-labels enhance segmentation performance?

Key findings

• The proposed pipeline achieves an overall F1 score of 0.8795 on the tuning set.
• Class-wise finetuning improves segmentation performance by up to 5.81% in F1 on the tuning set.
• Using class-wise models (Stardist for some classes, Hover for Class 1) yields performance gains over pretrained, non-finetuned counterparts.
• Mean F1 on the final tuning set shows improvements: Ours reaches 0.8795 vs pretrained variants.
• Qualitative results indicate strong performance on densely and sparsely distributed cells, with some failures on bright foreground/dark background images.
• Running time is within the allowed time tolerance for all tuning-set cases.