[Paper Review] The Multi-modality Cell Segmentation Challenge: Towards Universal Solutions

Jun Ma, Ronald Xie+38 more|arXiv (Cornell University)|Aug 10, 2023

Cell Image Analysis TechniquesBiochemistry, Genetics and Molecular Biology55 references15 citations

TL;DR

The paper introduces a NeurIPS multi-modality cell segmentation benchmark and shows Transformer-based models achieve superior generalization across diverse microscopy images, with public Dockerized implementations and Napari integration.

ABSTRACT

Cell segmentation is a critical step for quantitative single-cell analysis in microscopy images. Existing cell segmentation methods are often tailored to specific modalities or require manual interventions to specify hyper-parameters in different experimental settings. Here, we present a multi-modality cell segmentation benchmark, comprising over 1500 labeled images derived from more than 50 diverse biological experiments. The top participants developed a Transformer-based deep-learning algorithm that not only exceeds existing methods but can also be applied to diverse microscopy images across imaging platforms and tissue types without manual parameter adjustments. This benchmark and the improved algorithm offer promising avenues for more accurate and versatile cell analysis in microscopy imaging.

Motivation & Objective

• Motivate development of universal, automatic cell segmentation across varied microscopy modalities without manual tuning.
• Provide a large, diverse benchmark dataset spanning multiple imaging platforms and tissue types.
• Evaluate generalist algorithms and identify effective architectural and data-augmentation strategies.
• Promote accessibility and reproducibility via open-source code, Napari integration, and Docker containers.

Proposed method

• Assemble a diverse training set (1000 labeled + 1500 unlabeled images) plus a 100-image tuning set and a 422-image holdout test set.
• Evaluate algorithms via a Docker-based submission system on a standardized platform with timing and accuracy metrics.
• Use Transformer-based architectures (e.g., SegFormer with multiscale attention) with multi-head outputs for distance/semantic maps to enable instance segmentation.
• Incorporate cell memory replay to mitigate catastrophic forgetting during fine-tuning on new data.
• Explore efficient inference via slide-window/patch-based processing to handle whole-slide images.
• Provide open-source code and Napari integration to facilitate practical adoption by biologists.

Experimental results

Research questions

• RQ1 Can a single model accurately segment cells across brightfield, fluorescence, phase-contrast, and DIC images without manual parameter tuning?
• RQ2 Do Transformer-based architectures generalize better than CNN-based approaches for universal cell segmentation across modalities?
• RQ3 What data augmentation, pre-training, and memory replay strategies most improve cross-domain generalization?
• RQ4 How does model efficiency (inference time and memory) trade off with accuracy in a universal segmentation setting?
• RQ5 Are the top-performing methods generalizable to unseen tissues/cell types and whole-slide images?

Key findings

• Top 3 algorithms achieve median F1 scores of 89.7%, 84.4%, and 84.4% on the holdout set.
• The winning method T1-osilab attains high accuracy with reduced outliers and about 2 seconds per 1000×1000 image inference time.
• Transformer-based models substantially outperform state-of-the-art generalist and CNN-based methods on this diverse benchmark.
• Cell memory replay helps mitigate catastrophic forgetting observed with fine-tuned Cellpose 2.0.
• Algorithms are made publicly available with Napari integration and Docker containers to ease practical use.