[Paper Review] Joint Multi-frame Detection and Segmentation for Multi-cell Tracking

Zibin Zhou, Fei Wang+4 more|arXiv (Cornell University)|Jun 26, 2019

Cell Image Analysis Techniques12 references4 citations

TL;DR

This paper proposes a joint multi-frame detection and segmentation framework for multi-cell tracking, leveraging a UNet with multi-frame input to enhance mitotic cell detection and a secondary UNet for primary segmentation. By fusing detection and segmentation results, it achieves state-of-the-art performance on the Fluo-Hela dataset, particularly in segmentation accuracy and combined tracking performance.

ABSTRACT

Tracking living cells in video sequence is difficult, because of cell morphology and high similarities between cells. Tracking-by-detection methods are widely used in multi-cell tracking. We perform multi-cell tracking based on the cell centroid detection, and the performance of the detector has high impact on tracking performance. In this paper, UNet is utilized to extract inter-frame and intra-frame spatio-temporal information of cells. Detection performance of cells in mitotic phase is improved by multi-frame input. Good detection results facilitate multi-cell tracking. A mitosis detection algorithm is proposed to detect cell mitosis and the cell lineage is built up. Another UNet is utilized to acquire primary segmentation. Jointly using detection and primary segmentation, cells can be fine segmented in highly dense cell population. Experiments are conducted to evaluate the effectiveness of our method, and results show its state-of-the-art performance.

Motivation & Objective

• To improve multi-cell tracking in highly dense and morphologically variable populations where cells are visually similar and often undergo mitosis.
• To address the challenge of tracking small, densely packed cells by enhancing detection robustness using multi-frame temporal context.
• To develop a fine segmentation method that leverages both detection and segmentation outputs to resolve overlapping or closely spaced cells.
• To achieve state-of-the-art performance on benchmark datasets, particularly in segmentation and combined tracking accuracy.

Proposed method

• A multi-frame UNet is used for cell centroid detection, incorporating spatio-temporal features across three consecutive frames to improve detection of mitotic cells.
• A separate UNet is employed for primary cell segmentation, generating initial instance-level cell masks from image sequences.
• A mitosis detection algorithm is applied to detect cell division events based on improved centroid detection, enabling lineage tracking.
• Fine segmentation is performed by combining primary detection and segmentation outputs, using tracking consistency and spatial proximity to resolve individual cells.
• The framework integrates detection and segmentation results through a multi-step tracker that assigns identities across frames using spatial and temporal priors.
• The method is trained and evaluated on the Cell Tracking Benchmark, using metrics including TRA, SEG, and OP CTB.

Experimental results

Research questions

• RQ1 Can multi-frame input improve detection performance for mitotic cells in dense cell populations?
• RQ2 How does joint use of detection and segmentation enhance fine-grained cell separation in highly crowded fields?
• RQ3 To what extent does spatio-temporal modeling via multi-frame UNet improve tracking robustness compared to single-frame baselines?
• RQ4 Can the proposed method achieve state-of-the-art performance on benchmark datasets like Fluo-Hela and Phc-PSC?
• RQ5 What is the impact of primary segmentation quality on final fine segmentation and tracking accuracy?

Key findings

• On the Fluo-Hela dataset, the method achieves a new state-of-the-art performance with 94.2% OP CTB and 91.9% SEG accuracy, ranking 1st in SEG and 3rd in OP CTB.
• The method achieves 95.9% TRA on Phc-PSC, ranking 1st in tracking accuracy, demonstrating strong performance on challenging dense cell tracking.
• Mitosis detection F1 score improves from 41.1% (single frame) to 61.5% (three-frame input) on Phc-PSC, indicating significant benefit from temporal context.
• On Fluo-SIM+ and Fluo-GOWT1, performance is less optimal, suggesting limitations in handling low-contrast or highly variable imaging conditions.
• Fine segmentation successfully resolves overlapping cells and maintains accuracy even when cells partially disappear or have low contrast to background.
• The method outperforms established baselines such as TUG-AT and BGU-IL, particularly in segmentation and combined performance metrics.