[Paper Review] CellTrack R-CNN: A Novel End-To-End Deep Neural Network for Cell Segmentation and Tracking in Microscopy Images

Yuqian Chen, Yang Song+5 more|arXiv (Cornell University)|Feb 20, 2021

Video Surveillance and Tracking Methods16 references4 citations

TL;DR

CellTrack R-CNN proposes an end-to-end deep learning framework that unifies cell segmentation and tracking in microscopy images by integrating a Siamese network with Mask R-CNN, enhancing tracking via learned relative position encodings. It achieves state-of-the-art performance on the DeepCell benchmark with 97.14% detection, 85.26% segmentation, and 97.05% tracking accuracy.

ABSTRACT

Cell segmentation and tracking in microscopy images are of great significance to new discoveries in biology and medicine. In this study, we propose a novel approach to combine cell segmentation and cell tracking into a unified end-to-end deep learning based framework, where cell detection and segmentation are performed with a current instance segmentation pipeline and cell tracking is implemented by integrating Siamese Network with the pipeline. Besides, tracking performance is improved by incorporating spatial information into the network and fusing spatial and visual prediction. Our approach was evaluated on the DeepCell benchmark dataset. Despite being simple and efficient, our method outperforms state-of-the-art algorithms in terms of both cell segmentation and cell tracking accuracies.

Motivation & Objective

• To develop an end-to-end deep learning framework that jointly performs cell segmentation and tracking without post-processing.
• To address the challenge of tracking cells with similar appearance and shape deformations in low-quality microscopy images.
• To improve tracking accuracy by incorporating spatial context through relative position encodings of neighboring cells.
• To demonstrate superior performance compared to state-of-the-art methods on the DeepCell benchmark dataset.

Proposed method

• Integrates a Siamese network branch into the Mask R-CNN architecture to enable end-to-end cell tracking across consecutive frames.
• Uses relative position encoding to represent spatial context: the x- and y-coordinate differences between a target cell and its n nearest neighbors.
• Normalizes spatial feature vectors and fuses them with visual features in the tracking head for improved instance matching.
• Trains the entire network end-to-end using a mini-batch of two adjacent frames, with ResNet-50-FPN as the backbone and SGD optimizer.
• Employs a tracking head with shared weights in the Siamese network to compute similarity scores between feature embeddings of cells in consecutive frames.
• Applies a weighted fusion of visual and spatial features using a learnable parameter α during inference, tuned to 0.1 on the validation set.

Experimental results

Research questions

• RQ1 Can an end-to-end deep learning framework effectively unify cell segmentation and tracking in microscopy images without post-processing?
• RQ2 How does incorporating spatial context via relative position encodings improve cell tracking performance in challenging microscopy conditions?
• RQ3 Does the fusion of visual and spatial features lead to better tracking accuracy compared to methods relying solely on visual features?
• RQ4 How does the proposed method compare to state-of-the-art approaches in terms of segmentation, detection, and tracking performance?

Key findings

• CellTrack R-CNN achieves 97.14% detection accuracy, 85.26% segmentation accuracy, and 97.05% tracking accuracy on the DeepCell benchmark dataset.
• The method reduces the difference between detection and tracking accuracy to 0.09, indicating minimal degradation in tracking performance relative to detection.
• Ablation studies confirm that the spatial tracking head significantly improves tracking performance, with a 2.74-point drop in D-T (detection-tracking difference) when spatial features are excluded.
• The integration of spatial information via relative position encodings leads to more robust tracking, especially in cases of high appearance similarity and shape deformation.
• The framework outperforms both FR-Ro-GE and DeepCell in all three metrics—segmentation, detection, and tracking—demonstrating state-of-the-art performance.
• The method maintains simplicity and efficiency, achieving superior results with only a small increase in model and computational complexity over standard Mask R-CNN.