[Paper Review] Large-Scale Multi-Hypotheses Cell Tracking Using Ultrametric Contours Maps

Jordão Bragantini, Merlin Lange+1 more|arXiv (Cornell University)|Aug 8, 2023

Immunotherapy and Immune Responses4 citations

TL;DR

This paper proposes a scalable, large-scale 3D cell tracking method that jointly optimizes segmentation and tracking using ultrametric contour maps (UCMs) and a novel integer linear programming (ILP) formulation. By leveraging hierarchical segmentation hypotheses and maximizing overlap between adjacent frames, the method achieves state-of-the-art performance on the Cell Tracking Challenge and scales to terabyte-scale datasets with over 21 million cell instances, outperforming prior ILP-based approaches in speed and efficiency while supporting both deep learning and traditional segmentation models.

ABSTRACT

In this work, we describe a method for large-scale 3D cell-tracking through a segmentation selection approach. The proposed method is effective at tracking cells across large microscopy datasets on two fronts: (i) It can solve problems containing millions of segmentation instances in terabyte-scale 3D+t datasets; (ii) It achieves competitive results with or without deep learning, which requires 3D annotated data, that is scarce in the fluorescence microscopy field. The proposed method computes cell tracks and segments using a hierarchy of segmentation hypotheses and selects disjoint segments by maximizing the overlap between adjacent frames. We show that this method achieves state-of-the-art results in 3D images from the cell tracking challenge and has a faster integer linear programming formulation. Moreover, our framework is flexible and supports segmentations from off-the-shelf cell segmentation models and can combine them into an ensemble that improves tracking. The code is available https://github.com/royerlab/ultrack.

Motivation & Objective

• To address the challenge of large-scale 3D cell tracking in terabyte-scale time-lapse microscopy datasets with millions of segmentation instances.
• To develop a joint segmentation and tracking framework that reduces dependency on high-quality 3D annotated data, which is scarce in fluorescence microscopy.
• To enable efficient, out-of-memory processing of large datasets using a computationally efficient ILP formulation with biological constraints (e.g., cell division into two).
• To support integration of multiple segmentation models—both deep learning and traditional—into an ensemble to improve tracking robustness.
• To provide a flexible, scalable solution that maintains high segmentation quality and tracking accuracy without requiring extensive ground-truth annotations.

Proposed method

• The method constructs a hierarchy of segmentation hypotheses using ultrametric contour maps (UCMs), representing nested segmentations across multiple scales.
• It formulates the selection of disjoint segments across frames as an integer linear program (ILP), maximizing overlap between adjacent frames while enforcing biological constraints such as cell division into two.
• The ILP formulation is optimized using a novel, faster formulation that scales sublinearly with problem size, with a slope of 1.37 ± 0.06 in runtime vs. time window size.
• The framework supports integration of off-the-shelf segmentation models (e.g., U-Net) and combines their outputs into an ensemble via UCM-based fusion.
• The algorithm processes data in a distributed, out-of-core manner using SLURM on a compute cluster, enabling processing of 3.4 TB datasets with 791×448×2174×2423 voxel dimensions.
• A two-pass tracking strategy is employed: first pass for initial track assignment, second pass for refinement, with GPU-accelerated contour detection and CPU-based processing for scalability.

Figure 1 : Example of hierarchy filtering by minimum size, maximum size, and contour strength; dashed lines show deleted elements. Leaves of the hierarchy are not shown.

Experimental results

Research questions

• RQ1 Can a joint segmentation and tracking framework based on hierarchical UCMs achieve state-of-the-art performance on large-scale 3D microscopy datasets without relying on expensive 3D annotations?
• RQ2 How does the proposed ILP formulation compare in efficiency and scalability to existing ILP-based tracking methods when processing increasingly large time windows?
• RQ3 To what extent can the method leverage diverse segmentation models—including deep learning and traditional methods—through ensemble fusion in the UCM space?
• RQ4 Can the method scale to real-world, terabyte-scale 3D+t datasets with over 20 million cell instances while maintaining high accuracy and computational feasibility?
• RQ5 Does the use of UCMs enable better robustness and flexibility compared to standard discrete-label segmentation in complex, dense cell tracking scenarios?

Key findings

• The method achieved state-of-the-art performance on two out of four datasets in the Cell Tracking Challenge benchmark, demonstrating strong generalization across diverse biological datasets.
• The proposed ILP formulation achieved a runtime slope of 1.37 ± 0.06 per unit time window, significantly outperforming Türetken et al. (1.75 ± 0.12) and Malin-Mayor et al. (1.84 ± 0.06), indicating superior scalability.
• The method successfully processed a 3.4 TB zebrafish embryo dataset with 21.5 million cell instances, completing tracking in approximately 9.5 hours of distributed computing on 100 CPU cores and 20 GPUs.
• The algorithm achieved high-quality segmentation and tracking results, with visualizations showing accurate 3D rendering and slice-wise segmentation across maximum intensity projections, YX, and ZY planes.
• The framework demonstrated robustness by integrating multiple segmentation models—including a U-Net trained on pseudo-labels from watershed segmentation—into a unified UCM-based ensemble.
• The method reduced dependency on high-quality 3D annotations by jointly optimizing segmentation and tracking, improving performance even when ground-truth data was unavailable.

Figure 2 : Example of linkage between two hierarchies and their weights matrix $w_{..}$ which includes all $w_{p,q}$ . Solution considering $w_{..}$ in blue and in red solution of $w^{p}_{..}$ when $p>=2$ .