[Paper Review] Fast 3D cell tracking with wide-field fluorescence microscopy through deep learning

Kan Liu, Hui Qiao+4 more|arXiv (Cornell University)|May 14, 2018

Advanced Vision and Imaging1 references3 citations

TL;DR

This paper proposes a deep learning framework using two cascaded convolutional neural networks (CNNs) to enable fast, accurate 3D cell tracking from single 2D wide-field fluorescence microscopy images without hardware modifications. By transforming the ill-posed 3D localization problem into a multi-label classification task, the method achieves robust 3D tracking at 100 fps in zebrafish in vivo, outperforming traditional kernel fitting methods in accuracy and noise resilience.

ABSTRACT

Tracking cells in 3D at high speed continues to attract extensive attention for many biomedical applications, such as monitoring immune cell migration and observing tumor metastasis in flowing blood vessels. Here, we propose a deep convolutional neural networks (CNNs) based method to retrieve the 3D locations of the fluorophores from a single 2D image captured by a conventional wide-field fluorescence microscope without any hardware modification. The reported method converts the challenging 3D localization from an ill-posed model-based fitting problem, especially with dense samples and low signal-to-noise ratio, to a solvable multi-label classification problem through two cascaded CNNs, where deep learning technique has a great advantage over other algorithms. Compared with traditional kernel-fitting methods, the proposed method achieves more accurate and robust localization of multiple objects across a much larger axial range, which is validated by both simulation and experimental results on 3D distributed fluorescent beads. Moreover, in vivo 3D tracking of multiple blood cells in zebrafish at 100 fps further verifies the feasibility of our framework.

Motivation & Objective

• To enable high-speed, accurate 3D cell tracking in live biological samples using standard wide-field fluorescence microscopes without hardware modifications.
• To overcome the limitations of traditional model-based 3D localization methods, which suffer from ill-posed fitting, noise amplification, and poor performance under low signal-to-noise ratio (SNR) and high object density.
• To leverage deep learning to convert the complex 3D localization problem into a solvable multi-label classification task, improving robustness and accuracy.
• To validate the method in both simulated and experimental settings, including in vivo tracking of zebrafish blood cells at 100 fps.
• To demonstrate the generalizability of the framework to diverse optical conditions and potential extension to other imaging modalities and PSF engineering techniques.

Proposed method

• A two-stage cascaded CNN architecture is employed: first, a lateral detection CNN identifies potential object locations within sliding windows of the 2D image.
• Second, an axial localization CNN estimates the 3D depth (z-position) of each detected object based on the defocus pattern in the 2D image.
• The method treats 3D localization as a multi-label classification problem rather than a model-based fitting problem, reducing sensitivity to noise and crosstalk.
• Training data is synthesized using experimental z-stacks of single fluorescent beads to accurately model real-world noise, aberrations, and SNR variations across axial planes.
• The network is trained end-to-end using supervised learning on a large dataset of simulated and real defocus patterns, enabling generalization across diverse imaging conditions.
• The framework is designed to be compatible with any wide-field microscope and can be extended to PSF-engineered systems for enhanced axial range and resolution.

Experimental results

Research questions

• RQ1 Can deep learning effectively transform the ill-posed 3D localization problem in wide-field fluorescence microscopy into a stable, end-to-end classification task?
• RQ2 How does the proposed CNN-based method compare to traditional kernel fitting and maximum-likelihood estimation (MLE) in terms of localization accuracy and robustness under low SNR and high object density?
• RQ3 To what extent can the method achieve high-speed 3D tracking (e.g., 100 fps) in vivo without hardware modifications?
• RQ4 Can the method generalize across different optical conditions, including optical aberrations, refractive index mismatches, and varying noise statistics?
• RQ5 How does using experimentally acquired z-stacks for training data synthesis improve performance compared to purely simulated PSFs?

Key findings

• The proposed method achieves significantly lower localization error (measured by RMSE) and higher Jaccard index in object recognition compared to traditional kernel fitting and MLE methods, especially under low SNR and high object density.
• The method enables 3D tracking at 100 fps in zebrafish, demonstrating feasibility for high-speed in vivo monitoring of dynamic cellular processes such as blood cell migration.
• Using experimental z-stacks for training data synthesis results in substantially better performance than using purely simulated PSFs, particularly in axial localization accuracy across different depths.
• The method achieves a larger effective axial localization range than conventional MLE-based methods due to better modeling of SNR variations across axial planes.
• The framework is robust to optical aberrations and noise, outperforming model-based methods that fail under challenging imaging conditions.
• The approach is generalizable and can be extended to other PSF engineering techniques and imaging modalities, such as bright-field and phase contrast, for broader biological and clinical applications.