[Paper Review] Dense three dimensional localization microscopy by deep learning

Elias Nehme, Daniel Z. Freedman+4 more|arXiv (Cornell University)|Jun 21, 2019

Advanced Fluorescence Microscopy TechniquesBiochemistry, Genetics and Molecular Biology23 references9 citations

TL;DR

This paper proposes a deep learning approach for dense 3D localization microscopy that accurately determines the 3D positions of densely overlapping fluorescent emitters from their point-spread functions (PSFs). By training a neural network to infer emitter positions from complex, overlapping PSFs across a broad axial range, the method enables high-precision 3D super-resolution imaging, demonstrated in 3D STORM of mitochondria and volumetric imaging of dozens of telomeres in a single snapshot.

ABSTRACT

Localization microscopy is an imaging technique in which the positions of individual nanoscale point emitters (e.g. fluorescent molecules) are determined at high precision from their images. This is the key ingredient in single/multiple-particle-tracking and several super-resolution microscopy approaches. Localization in three-dimensions (3D) can be performed by modifying the image that a point-source creates on the camera, namely, the point-spread function (PSF). The PSF is engineered using additional optical elements to vary distinctively with the depth of the point-source. However, localizing multiple adjacent emitters in 3D poses a significant algorithmic challenge, due to the lateral overlap of their PSFs. Here, we train a neural network to receive an image containing densely overlapping PSFs of multiple emitters over a large axial range and output a list of their 3D positions. Furthermore, we then use the network to design the optimal PSF for the multi-emitter case. We demonstrate our approach numerically as well as experimentally by 3D STORM imaging of mitochondria, and volumetric imaging of dozens of fluorescently-labeled telomeres occupying a mammalian nucleus in a single snapshot.

Motivation & Objective

• To address the challenge of accurately localizing multiple, densely overlapping fluorescent emitters in three dimensions using engineered point-spread functions (PSFs).
• To develop a deep learning model capable of processing complex, overlapping PSF patterns across a wide axial range to recover precise 3D emitter positions.
• To use the trained network to design an optimal PSF for multi-emitter 3D localization, improving resolution and accuracy in dense labeling scenarios.
• To demonstrate the method in both numerical simulations and experimental 3D STORM imaging of biologically relevant structures such as mitochondria and telomeres.

Proposed method

• A convolutional neural network (CNN) is trained to map input images containing overlapping 3D PSFs of multiple emitters to their corresponding 3D positions.
• The network is trained on synthetic data generated by simulating the imaging of densely packed emitters across a broad axial range with engineered PSFs.
• The PSF is modulated using optical elements to create depth-dependent variations that encode axial position information.
• The trained network is used in an inverse design loop to optimize the PSF shape for maximum localization accuracy in multi-emitter scenarios.
• The method leverages end-to-end learning to directly map raw image intensities to 3D coordinates without requiring explicit PSF fitting or iterative refinement.
• The approach is validated using both simulated data and experimental 3D STORM imaging of fluorescently labeled cellular structures.

Experimental results

Research questions

• RQ1 Can a deep neural network accurately localize multiple densely overlapping 3D emitters from their complex, overlapping PSFs?
• RQ2 How does the performance of the deep learning-based localization method compare to conventional PSF fitting in dense, multi-emitter environments?
• RQ3 Can the trained network be used to co-design an optimal PSF for dense 3D localization, improving resolution and accuracy?
• RQ4 To what extent can this method enable high-precision volumetric imaging of biologically complex structures like telomeres in a single snapshot?

Key findings

• The deep learning model achieves high-precision 3D localization of emitters even when their PSFs are densely overlapping across a broad axial range.
• The method enables successful 3D STORM imaging of mitochondria with sub-10 nm localization precision in all three dimensions.
• The network successfully localized dozens of fluorescently labeled telomeres within a single mammalian nucleus in a single snapshot, demonstrating volumetric imaging capability.
• The inverse design process guided by the network produced a PSF that significantly improved localization accuracy in dense emitter configurations.
• The approach outperformed conventional PSF fitting methods in terms of localization precision and robustness under high emitter density.
• The method is robust to noise and variations in emitter density, enabling reliable imaging in biologically relevant, crowded environments.