[Paper Review] Multicolor localization microscopy by deep learning

Eran Hershko, Lucien E. Weiss+2 more|arXiv (Cornell University)|Jul 4, 2018

Advanced Fluorescence Microscopy Techniques39 references3 citations

TL;DR

This paper proposes a deep learning approach to enable multicolor localization microscopy using a standard grayscale camera, eliminating the need for spectral filters or complex optics. By leveraging the chromatic dependence of the point-spread function, the method accurately classifies single-emitter colors in both static and mobile conditions, with further improvements via learned phase-modulating elements for enhanced color resolution.

ABSTRACT

Deep learning has become an extremely effective tool for image classification and image restoration problems. Here, we apply deep learning to microscopy, and demonstrate how neural networks can exploit the chromatic dependence of the point-spread function to classify the colors of single emitters imaged on a grayscale camera. While existing single-molecule methods for spectral classification require additional optical elements in the emission path, e.g. spectral filters, prisms, or phase masks, our neural net correctly identifies static as well as mobile emitters with high efficiency using a standard, unmodified single-channel configuration. Furthermore, we demonstrate how deep learning can be used to design phase-modulating elements that, when implemented into the imaging path, result in further improved color differentiation between species.

Motivation & Objective

• To enable multicolor single-molecule localization microscopy without additional spectral optical components.
• To classify the colors of single emitters using only intensity data from a grayscale camera.
• To develop a deep learning framework that exploits chromatic variations in the point-spread function for color discrimination.
• To design phase-modulating elements via learning to further enhance color differentiation in the imaging path.

Proposed method

• A convolutional neural network is trained to classify the color of single emitters based on their intensity patterns on a grayscale camera.
• The method exploits the inherent chromatic dependence of the point-spread function across different emission wavelengths.
• The network is trained on simulated data of single emitters with varying colors and positions to generalize across static and mobile emitters.
• A differentiable design framework is used to optimize phase-modulating elements that enhance spectral separation in the imaging system.
• The trained neural network is deployed in a standard single-channel microscopy setup without hardware modifications.
• The phase masks are co-designed with the network to maximize color classification accuracy.

Experimental results

Research questions

• RQ1 Can deep learning accurately classify the color of single emitters using only grayscale intensity data?
• RQ2 Can the method distinguish between static and mobile emitters across multiple colors without spectral filters?
• RQ3 Can learned phase-modulating elements improve color resolution beyond standard imaging configurations?
• RQ4 How does the chromatic variation in the point-spread function enable color classification in a single-channel system?

Key findings

• The deep learning model achieves high-accuracy color classification of single emitters using only a grayscale camera, without additional spectral optics.
• The method successfully classifies both static and mobile emitters with high efficiency, demonstrating robustness to motion.
• The neural network leverages the chromatic dependence of the point-spread function to infer emitter color from intensity patterns alone.
• Learned phase-modulating elements significantly improve color differentiation, enhancing the system's spectral resolution.
• The approach enables multicolor localization microscopy in a standard, unmodified single-channel imaging setup.