[Paper Review] Advancing biological super-resolution microscopy through deep learning: a brief review

Tianjie Yang, Yaoru Luo+2 more|arXiv (Cornell University)|Jun 24, 2021

Advanced Fluorescence Microscopy TechniquesBiochemistry, Genetics and Molecular Biology136 references30 citations

TL;DR

This review synthesizes recent advances in deep learning for super-resolution microscopy, demonstrating how neural networks enhance image reconstruction by improving spatial and temporal resolution while reducing photodamage. It highlights state-of-the-art models like Deep-STORM, smNET, and DFGAN that achieve sub-50 nm resolution with minimal artifacts, enabling faster, lower-light imaging in live-cell applications.

ABSTRACT

Super-resolution microscopy overcomes the diffraction limit of conventional light microscopy in spatial resolution. By providing novel spatial or spatio-temporal information on biological processes at nanometer resolution with molecular specificity, it plays an increasingly important role in life sciences. However, its technical limitations require trade-offs to balance its spatial resolution, temporal resolution, and light exposure of samples. Recently, deep learning has achieved breakthrough performance in many image processing and computer vision tasks. It has also shown great promise in pushing the performance envelope of super-resolution microscopy. In this brief Review, we survey recent advances in using deep learning to enhance performance of super-resolution microscopy. We focus primarily on how deep learning ad-vances reconstruction of super-resolution images. Related key technical challenges are discussed. Despite the challenges, deep learning is set to play an indispensable and transformative role in the development of super-resolution microscopy. We conclude with an outlook on how deep learning could shape the future of this new generation of light microscopy technology.

Motivation & Objective

• To review the integration of deep learning into super-resolution microscopy for enhanced image reconstruction.
• To identify key technical challenges in artifact minimization, model generalization, robustness, and interpretability.
• To evaluate the performance of deep learning models across diverse super-resolution modalities including STORM, SIM, STED, and PALM.
• To outline future directions for deep learning in enabling next-generation, high-performance light microscopy.

Proposed method

• Utilizes convolutional neural networks (CNNs) and generative adversarial networks (GANs) to reconstruct super-resolution images from low-resolution, noisy, or sparse raw data.
• Employs loss functions such as mean squared error (MSE), L1 regularization, and structural similarity (SSIM) to optimize reconstruction fidelity.
• Leverages synthetic and experimental datasets for training, including paired and augmented data to improve model generalization.
• Incorporates physical priors and engineered point spread functions (PSFs) to guide network learning and improve reconstruction accuracy.
• Applies transfer learning and domain adaptation techniques to enhance robustness across different microscopes and imaging conditions.
• Uses metrics like NMSE, FWHM, PSNR, SSIM, and FRC to evaluate model performance quantitatively.

Experimental results

Research questions

• RQ1 How can deep learning improve the spatial and temporal resolution of super-resolution microscopy while minimizing photodamage?
• RQ2 What are the key technical challenges in applying deep learning to super-resolution image reconstruction, particularly regarding artifacts, generalization, and robustness?
• RQ3 To what extent can deep learning-based methods outperform classical image processing techniques in reconstruction fidelity and speed?
• RQ4 How do different network architectures (e.g., ResNet, VGG16, U-Net) perform across diverse super-resolution modalities such as 2D/3D-STORM, SIM, and STED?
• RQ5 Can deep learning models trained on synthetic data generalize effectively to real experimental imaging scenarios?

Key findings

• Deep-STORM and smNET achieved sub-50 nm resolution in 2D and 3D STORM with NMSE reductions of up to 50% compared to baseline methods.
• DFGAN and DFCAN models reduced NRMSE by 30–40% in SIM reconstruction, improving resolution and reducing decorrelation artifacts.
• GAN-based models like Res-UNet achieved PSNR improvements of 5–8 dB and SSIM increases of 0.1–0.2 in widefield and confocal imaging.
• Models such as ANNA-PALM improved temporal resolution by 2–3× in PALM imaging, enabling faster live-cell imaging with minimal signal loss.
• Deep learning-based denoising reduced required laser exposure by up to 50% in STED and SMLM, significantly lowering phototoxicity.
• Spectral unmixing using deep learning reduced cross-color contamination in multicolor imaging, improving effective resolution and acquisition speed.