[Paper Review] Real-time sparse-sampled Ptychographic imaging through deep neural networks

Mathew J. Cherukara, Tao Zhou+5 more|arXiv (Cornell University)|Apr 15, 2020

Advanced X-ray Imaging Techniques4 citations

TL;DR

This paper introduces PtychoNN, a deep convolutional neural network that enables real-time ptychographic imaging by directly mapping far-field diffraction patterns to real-space object structure and phase. Trained on experimental X-ray ptychography data, PtychoNN achieves reconstructions up to 300 times faster than state-of-the-art iterative methods and successfully inverts sub-sampled data with less than 50% overlap, enabling faster, lower-dose, and dynamic imaging of sensitive or large-volume samples.

ABSTRACT

Ptychography has rapidly grown in the fields of X-ray and electron imaging for its unprecedented ability to achieve nano or atomic scale resolution while simultaneously retrieving chemical or magnetic information from a sample. A ptychographic reconstruction is achieved by means of solving a complex inverse problem that imposes constraints both on the acquisition and on the analysis of the data, which typically precludes real-time imaging due to computational cost involved in solving this inverse problem. In this work we propose PtychoNN, a novel approach to solve the ptychography reconstruction problem based on deep convolutional neural networks. We demonstrate how the proposed method can be used to predict real-space structure and phase at each scan point solely from the corresponding far-field diffraction data. The presented results demonstrate how PtychoNN can effectively be used on experimental data, being able to generate high quality reconstructions of a sample up to hundreds of times faster than state-of-the-art ptychography reconstruction solutions once trained. By surpassing the typical constraints of iterative model-based methods, we can significantly relax the data acquisition sampling conditions and produce equally satisfactory reconstructions. Besides drastically accelerating acquisition and analysis, this capability can enable new imaging scenarios that were not possible before, in cases of dose sensitive, dynamic and extremely voluminous samples.

Motivation & Objective

• To overcome the computational bottleneck of iterative ptychographic phase retrieval, which limits real-time imaging in X-ray and electron microscopy.
• To enable high-fidelity ptychographic reconstructions from sparsely sampled data, relaxing the 50% overlap constraint typical of iterative methods.
• To develop a deep learning solution that is fast, deployable on edge devices, and requires minimal training data for practical use in experimental beamlines.
• To demonstrate the feasibility of end-to-end deep learning for real-world ptychographic imaging using experimental data from the Advanced Photon Source.

Proposed method

• PtychoNN employs a deep convolutional neural network architecture with an encoder-decoder structure to learn a direct mapping from raw far-field diffraction patterns to real-space amplitude and phase.
• The network is trained using 16,000 experimental ptychographic diffraction patterns acquired at the Advanced Photon Source's X-ray nanoprobe beamline (26-ID).
• The training data consists of 161×161 scan points with 30 nm step size and 50% overlap, using a 60 nm coherent beam focused by a Fresnel zone plate.
• The network is optimized to predict both object amplitude and phase simultaneously from a single forward pass, enabling inference in ~1 ms per scan point.
• The method bypasses iterative phase retrieval by learning the inverse mapping directly from data, eliminating the need for overlap constraints.
• The model is evaluated on sub-sampled data (down to 1/5 of original sampling) to test robustness to sparse acquisition.

Experimental results

Research questions

• RQ1 Can a deep neural network achieve real-time ptychographic reconstruction from experimental X-ray diffraction data?
• RQ2 To what extent can PtychoNN reconstruct high-fidelity images from sub-sampled ptychographic data with less than 50% overlap?
• RQ3 How much training data is required for PtychoNN to achieve reliable reconstructions in practical settings?
• RQ4 Can PtychoNN be trained and deployed efficiently on limited hardware, enabling on-the-fly imaging at beamlines?

Key findings

• PtychoNN achieves reconstruction speeds up to 300 times faster than Ptycholib, a high-performance iterative ptychography software, with inference taking only ~1 ms per scan point.
• The network produces high-quality reconstructions of amplitude and phase even when trained on as few as 800 samples, with training completed in under a minute on a single NVIDIA V100 GPU.
• PtychoNN successfully reconstructs images from ptychographic data with only 1/5 of the original sampling density, maintaining accuracy despite significant sub-sampling.
• Unlike iterative methods, which fail or produce artifacts when overlap is below 50%, PtychoNN maintains fidelity under sparse sampling, demonstrating robustness to reduced data acquisition.
• The model generalizes well across different sample features, as demonstrated on a tungsten test pattern with random features, showing consistent performance across diverse structures.
• The method enables real-time feedback and opens new imaging scenarios for dose-sensitive, dynamic, or large-volume samples by drastically reducing radiation dose and acquisition time.