[Paper Review] Local Conditional Neural Fields for Versatile and Generalizable Large-Scale Reconstructions in Computational Imaging

Hao Wang, Zhu, Jiabei+3 more|arXiv (Cornell University)|Jul 12, 2023

Advanced X-ray Imaging Techniques4 citations

TL;DR

This paper proposes Local Conditional Neural Fields (LCNF), a continuous implicit neural representation framework that enables scalable, generalizable, and high-fidelity large-scale phase reconstruction in computational imaging. By combining a measurement-specific latent code with a local conditional decoder, LCNF achieves robust super-resolution reconstruction from few multiplexed Fourier ptychographic measurements, even when trained on limited or simulated data, with inference speeds of ~1×10⁻⁵ seconds per pixel on a single GPU.

ABSTRACT

Deep learning has transformed computational imaging, but traditional pixel-based representations limit their ability to capture continuous, multiscale details of objects. Here we introduce a novel Local Conditional Neural Fields (LCNF) framework, leveraging a continuous implicit neural representation to address this limitation. LCNF enables flexible object representation and facilitates the reconstruction of multiscale information. We demonstrate the capabilities of LCNF in solving the highly ill-posed inverse problem in Fourier ptychographic microscopy (FPM) with multiplexed measurements, achieving robust, scalable, and generalizable large-scale phase retrieval. Unlike traditional neural fields frameworks, LCNF incorporates a local conditional representation that promotes model generalization, learning multiscale information, and efficient processing of large-scale imaging data. By combining an encoder and a decoder conditioned on a learned latent vector, LCNF achieves versatile continuous-domain super-resolution image reconstruction. We demonstrate accurate reconstruction of wide field-of-view, high-resolution phase images using only a few multiplexed measurements. LCNF robustly captures the continuous object priors and eliminates various phase artifacts, even when it is trained on imperfect datasets. The framework exhibits strong generalization, reconstructing diverse objects even with limited training data. Furthermore, LCNF can be trained on a physics simulator using natural images and successfully applied to experimental measurements on biological samples. Our results highlight the potential of LCNF for solving large-scale inverse problems in computational imaging, with broad applicability in various deep-learning-based techniques.

Motivation & Objective

• To address the limitations of pixel-based deep learning in computational imaging, particularly the inability to represent continuous, multiscale object details and poor scalability to large fields of view.
• To overcome the generalization and computational inefficiency of existing neural field methods that require retraining per object or rely on global latent codes.
• To enable high-fidelity, wide-field-of-view phase reconstruction from minimal multiplexed measurements using a single trained model.
• To evaluate the framework’s robustness when trained on imperfect, limited, or simulated datasets and applied to real biological samples.

Proposed method

• LCNF employs a CNN encoder to extract measurement-specific features and map them into a latent space, enabling local conditional representation.
• A learnable MLP decoder reconstructs continuous phase values at arbitrary spatial coordinates using the latent vector as a conditional input.
• The framework uses a coordinate-based representation where spatial coordinates are mapped to physical phase values via a multi-layer perceptron, enabling resolution-invariant synthesis.
• Local conditioning allows the model to capture multiscale features and generalize across diverse objects without retraining.
• Inference is performed via dense coordinate querying across a grid, with patch-wise reconstruction used to manage GPU memory constraints during wide-field imaging.
• The model is trained end-to-end using a pixel-wise loss between predicted and ground-truth phase maps, with data augmentation and normalization applied during training.

Figure 1: Conceptual illustration of our LCNF framework for FPM reconstruction. (a) LCNF employs a CNN encoder to learn measurement-specific information and encode them into a latent-space representation. The MLP decoder reconstructs the phase values at specific locations with an increased spatial r

Experimental results

Research questions

• RQ1 Can a continuous neural field representation generalize across diverse biological samples when trained on limited or imperfect experimental data?
• RQ2 How does local conditioning improve generalization and multiscale feature learning compared to global latent codes in neural fields?
• RQ3 To what extent can a model trained on simulated natural images generalize to real experimental measurements in Fourier ptychographic microscopy?
• RQ4 What is the inference efficiency and scalability of the framework for large-scale, wide-field-of-view phase reconstruction?

Key findings

• LCNF achieved high-fidelity reconstruction of 1500×1500 pixel phase images from only a few multiplexed measurements, with inference taking approximately 25 seconds on an NVIDIA Quadro RTX8000 GPU.
• The framework successfully reconstructed a 12960-pixel diameter wide-field-of-view phase image by stitching 250×250 pixel patches, demonstrating scalability beyond single-GPU memory limits.
• Networks trained on a single experimental image pair (e.g., Hela-E(1)) achieved meaningful reconstruction quality, indicating strong generalization from minimal data.
• The model trained on simulated natural images generalized effectively to real biological samples, achieving competitive reconstruction quality without any real experimental data in training.
• LCNF reduced phase artifacts and captured continuous object priors even when trained on imperfect datasets, demonstrating robustness to distribution shift.
• The average inference speed was ~1×10⁻⁵ seconds per pixel, enabling efficient reconstruction of high-resolution images.

Figure 2: Reconstruction results using the LCNF trained with our experimental dataset. An example of BF low-resolution intensity image, the model-based FPM reconstruction, and our LCNF network-based reconstruction for (a) Hela cells fixed with ethanol, and (b) Hela cells fixed with formalin. Subarea