[Paper Review] Hybrid Diffractive Optics Design via Hardware-in-the-Loop Methodology for Achromatic Extended-Depth-of-Field Imaging

Samuel Pinilla, Seyyed Reza Miri Rostami+3 more|arXiv (Cornell University)|Mar 30, 2022

Advanced optical system design51 references37 citations

TL;DR

This paper proposes a hardware-in-the-loop (HIL) optimization framework for designing hybrid diffractive optics using a spatial light modulator (SLM) as a programmable phase-only diffractive optical element (DOE), enabling achromatic extended-depth-of-field (EDoF) imaging. By co-optimizing the SLM phase pattern and image reconstruction algorithm via a differentiable end-to-end pipeline with quantitative and qualitative losses, the method achieves high-quality all-in-focus imaging across 0.4–1.9 m depth range, outperforming both lens-only systems and commercial cameras like Sony A7 III and iPhone Xs Max in sharpness and color fidelity.

ABSTRACT

End-to-end optimization of diffractive optical elements (DOEs) profile through a digital differentiable model combined with computational imaging have gained an increasing attention in emerging applications due to the compactness of resultant physical setups. Despite recent works have shown the potential of this methodology to design optics, its performance in physical setups is still limited and affected by manufacturing artifacts of DOE, mismatch between simulated and resultant experimental point spread functions, and calibration errors. Additionally, the computational burden of the digital differentiable model to effectively design the DOE is increasing, thus limiting the size of the DOE that can be designed. To overcome the above mentioned limitations, the broadband imaging system with phase-only spatial light modulator (SLM) as an encoded DOE is proposed and developed in this paper. A co-design of the SLM phase pattern and image reconstruction algorithm is produced following the end-to-end strategy, using for optimization a convolutional neural network equipped with quantitative and qualitative loss functions. The optics of the imaging system is hybrid consisting of SLM as DOE and refractive lens. SLM phase-pattern is optimized by applying the Hardware-in-the-loop technique, which helps to eliminate the mismatch between numerical modeling and physical reality of image formation as light propagation is not numerically modeled but is physically done. In our experiments, the hybrid optics is implemented by the optical projection of the SLM phase-pattern on a lens plane for a depth range 0.4-1.9m. Comparison with compound multi-lens optics such as Sony A7 III and iPhone Xs Max cameras show that the proposed system is advanced in all-in-focus sharp imaging.

Motivation & Objective

• To address the performance gap between theoretical DOE designs and physical implementations due to manufacturing artifacts, PSF mismatch, and calibration errors.
• To overcome the high computational cost and scalability limits of purely digital differentiable modeling for large DOE designs.
• To develop a practical, end-to-end co-design framework for diffractive optics that bridges simulation and real-world optical performance.
• To achieve high-quality, achromatic, extended-depth-of-field imaging using a compact hybrid system of refractive lens and SLM-based DOE.

Proposed method

• Employ a hardware-in-the-loop (HIL) setup where physical light propagation replaces numerical simulation in the optimization loop, eliminating model-experiment mismatch.
• Use a spatial light modulator (SLM) as a pixel-wise programmable phase-only DOE, enabling real-time, reconfigurable wavefront modulation.
• Co-optimize the SLM phase pattern and image reconstruction algorithm using a differentiable convolutional neural network (CNN) with combined quantitative (e.g., PSNR) and qualitative (e.g., perceptual) loss functions.
• Implement the hybrid optical system by projecting the SLM phase pattern onto the lens plane, forming a compact refractive-DOE system.
• Conduct physical experiments using two setups: one with multiple objects at different depths (Setup 1) and one with a single 3D scene (Setup 2) to validate EDoF performance.
• Use real optical data from physical imaging (not simulated PSFs) to train and validate the end-to-end system, ensuring robustness to real-world distortions.

Experimental results

Research questions

• RQ1 Can hardware-in-the-loop optimization effectively close the gap between simulated and physical performance in DOE design?
• RQ2 How does co-optimizing the SLM phase pattern and image reconstruction algorithm improve achromatic EDoF imaging quality compared to standalone lens or fixed-phase DOE systems?
• RQ3 To what extent does the proposed HIL methodology outperform commercial compound-lens cameras (e.g., Sony A7 III, iPhone Xs Max) in all-in-focus sharp imaging across a wide depth range?
• RQ4 Can the system maintain high image quality and color fidelity under dynamic depth variations, such as moving objects?
• RQ5 What is the impact of using a differentiable CNN with mixed loss functions on the robustness and generalization of the optimized DOE?

Key findings

• The HIL-optimized hybrid system achieved a PSNR of approximately 25 dB across all RGB channels and depths (0.4–1.9 m), confirming stable, achromatic performance.
• For defocus distances of 1.8 m, the hybrid system improved PSNR by about 2 dB over the lens-only system, demonstrating significant enhancement in image quality.
• Visual comparisons showed that the designed hybrid system outperformed both lens-only and cubic-phase SLM systems in sharpness and color preservation, especially at extreme depths.
• The system maintained high image quality during dynamic depth variations, as demonstrated by a video of a sunflower moving from 0.3 m to 1.9 m, with consistently sharp and well-colored reconstructions.
• The hybrid system surpassed commercial cameras like the Sony A7 III and iPhone Xs Max in all-in-focus imaging performance, particularly in out-of-focus regions, despite the latter having complex multi-lens optics.
• The HIL methodology successfully mitigated the mismatch between simulation and physical reality, enabling reliable design of high-performance DOEs without relying on numerically modeled PSFs.