[论文解读] Non-line-of-sight imaging off a Phong surface through deep learning

Chen Zhou, Chengyu Wang+1 more|arXiv (Cornell University)|Apr 30, 2020

Advanced Optical Sensing Technologies参考文献 1被引用 7

一句话总结

本文提出了一种基于深度学习的非视域（NLOS）成像系统，利用Phong表面的散射光重建隐藏物体，通过在手写数字数据集上训练的神经网络实现对未见模式和视频的泛化。该方法在仅具有5%镜面反射的表面上实现了高达0.93的结构相似性指数（SSIM），由于奇异值谱带宽增强，其性能优于纯朗伯表面。

ABSTRACT

A deep learning based non-line-of-sight (NLOS) imaging system is developed to image an occluded object off a scattering surface. The neural net is trained using only handwritten digits, and yet exhibits capability to reconstruct patterns distinct from the training set, including physical objects. It can also reconstruct a cartoon video from its scattering patterns in real time, demonstrating the robustness and generalization capability of the deep learning based approach. Several scattering surfaces with varying degree of Lambertian and specular contributions were examined experimentally; it is found that for a Lambertian surface the structural similarity index (SSIM) of reconstructed images is about 0.63, while the SSIM obtained from a scattering surface possessing a specular component can be as high as 0.93. A forward model of light transport was developed based on the Phong scattering model. Scattering patterns from Phong surfaces with different degrees of specular contribution were numerically simulated. It is found that a specular contribution of as small as 5% can enhance the SSIM from 0.83 to 0.93, consistent with the results from experimental data. Singular value spectra of the underlying transfer matrix were calculated for various Phong surfaces. As the weight and the shininess factor increase, i.e., the specular contribution increases, the singular value spectrum broadens and the 50-dB bandwidth is increased by more than 4X with a 10% specular contribution, which indicates that at the presence of even a small amount of specular contribution the NLOS measurement can retain significantly more singular value components, leading to higher reconstruction fidelity. With an ordinary camera and incoherent light source, this work enables a low-cost, real-time NLOS imaging system without the need of an explicit physical model of the underlying light transport process.

研究动机与目标

• 开发一种无需显式光传输物理建模的低成本、实时非视域成像系统。
• 研究Phong散射表面中的镜面成分对非视域重建保真度的影响。
• 评估在简单数据集（如手写数字）上训练的深度学习模型对复杂、未见模式和动态场景的泛化能力。
• 量化镜面反射对光传输矩阵奇异值谱的影响及其对重建质量的含义。

提出的方法

• 仅使用基于Phong光传输模型生成的手写数字的合成散射图案，对深度神经网络进行训练。
• 采用Phong散射模型模拟具有不同朗伯和镜面反射成分的表面的光传输。
• 通过数值模拟生成不同镜面贡献度（0%至100%）的散射图案，用于网络的训练与评估。
• 在具有受控镜面特性的物理表面采集的真实散射图案上测试网络，实现对静态图案和动态卡通视频的实时重建。
• 对底层光传输矩阵应用奇异值分解（SVD），分析镜面贡献增加对奇异值分布和带宽的影响。
• 系统使用普通相机和非相干光源运行，无需专用硬件或显式物理建模。

实验结果

研究问题

• RQ1 在手写数字上训练的深度学习模型能否泛化至重建非视域成像中复杂、未见的物理物体？
• RQ2 Phong表面中的镜面反射存在如何影响非视域成像的重建质量？
• RQ3 随着镜面贡献的增加，光传输矩阵的奇异值谱变化程度如何？这种变化如何影响重建保真度？
• RQ4 基于深度学习的方法能否在不依赖显式光传输物理建模的情况下实现实时非视域成像？

主要发现

• 深度学习模型在纯朗伯表面的结构相似性指数（SSIM）为0.63，当引入镜面成分后显著提升至0.93。
• 即使仅5%的镜面贡献，SSIM也从0.83提升至0.93，表明极少量镜面反射即可极大提升重建质量。
• 随着镜面贡献增加，奇异值谱带宽拓宽，50-dB带宽在仅10%镜面反射时增加超过400%。
• 模型成功实现了从散射图案中实时重建卡通视频，证实其在训练数据之外的鲁棒性与泛化能力。
• 该方法实现了使用标准相机和非相干光源的实时、低成本非视域成像，无需显式光传输物理建模。
• 结果表明，散射表面中的镜面成分可被利用以在传输矩阵中保留更多奇异值分量，从而实现更高保真度的重建。