[Paper Review] Real-time 3D reconstruction of complex scenes using single-photon lidar: when image processing meets computer graphics

Julián Tachella, Yoann Altmann+6 more|arXiv (Cornell University)|May 16, 2019

Advanced Optical Sensing Technologies29 references2 citations

TL;DR

This paper presents a real-time 3D reconstruction framework for single-photon lidar that combines statistical modeling with scalable computer graphics algorithms, enabling sub-20 ms processing of outdoor scenes up to 320 m away, even in complex, cluttered environments with multiple surfaces per pixel, achieving video-rate reconstruction for practical applications.

ABSTRACT

Single-photon lidar has emerged as a prime candidate technology for depth imaging through challenging environments. Until now, a major limitation has been the significant amount of time required for the analysis of the recorded data. Here we show a new computational framework for real-time three-dimensional (3D) scene reconstruction from single-photon data. By combining statistical models with highly scalable computational tools from the computer graphics community, we demonstrate 3D reconstruction of complex outdoor scenes with processing times of the order of 20 ms, where the lidar data was acquired in broad daylight from distances up to 320 metres. The proposed method can handle an unknown number of surfaces in each pixel, allowing for target detection and imaging through cluttered scenes. This enables robust, real-time target reconstruction of complex moving scenes, paving the way for single-photon lidar at video rates for practical 3D imaging applications.

Motivation & Objective

• To overcome the long processing times that have limited real-time application of single-photon lidar in complex environments.
• To enable robust 3D reconstruction of moving, cluttered outdoor scenes with multiple surfaces per pixel.
• To achieve video-rate processing (sub-20 ms per frame) using scalable computational techniques from computer graphics.
• To support target detection and imaging through dense clutter by handling an unknown number of surfaces per pixel.
• To demonstrate practical, real-time 3D imaging in broad daylight using single-photon lidar.

Proposed method

• The framework integrates statistical models of single-photon return signals with highly scalable rendering algorithms from computer graphics.
• It employs a probabilistic approach to model multiple surface returns per pixel, enabling reconstruction in cluttered scenes.
• The method uses a voxel-based representation to reconstruct 3D geometry from photon timestamps and spatial coordinates.
• It applies advanced filtering and clustering techniques to separate multiple reflections and reduce noise in photon data.
• The computational pipeline is optimized for GPU acceleration, enabling real-time performance.
• The system processes data from single-photon lidar sensors with high dynamic range and low signal-to-noise ratios.

Experimental results

Research questions

• RQ1 Can single-photon lidar data be processed in real time for complex outdoor scenes?
• RQ2 How can multiple surfaces per pixel be accurately reconstructed in cluttered environments?
• RQ3 What computational techniques enable sub-20 ms processing of 3D lidar data?
• RQ4 Can robust target detection and imaging be achieved in broad daylight with single-photon lidar?
• RQ5 What role do scalable computer graphics algorithms play in accelerating 3D reconstruction from sparse photon data?

Key findings

• The framework achieves 3D scene reconstruction in under 20 ms per frame, enabling video-rate processing.
• The method successfully reconstructs complex outdoor scenes up to 320 metres away in broad daylight.
• It handles an unknown number of surfaces per pixel, allowing for accurate imaging through cluttered environments.
• The system enables robust reconstruction of moving scenes, demonstrating practical real-time performance.
• The integration of statistical modeling with computer graphics techniques significantly accelerates processing without sacrificing reconstruction quality.