[Paper Review] Video super-resolution for single-photon LIDAR

Germán Mora Martín, Stirling Scholes+4 more|arXiv (Cornell University)|Oct 19, 2022

Advanced Optical Sensing Technologies4 citations

TL;DR

This paper proposes a 3D convolutional neural network (CNN) for video super-resolution and denoising of single-photon LIDAR depth data, trained on synthetic SPAD-based depth sequences to upscale low-resolution, noisy frames by ×4. The method achieves real-time performance (>30 FPS on GPU) and significantly improves PSNR and SSIM over state-of-the-art methods, even when applied to real experimental data without fine-tuning.

ABSTRACT

3D Time-of-Flight (ToF) image sensors are used widely in applications such as self-driving cars, Augmented Reality (AR) and robotics. When implemented with Single-Photon Avalanche Diodes (SPADs), compact, array format sensors can be made that offer accurate depth maps over long distances, without the need for mechanical scanning. However, array sizes tend to be small, leading to low lateral resolution, which combined with low Signal-to-Noise Ratio (SNR) levels under high ambient illumination, may lead to difficulties in scene interpretation. In this paper, we use synthetic depth sequences to train a 3D Convolutional Neural Network (CNN) for denoising and upscaling (x4) depth data. Experimental results, based on synthetic as well as real ToF data, are used to demonstrate the effectiveness of the scheme. With GPU acceleration, frames are processed at >30 frames per second, making the approach suitable for low-latency imaging, as required for obstacle avoidance.

Motivation & Objective

• To address the low lateral resolution and high noise levels in single-photon LIDAR depth maps from compact SPAD array sensors.
• To develop a video super-resolution framework that leverages temporal correlations across consecutive frames to improve depth map quality.
• To train a deep learning model on synthetic SPAD data that mimics real-world noise and resolution constraints.
• To enable real-time, low-latency depth map enhancement suitable for obstacle avoidance in autonomous systems.
• To evaluate the model’s robustness on both synthetic and real experimental SPAD dToF data.

Proposed method

• Synthetic depth sequences were generated using the AirSim simulator integrated with Unreal Engine to model realistic scenes with varying signal-to-noise ratios (SNR).
• The synthetic data was processed to emulate SPAD-based depth measurements, including photon noise and low SNR conditions, to create a diverse training dataset.
• A 3D CNN architecture was trained to reconstruct high-resolution depth maps from multiple low-resolution input frames, using spatio-temporal feature learning.
• The network employs 3D convolutions to exploit both spatial and temporal correlations across consecutive frames for improved feature representation.
• Training was performed exclusively on synthetic data, with no real data used during training to enable generalization to real-world conditions.
• Inference was accelerated using GPU, achieving frame rates exceeding 30 FPS, suitable for real-time applications.

Experimental results

Research questions

• RQ1 Can a deep learning-based video super-resolution method effectively enhance the resolution and reduce noise in single-photon LIDAR depth maps?
• RQ2 How does the performance of the proposed method compare to state-of-the-art approaches like iSeeBetter under low SNR conditions?
• RQ3 To what extent can a model trained solely on synthetic data generalize to real experimental SPAD dToF sensor data?
• RQ4 What frame rate is required to maintain sub-pixel motion between objects for effective temporal feature fusion in the network?
• RQ5 Does using multiple input frames significantly improve reconstruction quality compared to single-frame super-resolution?

Key findings

• The proposed method achieves higher PSNR and SSIM values than the iSeeBetter baseline across all tested SNR levels, including the lowest SNR of 0.25.
• The network demonstrates robustness to noise, effectively reconstructing fine details and smooth surfaces even in highly noisy low-SNR conditions.
• Despite being trained only on synthetic data, the model produces visually superior and more temporally coherent super-resolved outputs on real SPAD dToF sensor data.
• The method maintains real-time performance, processing frames at over 30 FPS on GPU, making it suitable for low-latency applications like obstacle avoidance.
• The use of multiple input frames significantly improves feature reconstruction and denoising compared to single-frame super-resolution, especially under low frame rates.
• Ablation studies confirm that the network’s performance is stable even when object displacements exceed sub-pixel levels, indicating strong generalization to dynamic scenes.