[Paper Review] DELTAR: Depth Estimation from a Light-weight ToF Sensor and RGB Image

Yijin Li, Xinyang Liu+6 more|arXiv (Cornell University)|Sep 27, 2022

Advanced Optical Sensing Technologies4 citations

TL;DR

DELTAR proposes a novel neural network that fuses a low-resolution depth distribution from a lightweight ToF sensor (8×8 zones) with an RGB image to produce high-fidelity, dense depth maps. By using a distribution-aware feature extractor and cross-attention fusion, it achieves depth quality on par with commodity RGB-D sensors like Intel RealSense D435i, significantly outperforming existing depth completion and super-resolution methods on low-resolution, distribution-based inputs.

ABSTRACT

Light-weight time-of-flight (ToF) depth sensors are small, cheap, low-energy and have been massively deployed on mobile devices for the purposes like autofocus, obstacle detection, etc. However, due to their specific measurements (depth distribution in a region instead of the depth value at a certain pixel) and extremely low resolution, they are insufficient for applications requiring high-fidelity depth such as 3D reconstruction. In this paper, we propose DELTAR, a novel method to empower light-weight ToF sensors with the capability of measuring high resolution and accurate depth by cooperating with a color image. As the core of DELTAR, a feature extractor customized for depth distribution and an attention-based neural architecture is proposed to fuse the information from the color and ToF domain efficiently. To evaluate our system in real-world scenarios, we design a data collection device and propose a new approach to calibrate the RGB camera and ToF sensor. Experiments show that our method produces more accurate depth than existing frameworks designed for depth completion and depth super-resolution and achieves on par performance with a commodity-level RGB-D sensor. Code and data are available at https://zju3dv.github.io/deltar/.

Motivation & Objective

• To address the challenge of generating high-resolution, accurate depth maps from extremely low-resolution (8×8) depth distributions provided by lightweight ToF sensors.
• To overcome the limitations of existing depth completion and super-resolution methods, which assume higher-resolution or point-based depth inputs.
• To design a joint learning framework that effectively fuses sparse, probabilistic depth distributions with high-resolution RGB images.
• To develop a new calibration method for aligning RGB and ToF sensors when direct correspondence between modalities is not trivial.
• To create a new real-world dataset (ZJU-L5) with paired, pixel-aligned RGB and ToF signals for training and evaluation.

Proposed method

• A distribution feature extractor samples depth hypotheses from the ToF sensor’s depth distribution per 8×8 zone, preserving probabilistic information.
• A cross-attention mechanism fuses features from the RGB image and the sampled depth hypotheses, with explicit modeling of patch-to-distribution correspondence.
• An image self-attention module propagates context across the full field of view, enabling dense output aligned with the RGB image.
• A refinement module predicts bin-widths and linear coefficients for depth reconstruction, improving accuracy through differentiable regression.
• A novel calibration method uses an EM-like algorithm to estimate plane geometry from ToF signals, followed by point-to-plane optimization to estimate extrinsic parameters between the RGB and ToF sensors.
• Synthetic ToF signals are generated from NYU-Depth V2 to augment real-world data, improving generalization during training.

Experimental results

Research questions

• RQ1 Can a neural network effectively learn to reconstruct dense, high-accuracy depth maps from sparse, distribution-based depth inputs of a lightweight ToF sensor?
• RQ2 How can cross-modal attention be designed to preserve the physical meaning of depth distributions while fusing them with RGB image features?
• RQ3 What is the impact of modeling patch-to-distribution correspondence in cross-attention for depth estimation?
• RQ4 Can the proposed method achieve depth quality comparable to commodity-level RGB-D sensors like Intel RealSense D435i?
• RQ5 How does the inclusion of probabilistic sampling and attention mechanisms affect performance compared to direct mean/variance input or simple concatenation?

Key findings

• DELTAR outperforms existing depth completion and super-resolution frameworks on the same task, particularly due to its distribution-aware feature extraction and attention-based fusion.
• The method improves the raw depth from the ST VL53L5CX (L5) sensor to a quality on par with the Intel RealSense D435i, especially within the L5 sensor’s operational range (up to 3 meters).
• The ablation study shows that removing image self-attention or image-to-distribution attention leads to significant performance drops, confirming their importance in feature propagation and fusion.
• Using probability-driven sampling instead of uniform sampling improves RMSE by 0.3 cm, demonstrating the benefit of modeling distribution uncertainty.
• Replacing the refinement module with a simple two-layer decoder causes only a minor performance drop, indicating that the main gains come from the feature extractor and attention fusion components.
• Quantitative evaluation on a flat wall shows that DELTAR achieves similar or better bias and jitter than RealSense D435i at distances below 3000 mm, confirming its accuracy and robustness.