[Paper Review] RGB-D Local Implicit Function for Depth Completion of Transparent Objects

Luyang Zhu, Arsalan Mousavian+5 more|arXiv (Cornell University)|Apr 1, 2021

Advanced Vision and Imaging54 references4 citations

TL;DR

This paper proposes a fast, generalizable depth completion method for transparent objects using a Local Implicit Depth Function (LIDF) based on ray-voxel pairs, enabling real-time inference and superior accuracy. The approach combines a learnable local implicit representation with iterative refinement, outperforming SOTA methods in both synthetic and real-world benchmarks while achieving 20× faster inference than ClearGrasp.

ABSTRACT

Majority of the perception methods in robotics require depth information provided by RGB-D cameras. However, standard 3D sensors fail to capture depth of transparent objects due to refraction and absorption of light. In this paper, we introduce a new approach for depth completion of transparent objects from a single RGB-D image. Key to our approach is a local implicit neural representation built on ray-voxel pairs that allows our method to generalize to unseen objects and achieve fast inference speed. Based on this representation, we present a novel framework that can complete missing depth given noisy RGB-D input. We further improve the depth estimation iteratively using a self-correcting refinement model. To train the whole pipeline, we build a large scale synthetic dataset with transparent objects. Experiments demonstrate that our method performs significantly better than the current state-of-the-art methods on both synthetic and real world data. In addition, our approach improves the inference speed by a factor of 20 compared to the previous best method, ClearGrasp. Code and dataset will be released at https://research.nvidia.com/publication/2021-03_RGB-D-Local-Implicit.

Motivation & Objective

• Address the challenge of depth completion for transparent objects, which are invisible to standard RGB-D sensors due to light refraction and absorption.
• Overcome limitations of existing methods that require contact edges, physical priors, or suffer from slow inference speeds.
• Enable generalization to unseen transparent objects and real-world deployment through a learnable, local implicit representation.
• Develop a scalable synthetic dataset to close the sim-to-real gap for transparent object perception.
• Achieve high-accuracy depth completion with fast inference, suitable for real-time robotic applications.

Proposed method

• Introduce a Local Implicit Depth Function (LIDF) that models depth as a function of camera rays and their intersecting voxels, enabling localized, efficient inference.
• Use positional encoding to enhance feature representation and capture fine geometric details in the implicit function.
• Apply argmax pooling over ray-voxel pairs to select the most probable terminating voxel, reducing noise from distant voxels.
• Learn the offset within each voxel to predict the precise ray termination point, replacing heuristic 3D point sampling used in NeRF.
• Integrate a self-correcting refinement model that iteratively improves depth predictions using RGB, input depth, and LIDF outputs.
• Train the entire pipeline on a large-scale synthetic dataset, Omnivers Object, generated via NVIDIA Omniverse with diverse object types, lighting, and viewpoints.

Figure 1: Our method can predict the depth of unseen transparent objects from a noisy RGB-D image. We back-project the depth map into the point cloud and render it in a novel viewpoint to better visualize the 3D shape. Zoom in to see details.

Experimental results

Research questions

• RQ1 Can a local implicit neural representation based on ray-voxel pairs generalize to unseen transparent objects in depth completion?
• RQ2 Does learning depth via voxel offset prediction outperform heuristic 3D point sampling in terms of accuracy and efficiency?
• RQ3 Can a self-correcting refinement module improve depth completion accuracy without requiring ground-truth geometry during inference?
• RQ4 To what extent does a large-scale synthetic dataset improve zero-shot generalization to real-world transparent objects?
• RQ5 Can the proposed method achieve real-time inference while maintaining state-of-the-art accuracy on transparent object depth completion?

Key findings

• The proposed method achieves a test RMSE of 0.028 on real-world novel objects when trained on both ClearGrasp and Omnivers Object datasets, outperforming training on either dataset alone.
• The method improves inference speed by a factor of 20 compared to ClearGrasp, enabling real-time deployment.
• Using argmax ray pooling over voxel features yields 65.17% accuracy in δ₁.₀₅, significantly outperforming the weighted sum operator (59.31%).
• Positional encoding improves fine-detail capture, increasing δ₁.₀₅ by 12.3 percentage points compared to no encoding.
• Learning the offset within voxels leads to 0.028 RMSE, outperforming heuristic point sampling (0.038 RMSE) in depth prediction.
• Training on the Omnivers Object dataset alone improves generalization to real-world novel transparent objects, with δ₁.₂₅ reaching 99.43%.

Figure 2: Pipeline overview. Our method takes as input RGB-D images with missing depth for transparent objects, and predicts the full depth map. We first generate ray-voxel pairs by finding all occupied voxels intersected by a ray. Then we learn a local implicit function for each pair to estimate th