[Paper Review] Deep Robotic Prediction with hierarchical RGB-D Fusion

Yaoxian Song, Jun Wen+2 more|arXiv (Cornell University)|Sep 14, 2019

Robot Manipulation and Learning34 references4 citations

TL;DR

This paper proposes a hierarchical RGB-D fusion network, UG-Net V3, for real-time robotic grasping in unstructured 3D environments using only partial observations from an eye-in-hand camera. By quantifying depth uncertainty and fusing RGB and depth features at multiple scales, the method achieves over 90% grasping success rate without relying on ICP or 3D reconstruction, outperforming state-of-the-art methods in both table-top and 3D scenarios.

ABSTRACT

Robotic arm grasping is a fundamental operation in robotic control task goals. Most current methods for robotic grasping focus on RGB-D policy in the table surface scenario or 3D point cloud analysis and inference in the 3D space. Comparing to these methods, we propose a novel real-time multimodal hierarchical encoder-decoder neural network that fuses RGB and depth data to realize robotic humanoid grasping in 3D space with only partial observation. The quantification of raw depth data's uncertainty and depth estimation fusing RGB is considered. We develop a general labeling method to label ground-truth on common RGB-D datasets. We evaluate the effectiveness and performance of our method on a physical robot setup and our method achieves over 90\% success rate in both table surface and 3D space scenarios.

Motivation & Objective

• To enable robust robotic grasping in unstructured 3D environments with only partial observations from an eye-in-hand camera system.
• To address the limitations of single-modality methods (RGB or depth-only) by fusing multimodal data to reduce noise and uncertainty.
• To develop a novel, end-to-end deep learning framework that bypasses time-consuming steps like ICP or 3D reconstruction.
• To create a general labeling method for training grasping networks without human-annotated ground-truth, leveraging existing RGB-D datasets with masks.
• To improve grasping reliability by quantifying uncertainty in raw depth data and reconstructing depth via RGB-D fusion.

Proposed method

• Proposes a hierarchical encoder-decoder neural network, UG-Net V3, to fuse RGB and depth features at multiple scales for robust grasp prediction.
• Introduces a confidence network to quantify uncertainty in raw depth images and reconstruct missing depth values using RGB features.
• Uses a novel 8D grasp representation combining 6D object pose (x,y,z,γx,βy,αz), gripper orientation, and opening width.
• Employs a background extraction module to reduce interference from complex backgrounds in depth images.
• Applies surface normal estimation to align the gripper’s orientation with the target object’s surface normal when possible.
• Uses a real-time inference pipeline that converts predicted grasp points in pixel space to 3D pose and gripper control commands.

Experimental results

Research questions

• RQ1 Can hierarchical fusion of RGB and depth features improve robotic grasping performance in partial-observation 3D environments?
• RQ2 How does uncertainty quantification in raw depth data affect grasping policy robustness?
• RQ3 Can RGB-D fusion eliminate the need for ICP or 3D reconstruction while maintaining high grasping success?
• RQ4 What is the impact of observation viewpoint (pitch angle) on grasping performance in partial-observation scenarios?
• RQ5 Can a general labeling method be developed to train grasping networks using existing RGB-D datasets with object masks?

Key findings

• The proposed method achieves a 95% ± 1% success rate on a standard test set, exceeding 90% in both table-top and 3D scenarios.
• The ablation study shows that combining the confidence network and background extraction module improves success rate from 79% to 95% on the test set.
• The method outperforms GQ-CNN (80% success rate) and GG-CNN (84% ± 8%) in terms of success rate and robustness.
• The average inference time is only 32ms, significantly faster than GQ-CNN (800ms), making it suitable for real-time deployment.
• The model size is 19.6MB, which is more efficient than GQ-CNN (75MB), enabling deployment on resource-constrained systems.
• The method achieves a 97% ± 1% robust grasp rate (RGR) on the house test set, indicating high reliability under diverse conditions.