[Paper Review] Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent Objects

Jeffrey Ichnowski, Yahav Avigal+2 more|arXiv (Cornell University)|Oct 27, 2021

Robot Manipulation and Learning36 references61 citations

TL;DR

Dex-NeRF uses NeRF-based scene reconstruction with transparency-aware depth rendering and lighting tricks to enable Dex-Net to reliably grasp transparent objects, achieving high physical grasp success on a YuMi robot where baselines fail.

ABSTRACT

The ability to grasp and manipulate transparent objects is a major challenge for robots. Existing depth cameras have difficulty detecting, localizing, and inferring the geometry of such objects. We propose using neural radiance fields (NeRF) to detect, localize, and infer the geometry of transparent objects with sufficient accuracy to find and grasp them securely. We leverage NeRF's view-independent learned density, place lights to increase specular reflections, and perform a transparency-aware depth-rendering that we feed into the Dex-Net grasp planner. We show how additional lights create specular reflections that improve the quality of the depth map, and test a setup for a robot workcell equipped with an array of cameras to perform transparent object manipulation. We also create synthetic and real datasets of transparent objects in real-world settings, including singulated objects, cluttered tables, and the top rack of a dishwasher. In each setting we show that NeRF and Dex-Net are able to reliably compute robust grasps on transparent objects, achieving 90% and 100% grasp success rates in physical experiments on an ABB YuMi, on objects where baseline methods fail.

Motivation & Objective

• Motivate robotic manipulation of transparent objects which challenge traditional depth sensing.
• Propose Dex-NeRF to recover geometry of transparent objects via NeRF for grasp planning.
• Show that added lighting improves NeRF learning of transparent geometry for reliable grasps.
• Provide synthetic and real datasets of transparent objects for NeRF-based training and evaluation.

Proposed method

• Train NeRF on multi-view images with known camera intrinsics/extrinsics to recover scene density and view-dependent color.
• Render depth maps from NeRF using a transparency-aware approach to feed Dex-Net for grasp planning.
• Increase object visibility to NeRF by placing multiple lights to induce specular reflections that reveal geometry.
• Use a fixed, pre-trained Dex-Net grasp planner to evaluate candidate grasps from NeRF-derived depth.
• Create synthetic and real datasets with transparent objects and multiple camera poses for training and evaluation.
• Compare depth rendering methods (Vanilla NeRF vs transparency-aware) and light setups in physical experiments.

Experimental results

Research questions

• RQ1 Can NeRF learn the geometry of transparent objects sufficiently for reliable robotic grasping?
• RQ2 Does adding lighting to induce specular reflections improve NeRF’s reconstruction of transparent objects?
• RQ3 Is a transparency-aware depth rendering necessary for successful Dex-Net grasp planning on transparent objects?
• RQ4 How do synthetic and real-image datasets of transparent objects support training NeRF for grasping?

Key findings

• Dex-NeRF achieves 90% to 100% grasp success in physical experiments on ABB YuMi for tested transparent objects.
• Dex-NeRF significantly outperforms PhoXi-depth and Vanilla NeRF baselines in physical grasping tests (10/10 vs 0/10 or 9/10 for several objects).
• Depth rendering from transparency-aware NeRF enables Dex-Net to generate reliable grasps in cluttered and singulated transparent-object scenes.
• Using a 5x5 lights array improves NeRF’s geometry recovery over a single light by increasing observable specular reflections.
• Grasp success rates improve with NeRF training iterations and can plateau around 50k–60k iterations for synthetic singulated objects.
• A grid of overhead cameras enhances depth quality for grasp planning, with diminishing returns beyond a 7x7 arrangement.