[论文解读] Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent Objects

Jeffrey Ichnowski, Yahav Avigal+2 more|arXiv (Cornell University)|Oct 27, 2021

Robot Manipulation and Learning参考文献 36被引用 61

一句话总结

Dex-NeRF 使用基于 NeRF 的场景重建、透明度感知深度渲染和照明技巧，使 Dex-Net 能够可靠地抓取透明物体，在 YuMi 机器人上的物理抓取成功率达到高水平，而基线方法失败。

ABSTRACT

The ability to grasp and manipulate transparent objects is a major challenge for robots. Existing depth cameras have difficulty detecting, localizing, and inferring the geometry of such objects. We propose using neural radiance fields (NeRF) to detect, localize, and infer the geometry of transparent objects with sufficient accuracy to find and grasp them securely. We leverage NeRF's view-independent learned density, place lights to increase specular reflections, and perform a transparency-aware depth-rendering that we feed into the Dex-Net grasp planner. We show how additional lights create specular reflections that improve the quality of the depth map, and test a setup for a robot workcell equipped with an array of cameras to perform transparent object manipulation. We also create synthetic and real datasets of transparent objects in real-world settings, including singulated objects, cluttered tables, and the top rack of a dishwasher. In each setting we show that NeRF and Dex-Net are able to reliably compute robust grasps on transparent objects, achieving 90% and 100% grasp success rates in physical experiments on an ABB YuMi, on objects where baseline methods fail.

研究动机与目标

• Motivate robotic manipulation of transparent objects which challenge traditional depth sensing.
• Propose Dex-NeRF to recover geometry of transparent objects via NeRF for grasp planning.
• Show that added lighting improves NeRF learning of transparent geometry for reliable grasps.
• Provide synthetic and real datasets of transparent objects for NeRF-based training and evaluation.

提出的方法

• Train NeRF on multi-view images with known camera intrinsics/extrinsics to recover scene density and view-dependent color.
• Render depth maps from NeRF using a transparency-aware approach to feed Dex-Net for grasp planning.
• Increase object visibility to NeRF by placing multiple lights to induce specular reflections that reveal geometry.
• Use a fixed, pre-trained Dex-Net grasp planner to evaluate candidate grasps from NeRF-derived depth.
• Create synthetic and real datasets with transparent objects and multiple camera poses for training and evaluation.
• Compare depth rendering methods (Vanilla NeRF vs transparency-aware) and light setups in physical experiments.

实验结果

研究问题

• RQ1 Can NeRF learn the geometry of transparent objects sufficiently for reliable robotic grasping?
• RQ2 Does adding lighting to induce specular reflections improve NeRF’s reconstruction of transparent objects?
• RQ3 Is a transparency-aware depth rendering necessary for successful Dex-Net grasp planning on transparent objects?
• RQ4 How do synthetic and real-image datasets of transparent objects support training NeRF for grasping?

主要发现

• Dex-NeRF achieves 90% to 100% grasp success in physical experiments on ABB YuMi for tested transparent objects.
• Dex-NeRF significantly outperforms PhoXi-depth and Vanilla NeRF baselines in physical grasping tests (10/10 vs 0/10 or 9/10 for several objects).
• Depth rendering from transparency-aware NeRF enables Dex-Net to generate reliable grasps in cluttered and singulated transparent-object scenes.
• Using a 5x5 lights array improves NeRF’s geometry recovery over a single light by increasing observable specular reflections.
• Grasp success rates improve with NeRF training iterations and can plateau around 50k–60k iterations for synthetic singulated objects.
• A grid of overhead cameras enhances depth quality for grasp planning, with diminishing returns beyond a 7x7 arrangement.