[Paper Review] Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics

Jeffrey Mahler, Jacky Liang+6 more|arXiv (Cornell University)|Mar 27, 2017

Robot Manipulation and Learning2 references276 citations

TL;DR

Dex-Net 2.0 trains a Grasp Quality CNN on 6.7 million synthetic depth images and analytic grasp metrics to rapidly predict robust grasps, enabling fast, high-precision robotic grasping on real robots.

ABSTRACT

To reduce data collection time for deep learning of robust robotic grasp plans, we explore training from a synthetic dataset of 6.7 million point clouds, grasps, and analytic grasp metrics generated from thousands of 3D models from Dex-Net 1.0 in randomized poses on a table. We use the resulting dataset, Dex-Net 2.0, to train a Grasp Quality Convolutional Neural Network (GQ-CNN) model that rapidly predicts the probability of success of grasps from depth images, where grasps are specified as the planar position, angle, and depth of a gripper relative to an RGB-D sensor. Experiments with over 1,000 trials on an ABB YuMi comparing grasp planning methods on singulated objects suggest that a GQ-CNN trained with only synthetic data from Dex-Net 2.0 can be used to plan grasps in 0.8sec with a success rate of 93% on eight known objects with adversarial geometry and is 3x faster than registering point clouds to a precomputed dataset of objects and indexing grasps. The Dex-Net 2.0 grasp planner also has the highest success rate on a dataset of 10 novel rigid objects and achieves 99% precision (one false positive out of 69 grasps classified as robust) on a dataset of 40 novel household objects, some of which are articulated or deformable. Code, datasets, videos, and supplementary material are available at http://berkeleyautomation.github.io/dex-net .

Motivation & Objective

• Motivate robust grasp planning under sensing and actuation uncertainty.
• Reduce data collection time by using synthetic data to train a grasp robustness model.
• Predict grasp success directly from depth images instead of relying on explicit object models.
• Leverage analytic grasp metrics to label synthetic data for robust grasp planning.
• Evaluate the trained planner on real robot experiments and novel objects.

Proposed method

• Generate a 6.7 million-point cloud dataset Dex-Net 2.0 by combining 1,500 3D object models with antipodal parallel-jaw grasps and robust analytic metrics.
• Train a Grasp Quality Convolutional Neural Network (GQ-CNN) to estimate grasp robustness Q(u,y) from depth-image inputs.
• Represent grasps as planar pose and depth relative to a camera and align depth images to the grasp center/orientation to simplify learning.
• Use Monte-Carlo sampling of object pose, gripper pose, and friction to compute the robust epsilon quality E_Q for each candidate grasp.
• Plan grasps by sampling antipodal candidates, ranking by predicted robustness, and executing the top feasible grasp.
• Evaluate performance on real-robot trials and compare with image-based metrics, random forests, SVMs, and registration-based approaches.

Experimental results

Research questions

• RQ1 Can a CNN trained exclusively on synthetic data predict robust grasps from depth images?
• RQ2 How does a synthetic-data-trained grasp planner compare to perception-plus-registration baselines on known and novel objects?
• RQ3 What is the impact of training data size and noise modeling on grasp robustness classification and planning performance?
• RQ4 Is it possible to achieve high success and precision with a fast, depth-image–based grasp planner on a real robot?

Key findings

• The Dex-Net 2.0 planner achieves 0.8 seconds planning time and 93% success on eight known objects with adversarial geometry.
• Dex-Net 2.0 planning is about 3 times faster than a registration-based approach that matches to a precomputed database.
• On a dataset of 10 novel rigid objects, the Dex-Net 2.0 planner yields the highest success rate among evaluated methods.
• On a dataset of 40 novel household objects, the planner achieves 99% precision (one false positive out of 69 robust classifications).
• The GQ-CNN trained purely on synthetic data generalizes to novel objects better than several baselines (IGQ, REG, ML-RF, ML-SVM) in the reported experiments.