[Paper Review] AnyGrasp: Robust and Efficient Grasp Perception in Spatial and Temporal Domains

Haoshu Fang, Chenxi Wang+7 more|arXiv (Cornell University)|Dec 16, 2022

Robot Manipulation and Learning4 citations

TL;DR

AnyGrasp proposes a unified, real-time grasp perception system for robots using a parallel gripper, leveraging dense spatial-temporal supervision with real-world data and center-of-mass awareness to generate 7-DoF, temporally smooth, and robust grasp poses. It achieves 93.3% success in bin-picking over 300 unseen objects and enables dynamic catching of swimming robot fish, outperforming sim-to-real transfer methods.

ABSTRACT

As the basis for prehensile manipulation, it is vital to enable robots to grasp as robustly as humans. Our innate grasping system is prompt, accurate, flexible, and continuous across spatial and temporal domains. Few existing methods cover all these properties for robot grasping. In this paper, we propose AnyGrasp for grasp perception to enable robots these abilities using a parallel gripper. Specifically, we develop a dense supervision strategy with real perception and analytic labels in the spatial-temporal domain. Additional awareness of objects' center-of-mass is incorporated into the learning process to help improve grasping stability. Utilization of grasp correspondence across observations enables dynamic grasp tracking. Our model can efficiently generate accurate, 7-DoF, dense, and temporally-smooth grasp poses and works robustly against large depth-sensing noise. Using AnyGrasp, we achieve a 93.3% success rate when clearing bins with over 300 unseen objects, which is on par with human subjects under controlled conditions. Over 900 mean-picks-per-hour is reported on a single-arm system. For dynamic grasping, we demonstrate catching swimming robot fish in the water. Our project page is at https://graspnet.net/anygrasp.html

Motivation & Objective

• To close the performance gap between human and robotic grasp perception in unstructured, real-world environments.
• To enable dense, 7-DoF, temporally coherent grasp prediction across space and time using a single feed-forward pass.
• To improve robustness against depth sensor noise and object instability through real-world data training and obstacle/center-of-mass awareness.
• To demonstrate generalization on unseen objects and dynamic grasping beyond static scene assumptions.
• To validate the superiority of real-world data over simulated data in grasp perception, especially under low-cost sensing.

Proposed method

• A geometry processing module predicts dense 7-DoF grasp configurations from monocular depth observations in a single forward pass.
• A temporal association module tracks grasp correspondences across consecutive observations using learned embedding similarity.
• Dense supervision is applied using real perception data and analytic labels in the spatial-temporal domain to improve data efficiency.
• The model incorporates implicit obstacle awareness by filtering out collision-prone grasp candidates during training.
• Center-of-mass (COG) awareness is integrated into the supervision signal to enhance grasp stability.
• Real-world data collection with 144 physical objects is used for training, avoiding sim-to-real domain shift.

Experimental results

Research questions

• RQ1 Can a grasp perception system achieve human-level robustness and efficiency in real-world, unstructured environments with a parallel gripper?
• RQ2 Does training on real-world data significantly outperform sim-to-real transfer in grasp perception under depth noise and object diversity?
• RQ3 How effective is dense spatial-temporal supervision in enabling continuous, smooth grasp tracking across time?
• RQ4 To what extent does incorporating center-of-mass and obstacle awareness improve grasp stability and success rate?
• RQ5 Can the system generalize to over 300 unseen objects and dynamic scenarios like catching moving fish?

Key findings

• AnyGrasp achieves a 93.3% success rate in clearing bins with over 300 unseen objects, matching human performance under identical conditions.
• The system achieves over 900 mean picks per hour on a single-arm robot, demonstrating high efficiency.
• It successfully catches small, fast-moving robot fish in water, demonstrating robust dynamic grasping under low-friction conditions.
• The model maintains high performance under significant depth-sensing noise, proving robustness to sensor imperfections.
• Training with real-world data (144 objects) outperforms training with thousands of simulated objects, highlighting the value of real data.
• The ablation study confirms that real-world data, grasp annotation density, and scene diversity significantly impact generalization and performance.