[Paper Review] CaTGrasp: Learning Category-Level Task-Relevant Grasping in Clutter from Simulation

Bowen Wen, Wenzhao Lian+2 more|arXiv (Cornell University)|Sep 19, 2021

Robot Manipulation and Learning4 citations

TL;DR

CaTGrasp proposes a simulation-only framework for learning category-level, task-relevant grasping in dense clutter, using a novel non-uniform normalized object coordinate space (NUNOCS) to enable dense 3D correspondence and transfer of task-relevant grasp knowledge across diverse object instances. It achieves 93.3% task-relevant grasp success in real-world testing, outperforming baselines by significant margins without real-world fine-tuning.

ABSTRACT

Task-relevant grasping is critical for industrial assembly, where downstream manipulation tasks constrain the set of valid grasps. Learning how to perform this task, however, is challenging, since task-relevant grasp labels are hard to define and annotate. There is also yet no consensus on proper representations for modeling or off-the-shelf tools for performing task-relevant grasps. This work proposes a framework to learn task-relevant grasping for industrial objects without the need of time-consuming real-world data collection or manual annotation. To achieve this, the entire framework is trained solely in simulation, including supervised training with synthetic label generation and self-supervised, hand-object interaction. In the context of this framework, this paper proposes a novel, object-centric canonical representation at the category level, which allows establishing dense correspondence across object instances and transferring task-relevant grasps to novel instances. Extensive experiments on task-relevant grasping of densely-cluttered industrial objects are conducted in both simulation and real-world setups, demonstrating the effectiveness of the proposed framework. Code and data are available at https://sites.google.com/view/catgrasp.

Motivation & Objective

• To address the challenge of learning task-relevant grasping for industrial objects in densely cluttered environments without manual annotation or real-world data collection.
• To enable generalization to novel, unseen object instances with shape and size variations within the same category.
• To model dense, point-wise task relevance across 3D shapes without relying on sparse keypoint annotations.
• To develop a canonical, object-centric representation that supports reliable dense correspondence across category-level instances.
• To train the entire framework end-to-end in simulation and deploy it directly in the real world with no fine-tuning.

Proposed method

• Introduces NUNOCS, a non-uniformly scaled canonical coordinate space that enables fine-grained dense correspondence across 3D object instances with large shape variations.
• Leverages self-supervised hand-object interaction in simulation to generate dense, task-relevant contact heatmaps that indicate likelihood of successful downstream task execution.
• Uses a hybrid grasp proposal mechanism combining a learned grasp codebook with category-level knowledge transfer via NUNOCS for improved coverage and stability.
• Employs supervised training with synthetic labels generated via simulation, including both geometric stability and task-relevance signals.
• Trains the entire system end-to-end in simulation using a combination of supervised and self-supervised objectives to learn category-level priors.
• Transfers the learned grasp policy directly to real-world novel instances without retraining or 3D model acquisition.

Experimental results

Research questions

• RQ1 Can task-relevant grasping be learned effectively in simulation without human-annotated labels or real-world data collection?
• RQ2 Can a canonical 3D representation enable reliable dense correspondence across diverse object instances within the same category?
• RQ3 Does modeling dense, point-wise task relevance outperform sparse keypoint-based approaches in complex cluttered scenarios?
• RQ4 Can a simulation-trained policy generalize to real-world, novel object instances with unseen dimensions and shape variations?
• RQ5 How does the proposed NUNOCS representation compare to standard NOCS in enabling knowledge transfer for task-relevant grasping?

Key findings

• The proposed method achieved 93.3% task-relevant grasp success rate in real-world testing, significantly outperforming all baselines, including PointNetGPD and Ours-NA.
• In simulation, the method achieved 94.5% task-relevant grasp success, demonstrating strong generalization from synthetic data to real-world deployment.
• The NUNOCS representation enabled superior knowledge transfer compared to standard NOCS, particularly on highly variable objects like HMN2, where performance gap was most pronounced.
• The method achieved high stable grasp success (83.3%) while maintaining strong task relevance, indicating effective coverage of occluded and complex regions via the grasp codebook.
• The performance gap between simulation and real-world was smallest for the Screws category, though still notable due to the challenges of tiny, reflective, and easily rolling objects.
• The ablation study confirmed that the dense contact heatmap and NUNOCS representation were critical for achieving high task-relevance, as Ours-NOCS and Ours-NA showed significantly lower success rates.