[Paper Review] Learning 6-DoF Fine-grained Grasp Detection Based on Part Affordance Grounding

Yaoxian Song, Penglei Sun+8 more|arXiv (Cornell University)|Jan 27, 2023

Robot Manipulation and Learning4 citations

TL;DR

This paper proposes Lang-SHAPE, a large-scale language-guided 3D shape grasping dataset with part-level affordance annotations, and PIONEER, a two-stage 6-DoF grasp detection network that grounds natural language queries to 3D parts for fine-grained robotic grasping. The method achieves 42.07% success rate in part-aware grasping and 52.58% in part-specific settings, outperforming baselines by over 20% in key metrics, demonstrating strong generalization and real-world effectiveness.

ABSTRACT

Robotic grasping is a fundamental ability for a robot to interact with the environment. Current methods focus on how to obtain a stable and reliable grasping pose in object level, while little work has been studied on part (shape)-wise grasping which is related to fine-grained grasping and robotic affordance. Parts can be seen as atomic elements to compose an object, which contains rich semantic knowledge and a strong correlation with affordance. However, lacking a large part-wise 3D robotic dataset limits the development of part representation learning and downstream applications. In this paper, we propose a new large Language-guided SHape grAsPing datasEt (named LangSHAPE) to promote 3D part-level affordance and grasping ability learning. From the perspective of robotic cognition, we design a two-stage fine-grained robotic grasping framework (named LangPartGPD), including a novel 3D part language grounding model and a part-aware grasp pose detection model, in which explicit language input from human or large language models (LLMs) could guide a robot to generate part-level 6-DoF grasping pose with textual explanation. Our method combines the advantages of human-robot collaboration and LLMs' planning ability using explicit language as a symbolic intermediate. To evaluate the effectiveness of our proposed method, we perform 3D part grounding and fine-grained grasp detection experiments on both simulation and physical robot settings, following language instructions across different degrees of textual complexity. Results show our method achieves competitive performance in 3D geometry fine-grained grounding, object affordance inference, and 3D part-aware grasping tasks. Our dataset and code are available on our project website https://sites.google.com/view/lang-shape

Motivation & Objective

• To address the lack of large-scale, part-level 3D robotic grasping datasets that incorporate affordance and human language instructions.
• To enable fine-grained robotic grasping by grounding natural language queries to specific 3D object parts for improved affordance understanding.
• To develop a novel 6-DoF grasp detection model that integrates 3D part language grounding with part-aware grasp pose prediction.
• To evaluate the model’s performance on compositional generalization and real-world deployment using physical robot systems.
• To bridge the gap between high-level human language intent and low-level robotic control through pre-trained language models and 3D scene understanding.

Proposed method

• Proposes Lang-SHAPE, a large-scale 3D dataset with 1.85M language-referring expressions, 35 part categories, and 6-DoF grasp annotations across 300+ objects.
• Designs a two-stage network: first, a 3D part language grounding model to localize language-desired parts in point clouds; second, a part-aware grasp detection head based on PointNetGPD.
• Introduces a visual servo eye-in-hand scanning policy to capture multi-view point clouds for complete object representation.
• Uses pre-trained language models to encode referring expressions and aligns them with 3D part features via contrastive learning.
• Applies a region-constrained sampling strategy guided by language grounding to improve grasp quality and reduce failure cases.
• Deploys the model on a real robot (Kinova Jaco 7-DoF) with an eye-in-hand camera for real-world inference and grasping.

Experimental results

Research questions

• RQ1 Can a 3D part-aware grasp detection model grounded in natural language achieve high success rates in fine-grained grasping tasks?
• RQ2 How well does the model generalize to unseen object-part combinations and compositional language queries?
• RQ3 To what extent does language grounding improve affordance inference and grasp reliability compared to object-wise baselines?
• RQ4 How effective is the proposed method in real-world robotic manipulation with human-robot interaction?
• RQ5 Does the integration of pre-trained language models enhance zero-shot or few-shot generalization in 3D grasping?

Key findings

• PIONEER achieves a 42.07% success rate in part-wise grasping detection on the Lang-SHAPE dataset, outperforming the baseline (Baseline 3) by 16.96 percentage points.
• In part-specific language mode, the success rate increases to 52.58%, demonstrating the benefit of explicit human-part grounding.
• The model shows strong compositional generalization, with improved performance on unseen object-part combinations, indicating robustness to distribution shift.
• Real-robot experiments confirm the method's effectiveness in real-world settings, successfully executing language-guided grasping on seen and unseen objects.
• The trial cost increases slightly due to region constraints, but the performance gain in success rate justifies the trade-off.
• The model achieves 48.76% part-agnostic success rate and 15.53 average sampling trials, indicating a balance between accuracy and efficiency.