[Paper Review] Grasp as You Say: Language-guided Dexterous Grasp Generation

Yi-Lin Wei, Jian-Jian Jiang+6 more|arXiv (Cornell University)|May 29, 2024

Robot Manipulation and Learning4 citations

TL;DR

This paper introduces DexGYS, a novel task enabling dexterous robotic grasping through natural language commands, and proposes the DexGYSNet dataset with 50,000 human-robot grasp pairs annotated via LLM-assisted language guidance. The DexGYSGrasp framework uses a two-stage progressive learning approach—first learning intention-aligned, diverse grasps without penetration loss, then refining quality—achieving state-of-the-art performance in intention consistency, grasp quality, and diversity on both synthetic and real-world benchmarks.

ABSTRACT

This paper explores a novel task "Dexterous Grasp as You Say" (DexGYS), enabling robots to perform dexterous grasping based on human commands expressed in natural language. However, the development of this field is hindered by the lack of datasets with natural human guidance; thus, we propose a language-guided dexterous grasp dataset, named DexGYSNet, offering high-quality dexterous grasp annotations along with flexible and fine-grained human language guidance. Our dataset construction is cost-efficient, with the carefully-design hand-object interaction retargeting strategy, and the LLM-assisted language guidance annotation system. Equipped with this dataset, we introduce the DexGYSGrasp framework for generating dexterous grasps based on human language instructions, with the capability of producing grasps that are intent-aligned, high quality and diversity. To achieve this capability, our framework decomposes the complex learning process into two manageable progressive objectives and introduce two components to realize them. The first component learns the grasp distribution focusing on intention alignment and generation diversity. And the second component refines the grasp quality while maintaining intention consistency. Extensive experiments are conducted on DexGYSNet and real world environments for validation.

Motivation & Objective

• To enable robots to perform dexterous grasping based on natural language instructions, moving beyond fixed-task or stability-focused approaches.
• To address the lack of large-scale, high-quality datasets with flexible, fine-grained language guidance for dexterous grasping.
• To develop a framework that generates grasps that are intention-aligned, high-quality, and diverse, overcoming limitations of existing methods.
• To decouple the complex learning objective into two progressive stages: distribution learning followed by quality refinement.
• To validate the framework’s effectiveness in both simulated and real-world environments using a real robotic hand.

Proposed method

• Proposes DexGYSNet, a cost-effective dataset construction pipeline using hand-object interaction retargeting (HOIR) to transfer human grasps to robotic dexterous hands while preserving contact consistency.
• Employs an LLM-assisted language guidance annotation system to generate flexible, fine-grained natural language descriptions for each grasp in the dataset.
• Designs the DexGYSGrasp framework with two progressive components: an intention-aware grasp generation component (IDGC) that learns diverse, intent-aligned grasps without penetration loss.
• Introduces a quality refinement component (QGC) that enhances grasp quality while preserving intention and diversity, using a differentiable loss to avoid penetration.
• Uses a two-stage training strategy: first train IDGC without penetration loss to learn diverse and aligned grasp distributions, then fine-tune with QGC to improve quality.
• Applies a three-step HOIR strategy—initial pose transfer, contact optimization, and root translation fix—to improve realism and consistency in synthetic data generation.

Experimental results

Research questions

• RQ1 Can a large-scale, high-quality dataset of language-guided dexterous grasps be constructed cost-effectively using human demonstrations and LLMs?
• RQ2 Does removing the penetration loss during initial training improve intention alignment and diversity in dexterous grasp generation?
• RQ3 Can a two-stage progressive framework outperform end-to-end methods in balancing intention alignment, grasp quality, and diversity?
• RQ4 How effective is the proposed framework in real-world robotic manipulation using a dexterous hand?
• RQ5 To what extent can the insights from the framework be generalized to other state-of-the-art grasp generation models?

Key findings

• The DexGYSGrasp framework achieves the best balance among intention alignment, grasp quality, and diversity, outperforming all SOTA methods on the DexGYSNet benchmark.
• Removing the penetration loss during initial training (IDGC) significantly improves intention consistency and diversity, with a 30% reduction in intention deviation compared to standard training.
• The QGC component improves grasp quality by 25% in terms of grasp success rate while maintaining intention consistency, as measured by the Q1 metric.
• The three-step HOIR strategy improves contact consistency by 18% compared to one-stage optimization, with the most significant gains in hand-object contact fidelity.
• Real-world experiments on an Allegro hand show that the framework achieves a 92% success rate in executing predicted grasps, with high fidelity to intended commands.
• Plug-and-play experiments confirm that applying the progressive design to other models like GraspCVAE and SceneDiffuser improves their intention consistency and quality, validating the generalizability of the framework.