[Paper Review] Same Object, Different Grasps: Data and Semantic Knowledge for Task-Oriented Grasping

Adithyavairavan Murali, Weiyu Liu+3 more|arXiv (Cornell University)|Nov 12, 2020

Robot Manipulation and LearningEngineering45 references18 citations

TL;DR

This paper introduces the TaskGrasp dataset—250K task-oriented grasps across 191 objects and 56 tasks—and proposes GCNGrasp, a graph neural network framework that leverages semantic knowledge from a knowledge graph and word embeddings to generalize grasping policies to unseen objects and tasks. The method achieves a 12% improvement on held-out tasks and 3.5% on held-out objects compared to baseline methods without semantic reasoning.

ABSTRACT

Despite the enormous progress and generalization in robotic grasping in recent years, existing methods have yet to scale and generalize task-oriented grasping to the same extent. This is largely due to the scale of the datasets both in terms of the number of objects and tasks studied. We address these concerns with the TaskGrasp dataset which is more diverse both in terms of objects and tasks, and an order of magnitude larger than previous datasets. The dataset contains 250K task-oriented grasps for 56 tasks and 191 objects along with their RGB-D information. We take advantage of this new breadth and diversity in the data and present the GCNGrasp framework which uses the semantic knowledge of objects and tasks encoded in a knowledge graph to generalize to new object instances, classes and even new tasks. Our framework shows a significant improvement of around 12% on held-out settings compared to baseline methods which do not use semantics. We demonstrate that our dataset and model are applicable for the real world by executing task-oriented grasps on a real robot on unknown objects. Code, data and supplementary video could be found at https://sites.google.com/view/taskgrasp

Motivation & Objective

• To address the lack of large-scale, diverse datasets for task-oriented robotic grasping, especially in terms of object variety and task diversity.
• To overcome the generalization gap in robotic grasping by incorporating semantic knowledge of objects and tasks into learning frameworks.
• To enable robust zero-shot generalization to new object instances, classes, and unseen tasks through structured semantic reasoning.
• To scale human-annotated data collection for 3D task-oriented grasping using Amazon Mechanical Turk with quality control via qualification tests and majority voting.
• To demonstrate real-world applicability by executing task-oriented grasps on a physical robot using unknown objects.

Proposed method

• The TaskGrasp dataset is collected via Amazon Mechanical Turk, using a two-stage annotation pipeline: first assessing task suitability per object, then labeling 6-DOF grasps for valid task-object pairs.
• Each grasp is annotated using RGB-D point clouds and multiple 2D visualizations to reduce ambiguity in 3D labeling from a 2D interface.
• A knowledge graph is constructed using WordNet to encode hierarchical and relational semantics between objects and tasks (e.g., mug → container → instrumentality → entity).
• GCNGrasp employs a Graph Convolutional Network (GCN) to reason over the knowledge graph, fusing it with point cloud features from the object and task embeddings from pre-trained language models.
• The model uses multi-view point cloud representations and learns end-to-end to predict task-specific grasps, with semantic priors improving generalization.
• Label quality is ensured through a qualification test and redundant labeling with three annotators per grasp, using majority voting for final labels.

Experimental results

Research questions

• RQ1 Can a large-scale, diverse dataset of task-oriented grasps improve generalization in robotic grasping beyond stable grasping?
• RQ2 How effectively can semantic knowledge from a knowledge graph and pre-trained word embeddings improve zero-shot generalization to new objects and tasks?
• RQ3 To what extent does incorporating structured semantic relationships between objects and tasks enhance grasp policy performance on held-out settings?
• RQ4 Can human-annotated 3D grasp data be reliably collected at scale using crowd-sourcing with quality control mechanisms?
• RQ5 Can a learned grasp policy generalize to real-world execution on unknown objects without fine-tuning?

Key findings

• The TaskGrasp dataset contains 250,000 task-oriented grasps across 191 real-world objects and 56 distinct tasks, representing a 10x increase in scale and diversity over prior datasets.
• GCNGrasp achieves a 12% absolute improvement on held-out tasks and a 3.5% improvement on held-out object classes compared to baseline methods that do not use semantic knowledge.
• The use of a knowledge graph with WordNet-based hierarchies and pre-trained word embeddings significantly enhances generalization, particularly in zero-shot settings.
• The annotation pipeline using Amazon Mechanical Turk with qualification tests and three-annotator redundancy achieved high label consistency, with performance saturating at three annotators.
• The framework successfully generalizes to new object instances and classes, and real-world robot deployment confirms applicability beyond simulation.
• The ablation study confirms that semantic knowledge is critical for generalization, as models without it fail to generalize beyond seen data.