[Paper Review] Generalization in Dexterous Manipulation via Geometry-Aware Multi-Task Learning

Wenlong Huang, Igor Mordatch+2 more|arXiv (Cornell University)|Nov 4, 2021

Reinforcement Learning in Robotics41 references24 citations

TL;DR

The paper demonstrates that a single geometry-aware, multi-task policy can perform in-hand manipulation on over 100 diverse real-world objects and generalize to unseen geometries, often outperforming single-object specialists. It introduces a point-cloud based object representation and multi-task learning to achieve zero-shot generalization.

ABSTRACT

Dexterous manipulation of arbitrary objects, a fundamental daily task for humans, has been a grand challenge for autonomous robotic systems. Although data-driven approaches using reinforcement learning can develop specialist policies that discover behaviors to control a single object, they often exhibit poor generalization to unseen ones. In this work, we show that policies learned by existing reinforcement learning algorithms can in fact be generalist when combined with multi-task learning and a well-chosen object representation. We show that a single generalist policy can perform in-hand manipulation of over 100 geometrically-diverse real-world objects and generalize to new objects with unseen shape or size. Interestingly, we find that multi-task learning with object point cloud representations not only generalizes better but even outperforms the single-object specialist policies on both training as well as held-out test objects. Video results at https://huangwl18.github.io/geometry-dex

Motivation & Objective

• Motivate dexterous in-hand manipulation that generalizes across many objects.
• Show that multi-task learning can match or exceed single-task specialists on trained objects.
• Introduce a geometry-aware object representation to improve generalization to unseen objects.
• Demonstrate that representation-augmented multi-task learning scales with more training objects.

Proposed method

• Extend DDPG with Hindsight Experience Replay to a multi-task setting by summing task gradients across objects.
• Introduce a geometry-aware object representation encoder based on two-point-cloud inputs (current and goal orientation) to predict object class and relative rotation.
• Pre-train the encoder on object point clouds and freeze it during RL training where both actor and critic conditioning incorporate the encoded representation.
• Train a single policy across many objects (Vanilla Multi-Task Policy) and compare to per-object oracle policies.
• Evaluate zero-shot generalization on a held-out set of geometrically diverse objects (114 total objects; 85 train, 29 test).
• Provide a simulated OpenAI Gym-based benchmark (Shadow Hand) and release datasets for future work.

Experimental results

Research questions

• RQ1 Can a vanilla multi-task policy achieve competitive performance across a large set of objects compared to per-object specialists?
• RQ2 Does adding a geometry-aware object representation improve zero-shot generalization to unseen objects?
• RQ3 How does the number of training objects affect generalization to held-out objects?
• RQ4 Is freezing the pre-trained object encoder beneficial versus fine-tuning it during policy training?

Key findings

• A single multi-task policy trained on 85 objects matches the average performance of per-object oracle policies and is 17x more sample-efficient (200M vs 3400M samples).
• With a geometry-aware object representation, the multi-task policy can outperform per-object oracles on held-out objects in zero-shot evaluation.
• Object representation benefits persist when used with single-task policies, improving held-out and training-object performance.
• Increasing the number of training objects improves zero-shot generalization, with representation-based methods showing stronger scaling.
• Freezing the encoder yields better performance than fine-tuning it during policy training, and pre-training on many objects improves single-task performance on held-out objects.