[Paper Review] TAX-Pose: Task-Specific Cross-Pose Estimation for Robot Manipulation

Chuer Pan, Brian Okorn+3 more|arXiv (Cornell University)|Nov 17, 2022

Robot Manipulation and Learning4 citations

TL;DR

TAX-Pose proposes a vision-based method for task-specific cross-pose estimation in robot manipulation, learning the relative pose between interacting object parts (e.g., pan in oven) using cross-object correspondences. It achieves state-of-the-art generalization to unseen objects with as few as 10 real-world demonstrations by modeling the task-specific pose relationship as a learnable, transferable concept.

ABSTRACT

How do we imbue robots with the ability to efficiently manipulate unseen objects and transfer relevant skills based on demonstrations? End-to-end learning methods often fail to generalize to novel objects or unseen configurations. Instead, we focus on the task-specific pose relationship between relevant parts of interacting objects. We conjecture that this relationship is a generalizable notion of a manipulation task that can transfer to new objects in the same category; examples include the relationship between the pose of a pan relative to an oven or the pose of a mug relative to a mug rack. We call this task-specific pose relationship "cross-pose" and provide a mathematical definition of this concept. We propose a vision-based system that learns to estimate the cross-pose between two objects for a given manipulation task using learned cross-object correspondences. The estimated cross-pose is then used to guide a downstream motion planner to manipulate the objects into the desired pose relationship (placing a pan into the oven or the mug onto the mug rack). We demonstrate our method's capability to generalize to unseen objects, in some cases after training on only 10 demonstrations in the real world. Results show that our system achieves state-of-the-art performance in both simulated and real-world experiments across a number of tasks. Supplementary information and videos can be found at https://sites.google.com/view/tax-pose/home.

Motivation & Objective

• To enable robots to generalize manipulation skills to unseen objects in the same category.
• To identify a generalizable, task-specific representation of object interaction beyond individual object poses.
• To reduce reliance on extensive demonstrations by learning from few real-world examples.
• To improve downstream motion planning by estimating relative pose between interacting object parts.
• To establish cross-pose as a transferable concept for zero-shot generalization in manipulation tasks.

Proposed method

• The method defines 'cross-pose' as the relative pose between two interacting object parts, such as a mug relative to a mug rack.
• It learns cross-object correspondences between parts of different objects using a deep neural network trained on RGB images.
• A differentiable rendering module estimates the cross-pose from predicted correspondences and image features.
• The system uses a two-stage pipeline: correspondence prediction followed by differentiable cross-pose regression.
• The learned cross-pose is used to guide a motion planner to achieve the desired interaction configuration.
• The approach is trained end-to-end on real-world demonstrations with minimal supervision.

Experimental results

Research questions

• RQ1 Can a task-specific relative pose representation generalize across novel objects in the same category?
• RQ2 How well can a vision-based system estimate cross-pose from few real-world demonstrations?
• RQ3 Does modeling cross-pose as a learnable, transferable concept improve zero-shot generalization in manipulation tasks?
• RQ4 Can the system achieve high accuracy in both simulation and real-world settings with minimal fine-tuning?
• RQ5 How does the performance of cross-pose estimation compare to baseline methods in unseen object scenarios?

Key findings

• The method achieves state-of-the-art performance in both simulated and real-world manipulation tasks.
• It generalizes to unseen objects after training on only 10 real-world demonstrations.
• The system successfully transfers learned cross-pose relationships to novel objects in the same category, such as placing different mugs on a mug rack.
• The cross-pose representation enables accurate motion planning even when object identities or appearances vary.
• The approach demonstrates robustness in complex, cluttered environments with diverse object shapes and configurations.
• Supplementary videos and results confirm consistent performance across multiple task categories, including pan-in-oven and mug-on-rack scenarios.