[Paper Review] Multi-Agent Manipulation via Locomotion using Hierarchical Sim2Real

Ofir Nachum, Michael J. Ahn+3 more|arXiv (Cornell University)|Aug 13, 2019

Reinforcement Learning in Robotics37 references37 citations

TL;DR

The paper presents hierarchical sim2real training for quadrupedal robots to achieve manipulation via locomotion, enabling zero-shot transfer to real-world multi-agent tasks through modular low-level locomotion policies and high-level goal-proposing controllers with domain randomization.

ABSTRACT

Manipulation and locomotion are closely related problems that are often studied in isolation. In this work, we study the problem of coordinating multiple mobile agents to exhibit manipulation behaviors using a reinforcement learning (RL) approach. Our method hinges on the use of hierarchical sim2real -- a simulated environment is used to learn low-level goal-reaching skills, which are then used as the action space for a high-level RL controller, also trained in simulation. The full hierarchical policy is then transferred to the real world in a zero-shot fashion. The application of domain randomization during training enables the learned behaviors to generalize to real-world settings, while the use of hierarchy provides a modular paradigm for learning and transferring increasingly complex behaviors. We evaluate our method on a number of real-world tasks, including coordinated object manipulation in a multi-agent setting. See videos at https://sites.google.com/view/manipulation-via-locomotion

Motivation & Objective

• Motivate and address the challenge of coordinating multiple legged agents to perform manipulation tasks.
• Propose a two-level hierarchical RL framework where a low-level goal-reaching policy handles locomotion and a high-level policy directs goals for the low-level policy.
• Leverage domain randomization in a staged (hierarchical) fashion to achieve robust sim2real transfer without real-world fine-tuning.
• Demonstrate zero-shot transfer to real robots on tasks of avoidance, pushing a block, and coordinating two agents to push a long block.

Proposed method

• Train a low-level policy pi_lo to reach randomly sampled 2D goals in the plane with intrinsic reward r_lo including negative distance to the goal and auxiliary terms (upright, heading, bonus).
• Freeze pi_lo and train a high-level policy pi_hi to produce high-level actions that map to low-level goals via a fixed transformation h, with task-specific reward r_hi aggregating short-horizon rewards.
• Use domain randomization separately for low- and high-level training to improve zero-shot transfer, including random height fields in simulation to vary terrain.
• Perform zero-shot transfer to real-world robots by deploying the fixed hierarchical policy without real-world fine-tuning, and add noise to high-level actions to account for residual gaps.
• Apply the framework to three quadrupedal tasks (Avoid, Push, Coordinate) and a multi-agent Coordinate task to demonstrate coordination and manipulation via locomotion.

Experimental results

Research questions

• RQ1 Can hierarchical sim2real enable robust zero-shot transfer of learned manipulation via locomotion from simulation to real-world multi-agent tasks?
• RQ2 How does modularizing learning into low-level locomotion and high-level task control affect domain randomization requirements and transfer performance?
• RQ3 What domain randomizations at each hierarchical level are most beneficial for real-world transfer in quadrupedal manipulation tasks?
• RQ4 To what extent can two quadrupeds coordinate to push a long block to a target location and orientation in the real world?
• RQ5 What ablations reveal the importance of hierarchy and high-level randomization in achieving success on these tasks?

Key findings

• Zero-shot real-world transfer is achievable for locomotion-driven manipulation tasks using hierarchical sim2real with domain randomization.
• Hierarchy substantially improves success rates across Avoid, Push, and Coordinate tasks compared to non-hierarchical ablations.
• Introducing high-level randomization in addition to a fixed low-level policy yields further gains, achieving up to 90% success on the Coordinate task in real-world trials.
• Low-level randomizations, notably including a randomized height field, are crucial for robust walking gaits and significantly increase real-world performance.
• Even with no high-level randomization, the high-level training in simulation can achieve good zero-shot real-world results, and adding targeted high-level randomization provides incremental improvements.
• Table-based results show clear performance gaps between hierarchical and non-hierarchical setups, underscoring the necessity of hierarchy for scalable sim2real domain randomization.