[论文解读] Multi-Agent Manipulation via Locomotion using Hierarchical Sim2Real

Ofir Nachum, Michael J. Ahn+3 more|arXiv (Cornell University)|Aug 13, 2019

Reinforcement Learning in Robotics参考文献 37被引用 37

一句话总结

论文提出了用于四足机器人通过移动实现操控的分层仿真到现实训练，通过模块化的低级 locomotion 策略和高级目标提议控制器以及领域随机化，实现对现实世界多智能体任务的零-shot 转移。

ABSTRACT

Manipulation and locomotion are closely related problems that are often studied in isolation. In this work, we study the problem of coordinating multiple mobile agents to exhibit manipulation behaviors using a reinforcement learning (RL) approach. Our method hinges on the use of hierarchical sim2real -- a simulated environment is used to learn low-level goal-reaching skills, which are then used as the action space for a high-level RL controller, also trained in simulation. The full hierarchical policy is then transferred to the real world in a zero-shot fashion. The application of domain randomization during training enables the learned behaviors to generalize to real-world settings, while the use of hierarchy provides a modular paradigm for learning and transferring increasingly complex behaviors. We evaluate our method on a number of real-world tasks, including coordinated object manipulation in a multi-agent setting. See videos at https://sites.google.com/view/manipulation-via-locomotion

研究动机与目标

• 激发并解决协调多只四足智能体执行操控任务的挑战。
• 提出一个两级分层强化学习框架，其中低级目标到达策略负责运动，A级策略为低级策略指引目标。
• 以分阶段（分层）的方式利用领域随机化，实现无需现实世界微调的鲁棒 sim2real 迁移。
• 在人畏 Avoid、推动物块、协调两台机器人共同推动一块长块的任务上，展示对真实机器人零-shot 转移。

提出的方法

• 训练一个低级策略 pi_lo，使其在平面上达到随机采样的二维目标，其内在奖励 r_lo 包括到目标的负距离及辅助项（直立、朝向、奖励）。
• 冻结 pi_lo 并训练高级策略 pi_hi 以产生通过固定变换 h 映射到低级目标的高级动作，任务特定的奖励 r_hi 汇总短期奖励。
• 分别对低级和高级训练应用领域随机化以提升零-shot 迁移，包括在仿真中使用随机高度场以改变地形。
• 通过部署固定的分层策略在现实世界机器人上进行零-shot 转移，无需现实世界微调，并对高级动作添加噪声以弥补残留差距。
• 将该框架应用于三个四足任务（Avoid、Push、Coordinate）以及一个多智能体 Coordinate 任务，以演示通过运动实现的协调与操控。

实验结果

研究问题

• RQ1 Can hierarchical sim2real enable robust zero-shot transfer of learned manipulation via locomotion from simulation to real-world multi-agent tasks?
• RQ2 How does modularizing learning into low-level locomotion and high-level task control affect domain randomization requirements and transfer performance?
• RQ3 What domain randomizations at each hierarchical level are most beneficial for real-world transfer in quadrupedal manipulation tasks?
• RQ4 To what extent can two quadrupeds coordinate to push a long block to a target location and orientation in the real world?
• RQ5 What ablations reveal the importance of hierarchy and high-level randomization in achieving success on these tasks?

主要发现

• Zero-shot real-world transfer is achievable for locomotion-driven manipulation tasks using hierarchical sim2real with domain randomization.
• Hierarchy substantially improves success rates across Avoid, Push, and Coordinate tasks compared to non-hierarchical ablations.
• Introducing high-level randomization in addition to a fixed low-level policy yields further gains, achieving up to 90% success on the Coordinate task in real-world trials.
• Low-level randomizations, notably including a randomized height field, are crucial for robust walking gaits and significantly increase real-world performance.
• Even with no high-level randomization, the high-level training in simulation can achieve good zero-shot real-world results, and adding targeted high-level randomization provides incremental improvements.
• Table-based results show clear performance gaps between hierarchical and non-hierarchical setups, underscoring the necessity of hierarchy for scalable sim2real domain randomization.