[论文解读] Benchmarking Reinforcement Learning Algorithms on Real-World Robots

A. Rupam Mahmood, Dmytro Korenkevych+3 more|arXiv (Cornell University)|Sep 20, 2018

Reinforcement Learning in Robotics被引用 46

一句话总结

本文展示了跨三个机器人、六个现实世界的机器人强化学习任务，并基准四个连续控制 RL 算法（TRPO、PPO、DDPG、Soft-Q）以研究超参数敏感性和跨任务迁移。结果表明超参数在很大程度上影响性能，良好的配置可作为基线进行泛化，而某些算法在某些任务上表现不佳。

ABSTRACT

Through many recent successes in simulation, model-free reinforcement learning has emerged as a promising approach to solving continuous control robotic tasks. The research community is now able to reproduce, analyze and build quickly on these results due to open source implementations of learning algorithms and simulated benchmark tasks. To carry forward these successes to real-world applications, it is crucial to withhold utilizing the unique advantages of simulations that do not transfer to the real world and experiment directly with physical robots. However, reinforcement learning research with physical robots faces substantial resistance due to the lack of benchmark tasks and supporting source code. In this work, we introduce several reinforcement learning tasks with multiple commercially available robots that present varying levels of learning difficulty, setup, and repeatability. On these tasks, we test the learning performance of off-the-shelf implementations of four reinforcement learning algorithms and analyze sensitivity to their hyper-parameters to determine their readiness for applications in various real-world tasks. Our results show that with a careful setup of the task interface and computations, some of these implementations can be readily applicable to physical robots. We find that state-of-the-art learning algorithms are highly sensitive to their hyper-parameters and their relative ordering does not transfer across tasks, indicating the necessity of re-tuning them for each task for best performance. On the other hand, the best hyper-parameter configuration from one task may often result in effective learning on held-out tasks even with different robots, providing a reasonable default. We make the benchmark tasks publicly available to enhance reproducibility in real-world reinforcement learning.

研究动机与目标

• 提出用于物理机器人、以实现可重复的现实世界 RL 研究的基准任务。
• 在多样化的现实世界机器人任务上评估多种开箱即用的 RL 算法。
• 分析学习性能的超参数敏感性以及跨任务的一致性。

提出的方法

• 使用三台商用机器人（UR5、Dynamixel MX-64AT、Create 2）定义六个 RL 任务。
• 实现环境和智能体分离进程的实时 RL 以减少延迟。
• 使用开源实现评估四种连续控制算法：TRPO、PPO、DDPG、Soft-Q 学习。
• 对 UR-Reacher-2 和 DXL-Reacher 任务进行随机超参数搜索以评估敏感性。
• 在未见任务上测试来自 UR-Reacher-2 的最佳配置以评估泛化。
• 分析重复性、初始化效应，并与脚本化基线进行比较。

实验结果

研究问题

• RQ1 在具有不同控制接口和传感模态的六个现实世界机器人任务上，最先进的 RL 算法如何表现？
• RQ2 RL 性能对不同任务中的超参数选择有多敏感？
• RQ3 超参数配置是否能作为对未见任务或机器人合理的默认值进行迁移？
• RQ4 在真实机器人上学习时，与仿真相比，实际面临哪些挑战和可重复性考虑？

主要发现

• 超参数选择对跨任务的策略质量有很大影响。
• TRPO 往往对超参数变化不太敏感，且保持了有竞争力的最终性能。
• Soft-Q 在若干 UR5 和 DXL 任务上学习速度最快，但在积极探索下可能会遇到过热问题。
• 在本研究中 DDPG 在 UR5 和 DXL 任务上表现较差。
• 某些超参数配置在保持任务和不同机器人间的未见任务上具有作为合理基线的泛化性。
• 在某些任务中，RL 解决方案有时被脚本化基线超越，但在如 Create-Docker 等缺乏明显脚本策略的任务中仍可具竞争力。