[论文解读] Generating Automatic Curricula via Self-Supervised Active Domain Randomization

Sharath Chandra Raparthy, Bhairav Mehta+2 more|arXiv (Cornell University)|Feb 18, 2020

Reinforcement Learning in Robotics参考文献 33被引用 5

一句话总结

该论文提出自监督主动领域随机化（SS-ADR），通过自博弈生成自监督奖励信号，联合优化目标空间与环境空间的课程，实现课程的协同进化。通过同时优化更具挑战性的目标与随机化环境，SS-ADR 在多个机器人任务中实现了最先进且方差极低的零样本仿真到现实世界迁移性能。

ABSTRACT

Goal-directed Reinforcement Learning (RL) traditionally considers an agent interacting with an environment, prescribing a real-valued reward to an agent proportional to the completion of some goal. Goal-directed RL has seen large gains in sample efficiency, due to the ease of reusing or generating new experience by proposing goals. One approach,self-play, allows an agent to "play" against itself by alternatively setting and accomplishing goals, creating a learned curriculum through which an agent can learn to accomplish progressively more difficult goals. However, self-play has been limited to goal curriculum learning or learning progressively harder goals within a single environment. Recent work on robotic agents has shown that varying the environment during training, for example with domain randomization, leads to more robust transfer. As a result, we extend the self-play framework to jointly learn a goal and environment curriculum, leading to an approach that learns the most fruitful domain randomization strategy with self-play. Our method, Self-Supervised Active Domain Randomization(SS-ADR), generates a coupled goal-task curriculum, where agents learn through progressively more difficult tasks and environment variations. By encouraging the agent to try tasks that are just outside of its current capabilities, SS-ADR builds a domain randomization curriculum that enables state-of-the-art results on varioussim2real transfer tasks. Our results show that a curriculum of co-evolving the environment difficulty together with the difficulty of goals set in each environment provides practical benefits in the goal-directed tasks tested.

研究动机与目标

• 通过用自监督内在信号替代外部奖励，解决目标导向强化学习中的奖励工程与课程设计挑战。
• 通过联合优化目标与环境变化空间，克服传统课程学习与领域随机化的局限性。
• 通过学习适应智能体当前能力的课程，减少现实差距，实现机器人强化学习中的稳健仿真到现实世界迁移。
• 通过主动探索使课程自动校准，消除对随机化范围的人工调参。
• 构建双层优化框架，同时学习最优任务难度与环境多样性，以提升策略泛化能力。

提出的方法

• 将问题形式化为双层优化：内层循环针对给定的环境-目标对优化策略性能，外层循环通过自博弈指标最大化以引导课程进展。
• 采用两名智能体（Alice 与 Bob）之间的非对称自博弈，基于相对性能生成自监督奖励信号，激励探索逐步更难的任务。
• 将主动领域随机化（ADR）与自博弈结合，动态采样最具信息量的环境变化，避免低效或不稳定的配置。
• 利用单一来自自博弈结果的自监督奖励信号，同时引导目标与环境课程的演化，无需外部奖励塑造。
• 采用 SVPG（带价值策略梯度的软演员评论家）进行策略优化，实现在高维连续动作空间中的稳定训练。
• 通过允许算法在训练过程中避开物理上不稳定或不可解的环境（如随机化系数过低的情况），实现自校准，动态调整随机化范围。

实验结果

研究问题

• RQ1 来自自博弈的自监督奖励信号是否能有效引导目标导向强化学习中目标与环境空间的课程学习？
• RQ2 与静态或独立优化的课程相比，同时进化环境与目标难度是否能带来更优的策略鲁棒性与仿真到现实世界迁移性能？
• RQ3 该方法是否能自动校准以避免低效或不稳定的环境配置，而无需手动调参随机化范围？
• RQ4 在训练稳定性、收敛速度与零样本迁移性能方面，SS-ADR 与标准领域随机化及主动领域随机化相比表现如何？
• RQ5 联合课程学习框架在多类真实世界环境中在多大程度上降低了策略性能的方差？

主要发现

• SS-ADR 在所有测试的机器人任务中均实现了最先进水平的零样本仿真到现实世界迁移性能，在仿真与真实世界环境中均优于所有基线方法。
• 与标准领域随机化及主动领域随机化相比，该方法显著降低了训练方差，展现出更稳定的策略学习能力。
• 在未校准的随机化范围内，SS-ADR 能够成功避免采样物理上不稳定的环境（如随机化系数低于 0.05 的情况），而 UDR（均匀领域随机化）则无法避免这些区域。
• 在校准范围内，SS-ADR 学习到的课程比 UDR 更具有效性，表现为在更难任务区域的采样更集中，表明其课程进展更优。
• 来自自博弈的自监督奖励有效实现了无需外部奖励塑造的课程演化，证明了在复杂任务空间中内在、自驱动学习的可行性。
• 双层优化框架成功协同进化了目标与环境课程，生成的策略在多种真实世界设置中具有优异泛化能力，且仅需极少微调。