[论文解读] Automatic Curriculum Learning through Value Disagreement

Yunzhi Zhang, Pieter Abbeel+1 more|arXiv (Cornell University)|Jun 17, 2020

Reinforcement Learning in Robotics参考文献 40被引用 34

一句话总结

论文提出 Value Disagreement Sampling (VDS)，通过使用价值函数集合在学习前沿采样目标，自动策划面向目标的强化学习的目标课程。

ABSTRACT

Continually solving new, unsolved tasks is the key to learning diverse behaviors. Through reinforcement learning (RL), we have made massive strides towards solving tasks that have a single goal. However, in the multi-task domain, where an agent needs to reach multiple goals, the choice of training goals can largely affect sample efficiency. When biological agents learn, there is often an organized and meaningful order to which learning happens. Inspired by this, we propose setting up an automatic curriculum for goals that the agent needs to solve. Our key insight is that if we can sample goals at the frontier of the set of goals that an agent is able to reach, it will provide a significantly stronger learning signal compared to randomly sampled goals. To operationalize this idea, we introduce a goal proposal module that prioritizes goals that maximize the epistemic uncertainty of the Q-function of the policy. This simple technique samples goals that are neither too hard nor too easy for the agent to solve, hence enabling continual improvement. We evaluate our method across 13 multi-goal robotic tasks and 5 navigation tasks, and demonstrate performance gains over current state-of-the-art methods.

研究动机与目标

• 激励多目标 RL 的自动课程学习以提高样本效率。
• 利用前沿目标采样提供有信息量的训练信号。
• 开发一个目标提议模块，利用来自价值函数集合的认知不确定性。
• 在多样的机器人和导航任务上展示该方法的有效性。

提出的方法

• 定义一个目标提议模块，从策略相关分布 C^π 采样目标。
• 使用 K 个 Q 函数的集合来估计目标条件 Q 函数的认知不确定性。
• 从集合不确定性计算采样分布并相应地采样目标。
• 使用采样的目标收集轨迹，并通过标准 RL 更新同时更新策略和 Q 函数。
• 与 Hindsight Experience Replay (HER) 集成以处理稀疏奖励；以 DDPG 作为基础 RL 算法进行评估。
• 算法 1 总结了通过 Value Disagreement Sampling (VDS) 进行课程生成。

实验结果

研究问题

• RQ1 Value Disagreement Sampling (VDS) 相比基线的目标条件 RL 方法是否提升了样本效率？
• RQ2 VDS 采样的目标是否对应具有信息量的前沿式挑战，提升学习？
• RQ3 VDS 对采样函数、集合大小以及与 HER 的结合等设计选项有多鲁棒？

主要发现

• VDS 在包括操作和导航领域的 18 个稀疏奖励任务上提升了样本效率。
• VDS 倾向于在学习前沿采样目标，随着策略熟练度提高向更难的目标前进。
• VDS 在大多数评估环境中优于基线如 HER、GoalGAN 以及其他课程。
• VDS 在各种采样函数和集合大小下仍然有效，并且与 HER 配对受益。
• 将 VDS 与 HER 结合在报道的实验中获得最佳性能。