[论文解读] Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraft

Ingmar Kanitscheider, Joost Huizinga+10 more|arXiv (Cornell University)|Jun 28, 2021

Reinforcement Learning in Robotics参考文献 29被引用 9

一句话总结

本文提出了一种基于学习进度的课程强化学习方法，用于在复杂、视觉化且探索困难的Minecraft环境中进行多任务训练。通过根据任务可学习性动态调整探索奖励，并利用学习进度指导课程调度，该方法成功学习了82项任务，显著优于均匀采样和静态奖励基线方法。

ABSTRACT

An important challenge in reinforcement learning is training agents that can solve a wide variety of tasks. If tasks depend on each other (e.g. needing to learn to walk before learning to run), curriculum learning can speed up learning by focusing on the next best task to learn. We explore curriculum learning in a complex, visual domain with many hard exploration challenges: Minecraft. We find that learning progress (defined as a change in success probability of a task) is a reliable measure of learnability for automatically constructing an effective curriculum. We introduce a learning-progress based curriculum and test it on a complex reinforcement learning problem (called "Simon Says") where an agent is instructed to obtain a desired goal item. Many of the required skills depend on each other. Experiments demonstrate that: (1) a within-episode exploration bonus for obtaining new items improves performance, (2) dynamically adjusting this bonus across training such that it only applies to items the agent cannot reliably obtain yet further increases performance, (3) the learning-progress based curriculum elegantly follows the learning curve of the agent, and (4) when the learning-progress based curriculum is combined with the dynamic exploration bonus it learns much more efficiently and obtains far higher performance than uniform baselines. These results suggest that combining intra-episode and across-training exploration bonuses with learning progress creates a promising method for automated curriculum generation, which may substantially increase our ability to train more capable, generally intelligent agents.

研究动机与目标

• 解决在稀疏奖励与高探索难度的复杂视觉环境中训练强化学习智能体的挑战。
• 开发一种能随智能体技能演变而自适应调整的自动化课程，以提升样本效率与性能表现。
• 克服多任务强化学习中均匀任务采样与静态探索奖励带来的局限性。
• 通过重新采样成功率下降的任务，减轻灾难性遗忘问题。
• 证明学习进度是现实强化学习场景中构建课程的可靠信号。

提出的方法

• 将学习进度定义为任务成功率随时间的变化，以此作为可学习性的代理指标。
• 实现一种基于单个智能体回合内的探索奖励，鼓励智能体在单次回合中获取新物品。
• 引入一种动态探索奖励，排除智能体已能稳定获取的物品，聚焦于更难获取的目标。
• 设计一种双向学习进度课程，根据性能提升与下降双重信号采样任务。
• 利用成功率估计值对可能被遗忘的任务进行重采样，从而减少灾难性遗忘。
• 将动态探索奖励与学习进度课程结合，用于指导任务采样与探索策略。

实验结果

研究问题

• RQ1 学习进度能否作为识别当前可学习任务的可靠、实时信号？
• RQ2 一种根据智能体当前技能水平自适应调整的动态探索奖励，如何影响多任务学习性能？
• RQ3 基于学习进度的课程是否能在高探索难度的多任务环境中超越均匀任务采样方法？
• RQ4 该课程在多任务强化学习中在多大程度上缓解了灾难性遗忘？
• RQ5 将动态探索与学习进度课程结合，是否能显著提升任务获取数量，优于基线方法？

主要发现

• 采用均匀采样且无探索奖励的方法仅完成了17项任务，且学习过程迅速停滞。
• 添加静态探索奖励后，通过鼓励回合内探索，任务完成数提升至43项。
• 通过动态调整探索奖励，排除已能稳定获取的物品，性能进一步提升至70项任务。
• 与均匀采样相比，采用双向学习进度课程将成功完成的任务数提高至82项。
• 学习进度课程自然地跟随了智能体的学习曲线，聚焦于技能获取的前沿区域。
• 双向课程通过重新采样成功率下降的任务，有效减轻了灾难性遗忘。