[论文解读] A Novel Automated Curriculum Strategy to Solve Hard Sokoban Planning Instances

Dieqiao Feng, Carla P. Gomes+1 more|arXiv (Cornell University)|Oct 3, 2021

Software Engineering Research被引用 6

一句话总结

本文提出了一种新颖的深度强化学习（RL）自动课程策略，采用难度量子动量多臂赌博机方法，动态选择日益复杂的无标签俄罗斯方块规划实例。通过聚焦于可解性边缘的子任务，该方法使RL智能体成功解决了225个困难的Sokoban实例中的179个（80%），显著优于此前最先进方法仅解决的31个（13%）。

ABSTRACT

In recent years, we have witnessed tremendous progress in deep reinforcement learning (RL) for tasks such as Go, Chess, video games, and robot control. Nevertheless, other combinatorial domains, such as AI planning, still pose considerable challenges for RL approaches. The key difficulty in those domains is that a positive reward signal becomes {\em exponentially rare} as the minimal solution length increases. So, an RL approach loses its training signal. There has been promising recent progress by using a curriculum-driven learning approach that is designed to solve a single hard instance. We present a novel {\em automated} curriculum approach that dynamically selects from a pool of unlabeled training instances of varying task complexity guided by our {\em difficulty quantum momentum} strategy. We show how the smoothness of the task hardness impacts the final learning results. In particular, as the size of the instance pool increases, the ``hardness gap'' decreases, which facilitates a smoother automated curriculum based learning process. Our automated curriculum approach dramatically improves upon the previous approaches. We show our results on Sokoban, which is a traditional PSPACE-complete planning problem and presents a great challenge even for specialized solvers. Our RL agent can solve hard instances that are far out of reach for any previous state-of-the-art Sokoban solver. In particular, our approach can uncover plans that require hundreds of steps, while the best previous search methods would take many years of computing time to solve such instances. In addition, we show that we can further boost the RL performance with an intricate coupling of our automated curriculum approach with a curiosity-driven search strategy and a graph neural net representation.

研究动机与目标

• 解决组合AI规划任务中极端稀疏奖励的问题，其中成功计划呈指数级稀少。
• 通过实现完全自动化、动态的训练实例选择，克服人工课程设计的局限性。
• 使深度强化学习智能体在无需标签解或大量领域特定启发式规则的情况下，解决Sokoban等PSPACE完全的困难规划问题。
• 通过聚焦于可解性边界附近的子任务，平衡探索与进展，提升样本效率和学习速度。
• 证明将自动课程与好奇心驱动搜索及图神经网络表示相结合的有效性。

提出的方法

• 使用一个包含各种复杂度的无标签Sokoban子任务池，包括不可解实例。
• 实施难度量子动量多臂赌博机策略，根据当前策略性能和可解性估计选择子任务批次。
• 将选定的子任务输入具有图神经网络（GNN）策略的深度强化学习智能体，以学习状态表示和动作策略。
• 使用蒙特卡洛树搜索（MCTS）评估可解性，并引导赌博机选择那些当前策略刚好可触及的子任务。
• 随着策略能力的提升，动态调整课程，逐步增加所选实例的难度，保持平滑的进展过程。
• 整合好奇心驱动探索，通过鼓励智能体探索新颖或具有挑战性的子任务，进一步提升性能。

实验结果

研究问题

• RQ1 基于动态子任务选择的自动课程策略，能否显著提升深度强化学习在困难组合规划问题上的性能？
• RQ2 任务难度进展的平滑性在稀疏奖励环境中如何影响学习效率和最终性能？
• RQ3 无监督、自监督课程学习在AI规划中，能在多大程度上超越人工设计或固定课程调度？
• RQ4 在不同难度层级上混合子任务在维持稳定且快速的学习轨迹方面发挥什么作用？
• RQ5 将自动课程与好奇心驱动探索及GNN表示相结合，能否进一步提升样本效率并解决此前难以处理的实例？

主要发现

• 所提出的自动课程策略成功解决了225个困难Sokoban实例中的179个（80%），而此前最先进基线方法仅解决了31个（13%）。
• 难度量子动量赌博机策略成功将训练聚焦于当前策略能力边界之外的子任务，形成了平滑且高效的学学习进展。
• 随着训练的推进，采样更大、更复杂子任务（如14-和15-box）的概率逐渐增加，表明策略有效适应了智能体能力的增长。
• 训练池中包含未解决的子任务显著加快了学习并提升了鲁棒性，表明接触具有挑战性但不可解的实例有助于泛化。
• 图神经网络与好奇心驱动探索的整合进一步提升了性能，使智能体能够发现需要数百步的复杂长路径。
• 该方法在无需任何标签解或手工设计启发式规则的情况下实现上述成果，证明了自监督、课程驱动的深度强化学习在困难规划领域中的可行性。