[论文解读] The Effect of Multi-step Methods on Overestimation in Deep Reinforcement Learning

Lingheng Meng, Rob Gorbet+1 more|arXiv (Cornell University)|Jun 23, 2020

Reinforcement Learning in Robotics参考文献 28被引用 8

一句话总结

本文提出多步DDPG（MDDPG）和混合多步DDPG（MMDDPG），通过使用多步回溯来减少深度强化学习中的过估计问题。通过利用经验回放缓冲区中的多步回报，两种方法相比标准DDPG显著减少了过估计，从而在计算开销极低的情况下实现更快的学习速度和更优的最终性能。

ABSTRACT

Multi-step (also called n-step) methods in reinforcement learning (RL) have been shown to be more efficient than the 1-step method due to faster propagation of the reward signal, both theoretically and empirically, in tasks exploiting tabular representation of the value-function. Recently, research in Deep Reinforcement Learning (DRL) also shows that multi-step methods improve learning speed and final performance in applications where the value-function and policy are represented with deep neural networks. However, there is a lack of understanding about what is actually contributing to the boost of performance. In this work, we analyze the effect of multi-step methods on alleviating the overestimation problem in DRL, where multi-step experiences are sampled from a replay buffer. Specifically building on top of Deep Deterministic Policy Gradient (DDPG), we propose Multi-step DDPG (MDDPG), where different step sizes are manually set, and its variant called Mixed Multi-step DDPG (MMDDPG) where an average over different multi-step backups is used as update target of Q-value function. Empirically, we show that both MDDPG and MMDDPG are significantly less affected by the overestimation problem than DDPG with 1-step backup, which consequently results in better final performance and learning speed. We also discuss the advantages and disadvantages of different ways to do multi-step expansion in order to reduce approximation error, and expose the tradeoff between overestimation and underestimation that underlies offline multi-step methods. Finally, we compare the computational resource needs of Twin Delayed Deep Deterministic Policy Gradient (TD3), a state-of-art algorithm proposed to address overestimation in actor-critic methods, and our proposed methods, since they show comparable final performance and learning speed.

研究动机与目标

• 探究多步方法如何影响深度强化学习（DRL）中的过估计问题，特别是在DDPG等演员-评论家框架中。
• 解决DRL中价值函数过估计的挑战，该问题会阻碍连续控制任务中稳定且高效的学习。
• 设计并评估DDPG的多步扩展方法，以减少过估计，同时保持或提升学习速度和最终性能。
• 比较多步方法与SOTA算法（如TD3）在计算效率与近似误差之间的权衡。
• 分析离线多步回溯中过估计与欠估计之间的权衡及其对学习稳定性的影响。

提出的方法

• 提出多步DDPG（MDDPG），在Q值目标计算中为不同固定的步长（n）手动设定多步回溯。
• 引入混合多步DDPG（MMDDPG），通过计算多个步长（n）的Q值目标加权平均来提高目标准确性。
• 使用经验回放缓冲区存储并采样多步转移，实现在无需在线滚动的情况下进行离线多步回溯。
• 采用n步的自举Q值估计来减少单个不准确值估计的影响，从而缓解过估计问题。
• 在学习速度、最终性能和计算成本（前向/反向传播次数）方面，将MDDPG和MMDDPG与DDPG及TD3进行比较。
• 分析自举n步回报中的过估计与基于离线回放缓冲区回溯中的欠估计之间的权衡，表明n可平衡这一权衡。

实验结果

研究问题

• RQ1 在DDPG中使用多步回溯如何影响Q值函数近似中的过估计程度？
• RQ2 多步方法是否能在不增加计算成本的情况下提升DRL中的学习速度和最终性能？
• RQ3 自举n步回报中的过估计与基于离线回放缓冲区回溯中的欠估计之间存在何种权衡？
• RQ4 MDDPG和MMDDPG的性能与专为减少过估计而设计的SOTA算法TD3相比如何？
• RQ5 在多步回溯中，选择步长n的最优策略是什么，以平衡过估计与欠估计？

主要发现

• 与标准DDPG相比，MDDPG和MMDDPG显著减少了过估计，从而在连续控制任务中实现了更稳定、更有效的学习。
• MMDDPG在最终性能和学习速度方面与TD3（一种专为缓解过估计而设计的SOTA算法）相当。
• MDDPG的计算资源消耗低于TD3，每个小批量仅需一次前向和一次反向传播，与DDPG类似。
• 当步长n ≥ 4时，MMDDPG的计算成本高于TD3，因为其目标Q值计算需要n次前向传播。
• 离线多步回溯在初期倾向于低估真实Q值，但随着回放缓冲区逐渐填满，其值会逐渐趋近于在线多步值。
• 步长n的选择涉及过估计（来自自举n步回报）与欠估计（来自离线回放缓冲区）之间的权衡，该权衡需在训练过程中加以平衡。