[论文解读] Mixed Policy Gradient.

Yang Guan, Jingliang Duan+4 more|arXiv (Cornell University)|Feb 23, 2021

Reinforcement Learning in Robotics参考文献 24被引用 4

一句话总结

本文提出混合策略梯度（MPG），一种新颖的强化学习方法，通过结合数据驱动与模型驱动的策略梯度，实现更快收敛且不损失渐近性能。通过随时间自适应加权两种策略梯度类型——逐步增加数据驱动策略梯度的权重——MPG在收敛速度和性能表现上均优于基线方法。

ABSTRACT

Reinforcement learning (RL) has great potential in sequential decision-making. At present, the mainstream RL algorithms are data-driven, relying on millions of iterations and a large number of empirical data to learn a policy. Although data-driven RL may have excellent asymptotic performance, it usually yields slow convergence speed. As a comparison, model-driven RL employs a differentiable transition model to improve convergence speed, in which the policy gradient (PG) is calculated by using the backpropagation through time (BPTT) technique. However, such methods suffer from numerical instability, model error sensitivity and low computing efficiency, which may lead to poor policies. In this paper, a mixed policy gradient (MPG) method is proposed, which uses both empirical data and the transition model to construct the PG, so as to accelerate the convergence speed without losing the optimality guarantee. MPG contains two types of PG: 1) data-driven PG, which is obtained by directly calculating the derivative of the learned Q-value function with respect to actions, and 2) model-driven PG, which is calculated using BPTT based on the model-predictive return. We unify them by revealing the correlation between the upper bound of the unified PG error and the predictive horizon, where the data-driven PG is regraded as 0-step model-predictive return. Relying on that, MPG employs a rule-based method to adaptively adjust the weights of data-driven and model-driven PGs. In particular, to get a more accurate PG, the weight of the data-driven PG is designed to grow along the learning process while the other to decrease. Besides, an asynchronous learning framework is proposed to reduce the wall-clock time needed for each update iteration. Simulation results show that the MPG method achieves the best asymptotic performance and convergence speed compared with other baseline algorithms.

研究动机与目标

• 解决数据驱动强化学习收敛速度慢的问题，其通常需要数百万次迭代才能收敛。
• 克服基于时间反向传播（BPTT）的模型驱动强化学习方法在数值稳定性、模型误差敏感性及低效性方面的缺陷。
• 将数据驱动与模型驱动策略梯度统一于单一框架中，在保持最优性保证的同时提升训练速度。
• 设计一种自适应加权机制，随训练过程逐步增加对数据驱动策略梯度的依赖，以提升梯度精度。
• 构建异步学习框架，以减少每次更新迭代的墙钟时间。

提出的方法

• 构建两类策略梯度：通过动作的Q值函数的直接导数计算数据驱动策略梯度，以及通过模型预测回报的时间反向传播（BPTT）计算模型驱动策略梯度。
• 通过建立统一梯度误差上界与预测时域之间的理论联系，实现两类策略梯度的统一。
• 将数据驱动策略梯度视为零步模型预测回报，以实现与模型驱动策略梯度的正式整合。
• 实施基于规则的自适应加权策略，训练过程中逐步提高数据驱动策略梯度的权重，同时降低模型驱动策略梯度的权重。
• 引入异步学习框架，以减少计算延迟并提升训练效率。

实验结果

研究问题

• RQ1 如何有效统一数据驱动与模型驱动策略梯度，以提升强化学习中的收敛速度？
• RQ2 统一策略梯度的误差界与预测时域之间存在何种理论关系？
• RQ3 一种在训练过程中动态演化的自适应加权方案，能否提升梯度精度与学习效率？
• RQ4 与现有强化学习算法相比，所提出的MPG方法在收敛速度与渐近性能方面表现如何？
• RQ5 异步学习框架在不降低策略质量的前提下，能在多大程度上减少墙钟时间？

主要发现

• 在仿真环境中，MPG在所有评估的基线算法中实现了最佳渐近性能。
• 所提方法相比纯数据驱动与纯模型驱动的强化学习方法，显著加速了收敛速度。
• 自适应加权机制成功提升了训练过程中数据驱动策略梯度的贡献，从而改善了梯度精度。
• 异步学习框架减少了每次更新迭代的墙钟时间，显著提升了训练效率。
• 理论分析证实，统一策略梯度的误差界随预测时域延长而减小，支持了该方法的设计合理性。