[论文解读] Proximal Reinforcement Learning: A New Theory of Sequential Decision Making in Primal-Dual Spaces

Sridhar Mahadevan, Bo Liu+6 more|arXiv (Cornell University)|May 26, 2014

Stochastic Gradient Optimization Techniques参考文献 113被引用 45

一句话总结

本文提出了一种新颖的近端强化学习框架，通过Legendre变换和近端算子，将时序差分学习与随机优化统一于对偶空间中。该框架实现了可证明收敛、稳定且安全的离策略学习，具有改进的收敛速率，包括GT D2-MP的加速$O(1/N)$速率，并为镜像下降、自然梯度和稀疏学习在强化学习中的应用提供了系统性理论基础。

ABSTRACT

In this paper, we set forth a new vision of reinforcement learning developed by us over the past few years, one that yields mathematically rigorous solutions to longstanding important questions that have remained unresolved: (i) how to design reliable, convergent, and robust reinforcement learning algorithms (ii) how to guarantee that reinforcement learning satisfies pre-specified "safety" guarantees, and remains in a stable region of the parameter space (iii) how to design "off-policy" temporal difference learning algorithms in a reliable and stable manner, and finally (iv) how to integrate the study of reinforcement learning into the rich theory of stochastic optimization. In this paper, we provide detailed answers to all these questions using the powerful framework of proximal operators. The key idea that emerges is the use of primal dual spaces connected through the use of a Legendre transform. This allows temporal difference updates to occur in dual spaces, allowing a variety of important technical advantages. The Legendre transform elegantly generalizes past algorithms for solving reinforcement learning problems, such as natural gradient methods, which we show relate closely to the previously unconnected framework of mirror descent methods. Equally importantly, proximal operator theory enables the systematic development of operator splitting methods that show how to safely and reliably decompose complex products of gradients that occur in recent variants of gradient-based temporal difference learning. This key technical innovation makes it possible to finally design "true" stochastic gradient methods for reinforcement learning. Finally, Legendre transforms enable a variety of other benefits, including modeling sparsity and domain geometry. Our work builds extensively on recent work on the convergence of saddle-point algorithms, and on the theory of monotone operators.

研究动机与目标

• 开发一种数学上严谨的强化学习理论，确保序列决策中收敛性、稳定性和安全性。
• 通过实现可靠、稳定且收敛的算法，解决离策略时序差分学习中的长期挑战。
• 在统一的近端算子框架下，将自然梯度方法与镜像下降方法统一起来。
• 通过算子分裂与近端更新，实现强化学习中真正的随机梯度方法。
• 将强化学习整合进更广泛的随机复合优化理论中，确保收敛性与稀疏性的保证。

提出的方法

• 利用Legendre变换在原空间与对偶空间之间映射，实现在对偶空间中的更新，从而提升稳定性和收敛性。
• 应用近端算子处理非光滑正则化项与复合目标函数，尤其在价值函数逼近中。
• 采用算子分裂策略——特别是前向-后向分裂与对偶-原分裂——将离策略TD学习中的复杂梯度乘积分解。
• 提出GTD2-MP算法，作为镜像-近端变体，通过外梯度风格更新实现加速收敛。
• 利用单调算子理论与鞍点公式化方法分析收敛性并推导最优收敛速率。
• 利用Bregman散度与镜像下降，实现稀疏学习与几何感知的价值函数逼近。

实验结果

研究问题

• RQ1 如何在离策略设置下设计可证明收敛且稳定的强化学习算法？
• RQ2 如何通过将参数保持在参数空间的稳定区域内，来保证安全性和稳定性？
• RQ3 如何系统性地推导出强化学习中价值函数学习的真正随机梯度方法？
• RQ4 如何在统一的理论框架下统一自然梯度与镜像下降方法？
• RQ5 如何在离策略时序差分学习中实现加速收敛速率？

主要发现

• GTD2-MP算法实现了加速收敛速率$O\big(\frac{L_{F^*} + L_K}{N} + \frac{\theta}{\nu}\big)$，优于标准GTD/GTD2的$O\big(\frac{L_{F^*} + L_K + \theta}{\nu}\big)$速率。
• GTD2-MP的价值逼近误差$||V - V_\theta||_\text{infty}$被有界为$\frac{L_\text{phi}^\Xi}{1 - \gamma} \cdot O\big(\frac{L_{F^*} + L_K}{N} + \frac{\sigma}{\sqrt{N}}\big)$，样本效率得到提升。
• 该框架通过Legendre变换建立了自然梯度下降与镜像下降之间的等价性，统一了强化学习中两种主要的优化范式。
• 近端算子实现了对复杂梯度乘积的系统性分解，使强化学习中真正可行的随机梯度方法成为可能。
• Bregman散度的使用实现了稀疏学习与领域几何建模，支持在高维空间中实现高效表示。
• 理论分析证实，在GTD/GTD2中增加原空间平均步骤，可将其转化为标准Polyak型算法，收敛速率为$O(1/\sqrt{N})$。