[论文解读] Constrained Reinforcement Learning Has Zero Duality Gap

Santiago Paternain, Luiz F. O. Chamon+2 more|arXiv (Cornell University)|Oct 29, 2019

Reinforcement Learning in Robotics参考文献 29被引用 10

一句话总结

本文证明了当策略由一类通用分布描述时，约束强化学习（CRL）问题的对偶间隙为零，这意味着对偶问题为凸优化问题，并能获得精确的最优解。作者进一步表明，该结果在参数化策略（如神经网络）下近似成立，并证明在温和假设下，原始-对偶算法可收敛至最优解。

ABSTRACT

Autonomous agents must often deal with conflicting requirements, such as completing tasks using the least amount of time/energy, learning multiple tasks, or dealing with multiple opponents. In the context of reinforcement learning~(RL), these problems are addressed by (i)~designing a reward function that simultaneously describes all requirements or (ii)~combining modular value functions that encode them individually. Though effective, these methods have critical downsides. Designing good reward functions that balance different objectives is challenging, especially as the number of objectives grows. Moreover, implicit interference between goals may lead to performance plateaus as they compete for resources, particularly when training on-policy. Similarly, selecting parameters to combine value functions is at least as hard as designing an all-encompassing reward, given that the effect of their values on the overall policy is not straightforward. The later is generally addressed by formulating the conflicting requirements as a constrained RL problem and solved using Primal-Dual methods. These algorithms are in general not guaranteed to converge to the optimal solution since the problem is not convex. This work provides theoretical support to these approaches by establishing that despite its non-convexity, this problem has zero duality gap, i.e., it can be solved exactly in the dual domain, where it becomes convex. Finally, we show this result basically holds if the policy is described by a good parametrization~(e.g., neural networks) and we connect this result with primal-dual algorithms present in the literature and we establish the convergence to the optimal solution.

研究动机与目标

• 在无需手动奖励设计或超参数调优的情况下，解决强化学习中冲突目标之间的平衡问题。
• 克服多目标强化学习的局限性，即奖励函数设计困难且目标在训练过程中相互干扰。
• 为广泛使用的约束强化学习原始-对偶算法提供理论依据，这些算法目前缺乏收敛性保证。
• 证明对于一般策略分布类，对偶间隙为零，从而可通过凸对偶优化获得精确解。
• 将零对偶间隙结果扩展至参数化策略（如神经网络），并证明在通用逼近能力下收敛性得以保持。

提出的方法

• 将约束强化学习形式化为具有多个奖励函数和硬约束的约束马尔可夫决策过程（CMDP）。
• 应用拉格朗日对偶理论，将非凸的原始问题转化为在对偶变量（拉格朗日乘子）上为凸的对偶问题。
• 证明对于一般分布类中的策略，对偶间隙为零，意味着原始问题与对偶问题的解完全等价。
• 通过证明子最优性界依赖于策略类的近似误差，将零对偶间隙结果扩展至参数化策略。
• 利用该结果分析原始-对偶算法，证明在温和假设下可收敛至最优解。
• 通过网格世界导航实验，对精确和近似原始求解下的对偶间隙行为进行经验验证。

实验结果

研究问题

• RQ1 当策略由一类通用分布描述时，约束强化学习是否具有零对偶间隙？
• RQ2 零对偶间隙结果能否扩展至参数化策略（如神经网络）？近似误差如何影响对偶间隙？
• RQ3 现有的约束强化学习原始-对偶算法是否收敛至最优解？在何种条件下收敛？
• RQ4 策略参数化方式的选择（如状态聚合）如何影响对偶间隙与学习性能？
• RQ5 在帕累托最优前沿方面，约束强化学习与正则化强化学习（手动选择乘子）是否存在等价性？

主要发现

• 对于一般分布类中的策略，约束强化学习问题的对偶间隙为零，意味着对偶问题能给出精确的最优解。
• 对于参数化策略（如神经网络），对偶间隙由策略类的近似误差所界定，且当策略类具备通用逼近能力时，该界趋于零。
• 由于对偶域中存在零对偶间隙，约束强化学习的原始-对偶算法在温和假设下可收敛至最优解。
• 实验结果表明，精确原始最大化（如通过Dijkstra算法）可使对偶间隙趋近于零；而近似原始步骤（如策略梯度）则导致对偶间隙收敛至零的邻域。
• 更粗糙的策略参数化方式（如状态聚合）会导致更大的对偶间隙，极端情况下（如过于粗糙）会导致智能体无法学习到可行策略。
• 理论框架建立了约束强化学习与正则化强化学习在帕累托最优前沿上的等价性，验证了对偶方法作为手动奖励加权的合理替代方案。