[论文解读] Off-policy Maximum Entropy Reinforcement Learning : Soft Actor-Critic with Advantage Weighted Mixture Policy(SAC-AWMP)

Zhimin Hou, Kuangen Zhang+4 more|arXiv (Cornell University)|Feb 7, 2020

Reinforcement Learning in Robotics参考文献 23被引用 10

一句话总结

本文提出SAC-AWMP，一种基于最大熵的离策略强化学习算法，通过用优势加权混合策略（AWMP）替代Soft Actor-Critic（SAC）中的单一随机高斯策略，从而增强SAC。AWMP利用先验网络根据状态和优势函数表示动态分配多个策略组件的权重，实现对不连续、非光滑策略的更好近似。与SAC和TD3相比，该方法在四个MuJoCo连续控制任务中展现出更优的样本效率、稳定性和性能，尤其在Ant-v2等高难度环境中表现突出。

ABSTRACT

The optimal policy of a reinforcement learning problem is often discontinuous and non-smooth. I.e., for two states with similar representations, their optimal policies can be significantly different. In this case, representing the entire policy with a function approximator (FA) with shared parameters for all states maybe not desirable, as the generalization ability of parameters sharing makes representing discontinuous, non-smooth policies difficult. A common way to solve this problem, known as Mixture-of-Experts, is to represent the policy as the weighted sum of multiple components, where different components perform well on different parts of the state space. Following this idea and inspired by a recent work called advantage-weighted information maximization, we propose to learn for each state weights of these components, so that they entail the information of the state itself and also the preferred action learned so far for the state. The action preference is characterized via the advantage function. In this case, the weight of each component would only be large for certain groups of states whose representations are similar and preferred action representations are also similar. Therefore each component is easy to be represented. We call a policy parameterized in this way an Advantage Weighted Mixture Policy (AWMP) and apply this idea to improve soft-actor-critic (SAC), one of the most competitive continuous control algorithm. Experimental results demonstrate that SAC with AWMP clearly outperforms SAC in four commonly used continuous control tasks and achieve stable performance across different random seeds.

研究动机与目标

• 解决标准函数逼近器在高维连续控制中难以表示不连续、非光滑最优策略的局限性。
• 通过专家混合方法提升离策略最大熵强化学习中的策略泛化能力与表征能力。
• 开发一种基于状态和动作偏好信息自适应分配策略组件权重的方法，以增强策略灵活性与学习稳定性。
• 证明AWMP可无缝集成至SAC中，在无需超参数调优的情况下超越基线SAC和TD3。

提出的方法

• 策略参数化为多个随机高斯策略组件的混合，组件权重由学习得到的先验网络决定。
• 先验网络以当前状态为输入，输出策略组件的概率分布，并利用优势加权信息最大化方法指导权重分配。
• 通过最大化当前策略诱导的状态-动作对与采样策略组件之间的互信息来优化权重。
• 该方法使用带有平滑系数的目标网络以稳定学习，门控策略通过近期策略的半离策略数据进行更新。
• 该方法与离策略算法兼容，并自然集成SAC的最大熵目标与自动熵调整机制。
• 策略组件数量为超参数，经验分析表明在各类任务中四组件时性能最优。

实验结果

研究问题

• RQ1 具有动态加权组件的混合策略是否能提升连续控制中对不连续、非光滑最优策略的策略表征能力？
• RQ2 优势加权信息最大化如何增强离策略强化学习中策略组件选择的泛化性与稳定性？
• RQ3 SAC结合AWMP是否在多样化的MuJoCo环境中实现比SAC和TD3更优的样本效率与训练稳定性？
• RQ4 策略组件数量的变化对学习性能与收敛性有何影响？
• RQ5 目标网络中平滑系数的选择如何影响训练稳定性和学习速度？

主要发现

• SAC-AWMP在所有四个MuJoCo连续控制任务中均优于标准SAC和TD3，学习效率与稳定性显著提升，尤其在具有挑战性的Ant-v2环境中表现最为突出。
• 在Ant-v2环境中，SAC-AWMP获得显著更高的回报，证明其在更高效解决复杂、高维控制任务方面的能力。
• 算法在不同随机种子下表现出稳定的学习曲线且方差较低，表明对初始化和探索噪声具有强鲁棒性。
• 使用四个策略组件时性能最佳，八组件也表现出较低方差，表明其在容量增加时具备可扩展性与稳定性。
• 该方法无需超参数调优即可保持卓越性能，凸显其实际应用价值与部署便捷性。
• 敏感性分析表明，较小的平滑系数（τQ = 0.001）可确保稳定性，而较大的值（τQ = 0.1）则导致不稳定与发散。