[论文解读] Distributional Soft Actor-Critic with Three Refinements

Jingliang Duan, Wenxuan Wang+7 more|arXiv (Cornell University)|Oct 9, 2023

Mosquito-borne diseases and control被引用 8

一句话总结

DSAC-T 通过引入期望值替代、双重价值分布以及基于方差的评论家梯度调整来改进分布式软策略评估器，实现稳定学习和在无需任务特定奖励缩放的情况下的出色表现。

ABSTRACT

Reinforcement learning (RL) has shown remarkable success in solving complex decision-making and control tasks. However, many model-free RL algorithms experience performance degradation due to inaccurate value estimation, particularly the overestimation of Q-values, which can lead to suboptimal policies. To address this issue, we previously proposed the Distributional Soft Actor-Critic (DSAC or DSACv1), an off-policy RL algorithm that enhances value estimation accuracy by learning a continuous Gaussian value distribution. Despite its effectiveness, DSACv1 faces challenges such as training instability and sensitivity to reward scaling, caused by high variance in critic gradients due to return randomness. In this paper, we introduce three key refinements to DSACv1 to overcome these limitations and further improve Q-value estimation accuracy: expected value substitution, twin value distribution learning, and variance-based critic gradient adjustment. The enhanced algorithm, termed DSAC with Three refinements (DSAC-T or DSACv2), is systematically evaluated across a diverse set of benchmark tasks. Without the need for task-specific hyperparameter tuning, DSAC-T consistently matches or outperforms leading model-free RL algorithms, including SAC, TD3, DDPG, TRPO, and PPO, in all tested environments. Additionally, DSAC-T ensures a stable learning process and maintains robust performance across varying reward scales. Its effectiveness is further demonstrated through real-world application in controlling a wheeled robot, highlighting its potential for deployment in practical robotic tasks.

研究动机与目标

• 解决连续控制任务中模型自由强化学习的过估计偏差。
• 在不需要任务特定奖励缩放的情况下改善分布式价值学习的稳定性。
• 通过双重价值分布和改进梯度更新来减少过估计。
• 提供一个鲁棒、可泛化的 DSAC 变体（DSAC-T），在各任务中表现稳定。

提出的方法

• 采用分布式软策略迭代来学习 Z(s,a) 的高斯价值分布。
• 在评论家更新中用稳定代理目标（期望值）替代随机目标（y_q 对比 y_z）。
• 引入双重价值分布并使用最小均值分布计算梯度（缓解过估计）。
• 用方差基界限 b = xi E[σ(s,a)] 取代固定裁剪边界，并用方差感知因子 ω 对梯度进行缩放。
• 引入裁剪和缩放参数的滑动平均更新以降低奖励尺度敏感性。
• 在策略更新中使用再参数化并调整温度 α 以维持目标熵。

实验结果

研究问题

• RQ1 DSAC-T 是否比 DSAC-v1 和其他基线更有效地降低过估计偏差？
• RQ2 这些改进是否提高学习稳定性并降低对奖励缩放的敏感性，在多样化任务中成立？
• RQ3 双重价值分布与期望值替代如何影响评论家和执行者的更新？
• RQ4 基于方差的梯度调整对对奖励量级的鲁棒性有何影响？

主要发现

• DSAC-T 在未进行任务特定超参数调优的情况下，超越或匹配了基线的模型自由强化学习算法（SAC、TD3、DDPG、TRPO、PPO）在测试环境中的表现。
• 与 DSAC-v1 相比，DSAC-T 实现了更好的学习稳定性和最终性能。
• 双重价值分布学习降低了过估计偏差，且倾向于略微低估，从而有助于稳定性。
• 方差基裁剪和梯度缩放降低了对奖励尺度的敏感性，减少了对任务特定奖励调优的需求。
• DSAC-T 在 OpenAI Gym 基准的向量和基于图像的任务上都显示出强劲表现。