[Paper Review] Distributional Soft Actor-Critic with Three Refinements

Jingliang Duan, Wenxuan Wang+7 more|arXiv (Cornell University)|Oct 9, 2023

Mosquito-borne diseases and control8 citations

TL;DR

DSAC-T improves distributional soft actor-critic by introducing expected value substituting, twin value distributions, and variance-based critic gradient adjusting, achieving stable learning and strong performance without task-specific reward scaling.

ABSTRACT

Reinforcement learning (RL) has shown remarkable success in solving complex decision-making and control tasks. However, many model-free RL algorithms experience performance degradation due to inaccurate value estimation, particularly the overestimation of Q-values, which can lead to suboptimal policies. To address this issue, we previously proposed the Distributional Soft Actor-Critic (DSAC or DSACv1), an off-policy RL algorithm that enhances value estimation accuracy by learning a continuous Gaussian value distribution. Despite its effectiveness, DSACv1 faces challenges such as training instability and sensitivity to reward scaling, caused by high variance in critic gradients due to return randomness. In this paper, we introduce three key refinements to DSACv1 to overcome these limitations and further improve Q-value estimation accuracy: expected value substitution, twin value distribution learning, and variance-based critic gradient adjustment. The enhanced algorithm, termed DSAC with Three refinements (DSAC-T or DSACv2), is systematically evaluated across a diverse set of benchmark tasks. Without the need for task-specific hyperparameter tuning, DSAC-T consistently matches or outperforms leading model-free RL algorithms, including SAC, TD3, DDPG, TRPO, and PPO, in all tested environments. Additionally, DSAC-T ensures a stable learning process and maintains robust performance across varying reward scales. Its effectiveness is further demonstrated through real-world application in controlling a wheeled robot, highlighting its potential for deployment in practical robotic tasks.

Motivation & Objective

• Address overestimation bias in model-free RL for continuous control tasks.
• Improve stability of distributional value learning without requiring task-specific reward scaling.
• Reduce overestimation through twin value distributions and refined gradient updates.
• Provide a robust, generalizable DSAC variant (DSAC-T) with stable performance across tasks.

Proposed method

• Adopt distributional soft policy iteration to learn a Gaussian value distribution for Z(s,a).
• Replace the random target with a stable surrogate target (expected value) in critic updates (y_q vs y_z).
• Introduce twin value distributions and use the minimum-mean distribution to compute gradients (mitigates overestimation).
• Replace fixed clipping boundary with a variance-based boundary b = xi E[sigma(s,a)], and scale gradients by a variance-aware factor omega.
• Incorporate a moving-average update of clipping and scaling parameters to reduce reward-scale sensitivity.
• Utilize reparameterization for policy updates and adapt temperature alpha to maintain target entropy.

Experimental results

Research questions

• RQ1 Can DSAC-T reduce overestimation bias more effectively than DSAC-v1 and other baselines?
• RQ2 Do the refinements lead to improved learning stability and reduced sensitivity to reward scaling across diverse tasks?
• RQ3 How do twin value distributions and expected value substitution affect critic and actor updates?
• RQ4 What is the impact of variance-based gradient adjustment on robustness to reward magnitudes?

Key findings

• DSAC-T surpasses or matches baseline model-free RL algorithms (SAC, TD3, DDPG, TRPO, PPO) across tested environments without task-specific hyperparameter tuning.
• Compared to DSAC-v1, DSAC-T achieves better learning stability and final performance.
• Twin value distribution learning reduces overestimation bias and tends to slight underestimation, contributing to stability.
• Variance-based clipping and gradient scaling mitigate sensitivity to reward scales, reducing need for task-specific reward tuning.
• DSAC-T demonstrates strong performance on both vector- and image-based tasks in OpenAI Gym benchmarks.