[Paper Review] Soft Policy Gradient Method for Maximum Entropy Deep Reinforcement Learning

Wenjie Shi, Shiji Song+1 more|arXiv (Cornell University)|Sep 7, 2019

Reinforcement Learning in Robotics24 references4 citations

TL;DR

This paper proposes Deep Soft Policy Gradient (DSPG), an off-policy, model-free maximum entropy deep reinforcement learning algorithm that combines soft policy gradient with a double-sampling-based soft Bellman equation to enable stable, high-performing learning in continuous control tasks. DSPG outperforms DDPG and SAC in sample efficiency, final performance, and training stability across benchmark environments like Ant, HalfCheetah, Hopper, and Walker2d.

ABSTRACT

Maximum entropy deep reinforcement learning (RL) methods have been demonstrated on a range of challenging continuous tasks. However, existing methods either suffer from severe instability when training on large off-policy data or cannot scale to tasks with very high state and action dimensionality such as 3D humanoid locomotion. Besides, the optimality of desired Boltzmann policy set for non-optimal soft value function is not persuasive enough. In this paper, we first derive soft policy gradient based on entropy regularized expected reward objective for RL with continuous actions. Then, we present an off-policy actor-critic, model-free maximum entropy deep RL algorithm called deep soft policy gradient (DSPG) by combining soft policy gradient with soft Bellman equation. To ensure stable learning while eliminating the need of two separate critics for soft value functions, we leverage double sampling approach to making the soft Bellman equation tractable. The experimental results demonstrate that our method outperforms in performance over off-policy prior methods.

Motivation & Objective

• Address the instability and poor sample efficiency of on-policy methods like DDPG in high-dimensional continuous control tasks.
• Overcome the limitations of off-policy value-based methods that require complex sampling procedures in continuous action spaces.
• Develop a stable, scalable, and sample-efficient deep RL algorithm under the maximum entropy framework to improve exploration and robustness.
• Eliminate the need for two separate critics in soft Q-learning by using a single critic with double sampling, reducing approximation error and improving training stability.
• Achieve superior performance and stability compared to prior off-policy methods such as DDPG and SAC in challenging continuous control benchmarks.

Proposed method

• Derive soft policy gradient from an entropy-regularized expected reward objective to optimize stochastic policies in continuous action spaces.
• Formulate an off-policy actor-critic framework using two deep neural networks: one for the stochastic policy (actor) and one for the soft Q-function (critic).
• Introduce a double sampling approach to make the soft Bellman equation tractable and stable, avoiding the need for two separate critics.
• Apply clipped soft policy gradient to ensure stable policy updates by constraining the magnitude of policy improvements.
• Use a replay buffer with 3 million transitions and train with mini-batches of 100 samples, updating the policy every 4 environment steps.
• Employ Adam optimizer with separate learning rates for actor and critic, and use soft target updates with a 0.01 update coefficient for stability.

Experimental results

Research questions

• RQ1 Can a soft policy gradient method be derived that enables stable, off-policy learning in high-dimensional continuous control tasks under the maximum entropy framework?
• RQ2 How does the use of double sampling in the soft Bellman equation improve stability and eliminate the need for two critics compared to prior soft Q-learning methods?
• RQ3 To what extent does DSPG improve sample efficiency and final performance compared to DDPG and SAC on standard continuous control benchmarks?
• RQ4 Does the combination of soft policy gradient and double-sampling-based soft Q-function learning lead to more robust and stable training dynamics?
• RQ5 Can DSPG generalize effectively across diverse tasks with varying state and action space dimensions, such as 3D humanoid locomotion?

Key findings

• DSPG significantly outperforms DDPG in both final performance and sample efficiency across all four benchmark environments: Ant-v2, HalfCheetah-v2, Hopper-v2, and Walker2d-v2.
• On Hopper-v2 and Walker2d-v2, DSPG achieves higher average total rewards than SAC, with 3674.029 and 6060.884 respectively, compared to SAC’s 3652.893 and 5520.419.
• In Ant-v2, DSPG achieves a best average total reward of 3384.074, outperforming DDPG (1012.242) and matching SAC’s best performance of 3472.346.
• The training curves show that DSPG exhibits superior stability compared to SAC, with less variance and consistent improvement across all tasks.
• The use of clipped soft policy gradient and a single critic with double sampling contributes to reduced approximation error and enhanced training stability.
• Despite using a single critic, DSPG maintains performance comparable to SAC, which relies on two critics, demonstrating the effectiveness of the double sampling approach.