[Paper Review] Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

Tuomas Haarnoja, Aurick Zhou+2 more|arXiv (Cornell University)|Jan 4, 2018

Reinforcement Learning in Robotics34 references3,481 citations

TL;DR

Soft Actor-Critic (SAC) is an off-policy, maximum entropy actor-critic method with a stochastic policy that achieves state-of-the-art performance and stability on continuous control tasks, improving sample efficiency over prior methods.

ABSTRACT

Model-free deep reinforcement learning (RL) algorithms have been demonstrated on a range of challenging decision making and control tasks. However, these methods typically suffer from two major challenges: very high sample complexity and brittle convergence properties, which necessitate meticulous hyperparameter tuning. Both of these challenges severely limit the applicability of such methods to complex, real-world domains. In this paper, we propose soft actor-critic, an off-policy actor-critic deep RL algorithm based on the maximum entropy reinforcement learning framework. In this framework, the actor aims to maximize expected reward while also maximizing entropy. That is, to succeed at the task while acting as randomly as possible. Prior deep RL methods based on this framework have been formulated as Q-learning methods. By combining off-policy updates with a stable stochastic actor-critic formulation, our method achieves state-of-the-art performance on a range of continuous control benchmark tasks, outperforming prior on-policy and off-policy methods. Furthermore, we demonstrate that, in contrast to other off-policy algorithms, our approach is very stable, achieving very similar performance across different random seeds.

Motivation & Objective

• Motivate and address the high sample complexity and hyperparameter sensitivity of model-free deep RL.
• Develop an off-policy maximum entropy actor-critic algorithm with a stochastic policy.
• Demonstrate stability and strong performance across challenging continuous control benchmarks.
• Provide theoretical convergence results for soft policy iteration and practical SAC instantiation.
• Compare SAC to state-of-the-art off-policy and on-policy baselines and analyze key hyperparameters.

Proposed method

• Formulate a maximum entropy reinforcement learning objective that includes an entropy term with a temperature parameter.
• Derive soft policy iteration and prove convergence to the optimal maximum entropy policy within a policy class.
• Introduce SAC with parametric networks for V, Q, and policy, and use two Q-functions to reduce positive bias.
• Optimize V, Q, and policy via off-policy stochastic gradient updates using a replay buffer.
• Use a reparameterization trick to obtain low-variance policy gradients.
• Evaluate SAC on continuous control benchmarks and compare to DDPG, PPO, and SQL.

Experimental results

Research questions

• RQ1 Can an off-policy maximum entropy framework yield stable and sample-efficient learning for continuous control?
• RQ2 Does incorporating stochastic policies and entropy maximization improve exploration and robustness compared to prior off-policy methods?
• RQ3 How does SAC perform relative to DDPG, PPO, and other baselines across challenging tasks (e.g., Humanoid)?
• RQ4 What are the key factors (reward scaling, target update smoothing) that influence SAC performance and stability?

Key findings

• SAC achieves superior performance and sample efficiency on challenging continuous control tasks compared with both off-policy and on-policy baselines.
• Using two Q-functions mitigates positive bias and improves training speed, especially on harder tasks.
• Stochastic policies with entropy maximization yield more stable training and better seed-to-seed consistency than deterministic variants.
• Reward scaling acts as a temperature control for the entropy term and greatly influences learning dynamics.
• Target network smoothing constant tau affects stability and speed of training, with a relatively wide viable range.
• Evaluation using the policy mean can yield better performance, although SAC optimizes a stochastic policy.