[Paper Review] Soft Actor-Critic Algorithms and Applications

Tuomas Haarnoja, Aurick Zhou+9 more|arXiv (Cornell University)|Dec 13, 2018

Reinforcement Learning in Robotics44 references1,935 citations

TL;DR

The paper introduces Soft Actor-Critic (SAC), an off-policy actor-critic algorithm based on maximum entropy RL with automatic temperature tuning, achieving strong sample efficiency and stability in continuous control tasks and real-world robotics.

ABSTRACT

Model-free deep reinforcement learning (RL) algorithms have been successfully applied to a range of challenging sequential decision making and control tasks. However, these methods typically suffer from two major challenges: high sample complexity and brittleness to hyperparameters. Both of these challenges limit the applicability of such methods to real-world domains. In this paper, we describe Soft Actor-Critic (SAC), our recently introduced off-policy actor-critic algorithm based on the maximum entropy RL framework. In this framework, the actor aims to simultaneously maximize expected return and entropy. That is, to succeed at the task while acting as randomly as possible. We extend SAC to incorporate a number of modifications that accelerate training and improve stability with respect to the hyperparameters, including a constrained formulation that automatically tunes the temperature hyperparameter. We systematically evaluate SAC on a range of benchmark tasks, as well as real-world challenging tasks such as locomotion for a quadrupedal robot and robotic manipulation with a dexterous hand. With these improvements, SAC achieves state-of-the-art performance, outperforming prior on-policy and off-policy methods in sample-efficiency and asymptotic performance. Furthermore, we demonstrate that, in contrast to other off-policy algorithms, our approach is very stable, achieving similar performance across different random seeds. These results suggest that SAC is a promising candidate for learning in real-world robotics tasks.

Motivation & Objective

• Motivate overcoming high sample complexity and hyperparameter brittleness in model-free deep RL for real-world tasks.
• Propose an off-policy actor-critic framework that maximizes return and policy entropy simultaneously.
• Introduce automatic entropy tuning to reduce per-task hyperparameter tuning.
• Empirically validate SAC on benchmark control tasks and real-world robotic manipulation and locomotion problems.

Proposed method

• Formulate SAC as an off-policy actor-critic algorithm with a stochastic policy and a soft Q-function.
• Optimize two soft Q-functions to reduce positive bias and use the minimum for updates.
• Use a reparameterization trick to backpropagate through the stochastic policy.
• Adopt an entropy-regularized objective with a learnable temperature parameter alpha via dual gradient updates.
• Employ a replay pool for off-policy data and target networks for stability.
• Provide an automatic entropy adjustment mechanism that constrains the policy entropy to match a target via a dual objective.

Experimental results

Research questions

• RQ1 Can SAC improve sample efficiency and final performance compared to existing on-policy and off-policy methods on continuous control tasks?
• RQ2 Does incorporating maximum entropy with automatic temperature tuning yield more stable training across tasks and random seeds?
• RQ3 How does SAC perform on challenging real-world robotic tasks using image observations or high-dimensional sensors?

Key findings

• SAC achieves state-of-the-art performance in terms of sample efficiency and asymptotic performance compared to prior off-policy and on-policy methods.
• The algorithm demonstrates strong stability, with similar performance across different random seeds.
• Two soft Q-functions and the automatic entropy tuning mechanism contribute to improved training stability and data efficiency.
• SAC reliably handles challenging real-world tasks such as quadrupedal locomotion and dexterous robotic manipulation from image observations.