[Paper Review] Soft Actor-Critic for Discrete Action Settings

Petros Christodoulou|arXiv (Cornell University)|Oct 16, 2019

Reinforcement Learning in RoboticsComputer Science12 references209 citations

TL;DR

This paper derives SAC for discrete action spaces (SAC-Discrete) and shows it is competitively sample-efficient with state-of-the-art Rainbow on Atari games without hyperparameter tuning.

ABSTRACT

Soft Actor-Critic is a state-of-the-art reinforcement learning algorithm for continuous action settings that is not applicable to discrete action settings. Many important settings involve discrete actions, however, and so here we derive an alternative version of the Soft Actor-Critic algorithm that is applicable to discrete action settings. We then show that, even without any hyperparameter tuning, it is competitive with the tuned model-free state-of-the-art on a selection of games from the Atari suite.

Motivation & Objective

• Motivate: SAC excels in continuous-action RL but lacks discrete-action applicability.
• Derive a discrete-action SAC variant by adjusting value, policy, and temperature updates.
• Demonstrate SAC-Discrete efficiency on Atari games and compare to Rainbow under limited tuning.
• Provide open-source implementation of SAC-Discrete.

Proposed method

• Adapt soft Q-function to discrete actions by outputting Q-values for all actions: Q:S -> R^{|A|}.
• Replace policy output with direct action distribution over A using softmax: pi:S -> [0,1]^{|A|}.
• Compute V(s) and temperature loss with direct expectations: V(s)=pi(s)^{T}[Q(s)-alpha log pi(s)], alpha-loss J(alpha)=pi(s)^{T}[-alpha(log pi(s)+H)].
• Remove reparameterisation trick since actions are discrete and expectations are tractable.
• Use two soft Q-networks and take their minimum to mitigate overestimation.
• Provide Algorithm 1 (SAC-Discrete) detailing updates for Q-functions, policy, and temperature.
• Report hyperparameters and experimental setup for Atari (no tuning beyond values from prior work).

Experimental results

Research questions

• RQ1 Can SAC be effectively adapted to discrete action spaces without sacrificing sample efficiency?
• RQ2 How does SAC-Discrete perform on Atari compared with a tuned, strong baseline (Rainbow) in terms of sample efficiency?
• RQ3 What architectural and algorithmic changes are needed to maintain low-variance, stable learning in discrete action SAC?
• RQ4 Does SAC-Discrete require hyperparameter tuning to outperform or match existing discrete-action algorithms?

Key findings

• SAC-Discrete achieves competitive sample efficiency relative to Rainbow on 20 Atari games with five seeds.
• Across 20 games, SAC-Discrete wins in 10, with a median performance difference of -1% and a range up to +4330% and down to -99%.
• SAC-Discrete does not rely on hyperparameter tuning to reach competitive results.
• The paper provides a public Python implementation (GitHub).