[Paper Review] Meta-SAC: Auto-tune the Entropy Temperature of Soft Actor-Critic via Metagradient

Yufei Wang, Tianwei Ni|arXiv (Cornell University)|Jul 3, 2020

Reinforcement Learning in RoboticsComputer Science22 references18 citations

TL;DR

This paper proposes Meta-SAC, a meta-gradient-based method that automatically tunes the entropy temperature in Soft Actor-Critic (SAC) without introducing additional adaptive hyperparameters. By optimizing a novel meta objective aligned with the standard RL return, Meta-SAC achieves state-of-the-art performance on challenging Mujoco tasks, outperforming SAC-v2 by over 10% on the difficult humanoid-v2 environment.

ABSTRACT

Exploration-exploitation dilemma has long been a crucial issue in reinforcement learning. In this paper, we propose a new approach to automatically balance between these two. Our method is built upon the Soft Actor-Critic (SAC) algorithm, which uses an "entropy temperature" that balances the original task reward and the policy entropy, and hence controls the trade-off between exploitation and exploration. It is empirically shown that SAC is very sensitive to this hyperparameter, and the follow-up work (SAC-v2), which uses constrained optimization for automatic adjustment, has some limitations. The core of our method, namely Meta-SAC, is to use metagradient along with a novel meta objective to automatically tune the entropy temperature in SAC. We show that Meta-SAC achieves promising performances on several of the Mujoco benchmarking tasks, and outperforms SAC-v2 over 10% in one of the most challenging tasks, humanoid-v2.

Motivation & Objective

• Address the sensitivity of SAC to the entropy temperature hyperparameter, which critically balances exploration and exploitation.
• Overcome the limitations of SAC-v2, which requires tuning a target entropy hyperparameter despite its automatic adaptation of α.
• Develop a method that automatically adapts the entropy temperature α during training without introducing new adaptive hyperparameters.
• Design a meta-objective that aligns with the standard RL evaluation metric to improve learning efficiency and final performance.

Proposed method

• Use metagradient to optimize the entropy temperature α by backpropagating through the learning process to minimize a novel meta loss.
• Define the meta loss as the expected return of the policy after one policy update, making it consistent with the standard RL objective and evaluation metric.
• Derive the gradient of the meta loss w.r.t. α using the chain rule, enabling end-to-end optimization of α during SAC training.
• Avoid numerical instability by not differentiating through the Q-value network's output with respect to α, instead relying on policy gradient updates.
• Integrate the meta-gradient update into the SAC algorithm, updating α at each training step using the computed gradient.
• Ensure the method remains sample-efficient and stable by using a differentiable meta-objective that reflects actual performance on the environment.

Experimental results

Research questions

• RQ1 Can a meta-gradient approach automatically tune the entropy temperature α in SAC without introducing additional adaptive hyperparameters?
• RQ2 How does the evolution of α differ between Meta-SAC and SAC-v2 during training, and what does this imply for exploration-exploitation trade-offs?
• RQ3 Does the proposed meta-objective, aligned with the standard RL return, lead to better final performance compared to existing automatic α tuning methods?
• RQ4 How does Meta-SAC perform on challenging Mujoco environments, particularly those with high-dimensional state spaces like humanoid-v2?

Key findings

• Meta-SAC outperforms SAC-v2 by over 10% in final return on the challenging humanoid-v2 environment, demonstrating superior performance on complex tasks.
• On easier Mujoco tasks (Ant-v2, Hopper-v2, Walker2d-v2), Meta-SAC achieves performance comparable to SAC-v2 and slightly worse than SAC-v1 with grid-search-tuned α.
• The entropy temperature α in Meta-SAC decreases significantly over time, approaching zero in later training stages, indicating a shift from exploration to exploitation.
• In contrast, SAC-v2's α remains nearly constant after early training, suggesting limited dynamic adaptation to learning progress.
• The dynamic adaptation of α in Meta-SAC correlates with improved sample efficiency and faster convergence, especially in high-dimensional control tasks.
• Ablation studies confirm that the proposed meta-objective is more effective than alternative formulations, as it better aligns with the actual evaluation metric used in RL benchmarks.