[Paper Review] A Self-Tuning Actor-Critic Algorithm

Tom Zahavy, Zhongwen Xu+6 more|arXiv (Cornell University)|Feb 28, 2020

Reinforcement Learning in RoboticsComputer Science29 references32 citations

TL;DR

STAC and STACX use metagradients to self-tune differentiable hyperparameters of an actor-critic loss, plus a Leaky V-trace variant and auxiliary tasks, yielding consistent performance gains across ALE and DM Control without substantial compute overhead.

ABSTRACT

Reinforcement learning algorithms are highly sensitive to the choice of hyperparameters, typically requiring significant manual effort to identify hyperparameters that perform well on a new domain. In this paper, we take a step towards addressing this issue by using metagradients to automatically adapt hyperparameters online by meta-gradient descent (Xu et al., 2018). We apply our algorithm, Self-Tuning Actor-Critic (STAC), to self-tune all the differentiable hyperparameters of an actor-critic loss function, to discover auxiliary tasks, and to improve off-policy learning using a novel leaky V-trace operator. STAC is simple to use, sample efficient and does not require a significant increase in compute. Ablative studies show that the overall performance of STAC improved as we adapt more hyperparameters. When applied to the Arcade Learning Environment (Bellemare et al. 2012), STAC improved the median human normalized score in 200M steps from 243% to 364%. When applied to the DM Control suite (Tassa et al., 2018), STAC improved the mean score in 30M steps from 217 to 389 when learning with features, from 108 to 202 when learning from pixels, and from 195 to 295 in the Real-World Reinforcement Learning Challenge (Dulac-Arnold et al., 2020).

Motivation & Objective

• Motivate reducing manual hyperparameter tuning in deep RL by enabling online self-tuning through meta-gradients.
• Develop STAC to automatically optimize all differentiable hyperparameters in the IMPALA loss and introduce Leaky V-trace.
• Extend STAC with auxiliary tasks (STACX) to discover beneficial auxiliary losses while self-tuning their metaparameters.
• Demonstrate empirical performance gains across diverse domains (ALE and DM Control) with ablations and robustness analyses.

Proposed method

• Formulate inner and outer losses where the inner loss is parameterized by metaparameters atter = {gamma, lambda, g_v, g_p, g_e} and the outer loss includes a KL regularizer to prevent policy drift.
• Apply metagradient updates to self-tune differentiable hyperparameters online, using a differentiable meta-optimizer (Adam) on the outer loss.
• Introduce Leaky V-trace, a differentiable interpolation between importance sampling and truncated IS, controlled by a leak parameter alpha.
• For STACX, add auxiliary heads with their own metaparameters to learn auxiliary tasks that improve the shared representation, with outer loss focusing on the main head.
• Use a shared representation backbone (ResNet-like) with multiple heads; each auxiliary head optimizes its own differentiable loss via Leaky V-trace for off-policy corrections.

Experimental results

Research questions

• RQ1 Can metagradients be used to self-tune a large set of differentiable hyperparameters in an online, single-lifetime RL setting?
• RQ2 Does self-tuning of hyperparameters improve sample efficiency and final performance across diverse domains (ALE and DM Control)?
• RQ3 What is the impact of Leaky V-trace on stability and performance in off-policy actor-critic learning?
• RQ4 Do auxiliary tasks (STACX) and their self-tuned metaparameters further improve representation learning and performance?

Key findings

• STACX achieves state-of-the-art median human-normalized score of 364% on Atari 200M frames (vs 243% baseline).
• On DM Control, STACX/ STAC improve mean scores across feature, pixel, and RWRL settings (e.g., from 217 to 389 with features, from 108 to 202 with pixels, and from 195 to 295 in RWRL).
• Ablation studies show performance improves as more metaparameters are self-tuned; STACX consistently outperforms IMPALA baselines.
• STACX demonstrates robustness to outer-hyperparameter perturbations and shows interpretable metaparameter trajectories during training.
• STACX scales to 21 self-tuned hyperparameters (vs prior work with fewer), without substantial compute increases.
• STACX’s auxiliary heads provide additional gains primarily in pixel-based DM Control and not universally in feature-based settings.