[Paper Review] Self-Tuning Deep Reinforcement Learning

Tom Zahavy, Zhongwen Xu+6 more|arXiv (Cornell University)|Feb 28, 2020

Reinforcement Learning in RoboticsComputer Science10 references8 citations

TL;DR

This paper proposes Self-Tuning Actor Critic (STAC), a deep reinforcement learning method that uses differentiable cross-validation and metagradients to automatically tune hyperparameters during training. STAC improves performance on Atari 2600 by boosting the median human-normalized score from 243% to 364% over 200 million frames, with no increase in compute and enhanced sample efficiency.

ABSTRACT

Reinforcement learning (RL) algorithms often require expensive manual or automated hyperparameter searches in order to perform well on a new domain. This need is particularly acute in modern deep RL architectures which often incorporate many modules and multiple loss functions. In this paper, we take a step towards addressing this issue by using metagradients (Xu et al., 2018) to tune these hyperparameters via differentiable cross validation, whilst the agent interacts with and learns from the environment. We present the Self-Tuning Actor Critic (STAC) which uses this process to tune the hyperparameters of the usual loss function of the IMPALA actor critic agent(Espeholt et. al., 2018), to learn the hyperparameters that define auxiliary loss functions, and to balance trade offs in off policy learning by introducing and adapting the hyperparameters of a novel leaky V-trace operator. The method is simple to use, sample efficient and does not require significant increase in compute. Ablative studies show that the overall performance of STAC improves as we adapt more hyperparameters. When applied to 57 games on the Atari 2600 environment over 200 million frames our algorithm improves the median human normalized score of the baseline from 243% to 364%.

Motivation & Objective

• To reduce the need for manual or automated hyperparameter tuning in deep reinforcement learning, especially in complex architectures with multiple loss functions.
• To enable automatic adaptation of hyperparameters during training, improving sample efficiency and performance.
• To extend the IMPALA actor-critic framework by learning optimal hyperparameters for both main and auxiliary loss functions.
• To introduce and adapt a novel leaky V-trace operator with learnable hyperparameters to balance off-policy learning trade-offs.
• To demonstrate that tuning more hyperparameters leads to consistent performance gains in diverse environments.

Proposed method

• Uses metagradients to compute gradients of the validation loss with respect to hyperparameters, enabling end-to-end hyperparameter optimization.
• Applies differentiable cross-validation to evaluate hyperparameter performance during training without requiring separate validation episodes.
• Introduces a learnable leaky V-trace operator that adjusts the off-policy correction via adaptive hyperparameters.
• Tunes hyperparameters of the main loss function, auxiliary losses, and the V-trace operator simultaneously within a unified training loop.
• Maintains sample efficiency by avoiding additional environment rollouts or compute overhead for hyperparameter search.
• Employs a meta-optimization loop where hyperparameters are updated based on performance on a differentiable validation metric.

Experimental results

Research questions

• RQ1 Can hyperparameters in deep RL be automatically tuned during training using differentiable methods?
• RQ2 Does self-tuning of multiple hyperparameters improve sample efficiency and final performance in complex RL agents?
• RQ3 Can a novel leaky V-trace operator with adaptive hyperparameters improve off-policy learning stability and performance?
• RQ4 How does the performance of a self-tuning agent compare to a baseline with fixed hyperparameters across diverse environments?
• RQ5 Does increasing the number of tunable hyperparameters lead to measurable performance gains?

Key findings

• STAC improves the median human-normalized score on Atari 2600 from 243% to 364% across 57 games over 200 million frames.
• The performance gain is consistent across environments, with improvements observed as more hyperparameters are tuned.
• The method achieves better performance without increasing compute or requiring additional environment interactions.
• Ablative studies confirm that tuning more hyperparameters leads to greater performance gains, validating the scalability of the approach.
• The use of differentiable cross-validation enables stable and effective hyperparameter updates during training.
• The self-tuned leaky V-trace operator effectively balances off-policy learning trade-offs, contributing to improved sample efficiency.