[Paper Review] Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRL

Jack Parker-Holder, Loc Nguyen+2 more|arXiv (Cornell University)|Jun 30, 2021

Neural Networks and Applications4 citations

TL;DR

This paper proposes a hierarchical, time-varying bandit algorithm that extends PB2 to efficiently optimize both continuous and categorical hyperparameters in population-based reinforcement learning. By modeling dependencies between data augmentation (categorical) and learning rates (continuous), the method achieves improved generalization on the Procgen benchmark with theoretical sublinear regret guarantees.

ABSTRACT

Despite a series of recent successes in reinforcement learning (RL), many RL algorithms remain sensitive to hyperparameters. As such, there has recently been interest in the field of AutoRL, which seeks to automate design decisions to create more general algorithms. Recent work suggests that population based approaches may be effective AutoRL algorithms, by learning hyperparameter schedules on the fly. In particular, the PB2 algorithm is able to achieve strong performance in RL tasks by formulating online hyperparameter optimization as time varying GP-bandit problem, while also providing theoretical guarantees. However, PB2 is only designed to work for continuous hyperparameters, which severely limits its utility in practice. In this paper we introduce a new (provably) efficient hierarchical approach for optimizing both continuous and categorical variables, using a new time-varying bandit algorithm specifically designed for the population based training regime. We evaluate our approach on the challenging Procgen benchmark, where we show that explicitly modelling dependence between data augmentation and other hyperparameters improves generalization.

Motivation & Objective

• To address the limitation of PB2, which only optimizes continuous hyperparameters, by enabling efficient joint optimization of continuous and categorical hyperparameters in population-based training.
• To model dependencies between categorical hyperparameters (e.g., data augmentation type) and continuous ones (e.g., learning rate) to improve generalization in RL.
• To develop a provably efficient, hierarchical bandit algorithm that supports time-varying hyperparameter scheduling with sublinear regret.
• To scale the approach to the challenging Procgen benchmark and demonstrate improved performance over random search baselines.

Proposed method

• Introduces a new time-varying batch multi-armed bandit algorithm that supports both continuous and categorical hyperparameters in a population-based training regime.
• Proposes a hierarchical design where categorical hyperparameters are selected first, followed by continuous hyperparameter optimization conditioned on the selected category.
• Uses a composite kernel function combining continuous, categorical, and time-varying components: $k_z = (1- heta)k_{ ext{Continuous}}k_{ ext{time1}} + heta k_{ ext{Categorical}}k_{ ext{time2}}$, with $\theta$ controlling the mixture.
• Employs a GP-bandit framework with a structured kernel to model cross-hyperparameter dependencies and enable efficient exploration.
• Derives gradients for hyperparameter optimization via backpropagation through the log-marginal likelihood of the GP model.
• Applies the method to online hyperparameter scheduling in PBT, enabling on-the-fly adaptation during training with theoretical regret bounds.

Experimental results

Research questions

• RQ1 Can a time-varying bandit algorithm efficiently optimize both continuous and categorical hyperparameters in a population-based RL setting?
• RQ2 Does explicitly modeling dependence between data augmentation (categorical) and learning rate (continuous) hyperparameters improve generalization in RL?
• RQ3 Is the proposed hierarchical bandit approach provably efficient, with sublinear regret guarantees, even when mixing categorical and continuous variables?
• RQ4 How does the method scale to complex benchmarks like Procgen, and how does it compare to random search baselines in terms of sample efficiency and performance?

Key findings

• The proposed method achieves sublinear regret in hyperparameter optimization, extending the theoretical guarantees of PB2 to mixed hyperparameter spaces.
• Explicitly modeling dependencies between data augmentation type and learning rate leads to improved generalization performance on the Procgen benchmark compared to random search over augmentation types.
• The hierarchical bandit approach outperforms standard PBT and PB2 in sample efficiency by effectively leveraging categorical and continuous hyperparameter interactions.
• Empirical results show that the method generalizes better across diverse Procgen environments when the hyperparameter schedule accounts for augmentation type.