[Paper Review] Provably Efficient Online Hyperparameter Optimization with Population-Based Bandits

Jack Parker-Holder, Loc Nguyen+1 more|arXiv (Cornell University)|Feb 6, 2020

Advanced Bandit Algorithms Research36 citations

TL;DR

PB2 introduces a provably efficient PBT-style algorithm by using time-varying Gaussian process bandits to guide hyperparameter schedules, achieving sublinear regret and improved RL performance with smaller populations. It replaces heuristic exploration with principled Bayesian optimization in a parallel training setting.

ABSTRACT

Many of the recent triumphs in machine learning are dependent on well-tuned hyperparameters. This is particularly prominent in reinforcement learning (RL) where a small change in the configuration can lead to failure. Despite the importance of tuning hyperparameters, it remains expensive and is often done in a naive and laborious way. A recent solution to this problem is Population Based Training (PBT) which updates both weights and hyperparameters in a single training run of a population of agents. PBT has been shown to be particularly effective in RL, leading to widespread use in the field. However, PBT lacks theoretical guarantees since it relies on random heuristics to explore the hyperparameter space. This inefficiency means it typically requires vast computational resources, which is prohibitive for many small and medium sized labs. In this work, we introduce the first provably efficient PBT-style algorithm, Population-Based Bandits (PB2). PB2 uses a probabilistic model to guide the search in an efficient way, making it possible to discover high performing hyperparameter configurations with far fewer agents than typically required by PBT. We show in a series of RL experiments that PB2 is able to achieve high performance with a modest computational budget.

Motivation & Objective

• Motivate efficient automatic hyperparameter optimization for reinforcement learning where hyperparameters change over time.
• Replace heuristic PBT exploration with a theoretically grounded, GP-based bandit approach.
• Provide regret bounds for a PBT-style algorithm and demonstrate practical gains in RL benchmarks.

Proposed method

• Model the time-varying hyperparameter optimization as a batch Gaussian Process (GP) bandit problem with time-varying kernels.
• Use a time-varying kernel to capture non-stationarity in neural network training dynamics (K = SE ∘ K_time).
• Select a batch of hyperparameters for parallel agents by sequentially maximizing an acquisition function that accounts for pending evaluations.
• Prove a sublinear regret bound for PB2 under mild smoothness and Lipschitz conditions (Theorem 2).
• Demonstrate PB2 experimentally against PBT, RS, BO, and ASHA on OpenAI Gym and IMPALA tasks.

Experimental results

Research questions

• RQ1 Can PB2 achieve sublinear regret in time-varying hyperparameter optimization settings?
• RQ2 Does a GP-based, time-varying exploration strategy outperform heuristic PBT exploration in RL tasks?
• RQ3 How does PB2 perform with smaller population sizes and under mis-specified hyperparameter ranges?
• RQ4 Can PB2 scale with larger populations and competitive RL benchmarks like IMPALA?
• RQ5 What is the practical impact of PB2 on sample efficiency and final RL performance compared to existing baselines?

Key findings

• PB2 achieves competitive or superior final RL performance compared with PBT, RS, BO, and ASHA across multiple tasks.
• With smaller populations, PB2 outperforms PBT and often matches or exceeds hand-tuned baselines in IMPALA experiments.
• PB2 remains robust when hyperparameter ranges are mis-specified, maintaining strong performance where PBT declines.
• Theoretical result: PB2 achieves a sublinear regret bound that improves with larger population sizes (B) and more correlated time-varying functions (lower ω).
• PB2 learns hyperparameter schedules online within a single training run, reducing the computational burden compared to traditional batch BO approaches.