[Paper Review] PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning

Neeratyoy Mallik, Edward M. Bergman+6 more|arXiv (Cornell University)|Jun 21, 2023

Machine Learning and Data Classification7 citations

TL;DR

PriorBand is a practical hyperparameter optimization (HPO) method for deep learning that integrates expert priors and cheap proxy tasks to achieve superior efficiency and robustness. It outperforms existing methods across diverse DL benchmarks, showing strong anytime performance and resilience to poor expert beliefs.

ABSTRACT

Hyperparameters of Deep Learning (DL) pipelines are crucial for their downstream performance. While a large number of methods for Hyperparameter Optimization (HPO) have been developed, their incurred costs are often untenable for modern DL. Consequently, manual experimentation is still the most prevalent approach to optimize hyperparameters, relying on the researcher's intuition, domain knowledge, and cheap preliminary explorations. To resolve this misalignment between HPO algorithms and DL researchers, we propose PriorBand, an HPO algorithm tailored to DL, able to utilize both expert beliefs and cheap proxy tasks. Empirically, we demonstrate PriorBand's efficiency across a range of DL benchmarks and show its gains under informative expert input and robustness against poor expert beliefs

Motivation & Objective

• Address the misalignment between existing HPO methods and modern deep learning practices, which demand high efficiency under limited compute budgets.
• Develop an HPO algorithm that effectively leverages both expert knowledge and cheap proxy evaluations to accelerate hyperparameter search.
• Ensure robustness against inaccurate expert priors while maintaining strong performance under low compute budgets.
• Support mixed-type hyperparameter spaces (categorical, numerical, log) and enable efficient parallelization.
• Provide a simple, model-free, and modular HPO framework that is easy to understand and implement in real-world DL pipelines.

Proposed method

• PriorBand extends multi-fidelity HPO algorithms like HyperBand by incorporating expert-provided priors over hyperparameter configurations.
• It uses a hierarchical bandit-style allocation strategy that prioritizes configurations with higher prior probability, especially during early, low-budget evaluations.
• The method dynamically allocates compute resources across configurations based on performance on cheap proxy tasks (e.g., smaller datasets or fewer training steps).
• PriorBand integrates expert beliefs via a prior distribution over hyperparameters, which is used to guide initial sampling and allocation decisions.
• It maintains model-free simplicity by avoiding complex surrogate models or kernel design for discrete hyperparameters.
• The algorithm supports asynchronous and parallel execution, enabling efficient scaling on modern hardware.

Figure 1: Both plots compare Random Search (RS), sampling from prior (RS+Prior), HyperBand (HB), and our method PriorBand which utilizes a good prior as defined by an expert. [ Left ] Tuning a large transformer on the 1B word benchmark. PriorBand leverages the prior, achieving strong anytime perform

Experimental results

Research questions

• RQ1 Can an HPO method effectively combine expert priors and cheap proxy tasks to improve hyperparameter optimization efficiency in deep learning?
• RQ2 How does PriorBand perform under low compute budgets compared to standard HPO baselines like random search, grid search, or HyperBand?
• RQ3 How robust is PriorBand to inaccurate or misleading expert priors in real-world deep learning settings?
• RQ4 To what extent can PriorBand scale efficiently in parallel and asynchronous computing environments?
• RQ5 Can PriorBand be flexibly extended to enhance existing multi-fidelity and model-based HPO algorithms?

Key findings

• PriorBand achieved the best final validation error across all 12 benchmarks tested, consistently outperforming baselines including HyperBand, BO, and Mobster.
• On the PD1-LM1B benchmark, PriorBand+BO achieved a final error of $0.630 /pm 0.018$, significantly better than Mobster+ESP ($0.634 /pm 0.014$) and Mobster ($0.643 /pm 0.014$).
• With a good expert prior, PriorBand reduced the final error on JAHS-C10 from $12.192 /pm 1.558$ (Mobster) to $8.281 /pm 0.221$ (PriorBand+BO).
• Even with a poor expert prior, PriorBand maintained strong performance, reducing error on PD1-ImageNet from $0.333 /pm 0.056$ (Mobster) to $0.328 /pm 0.035$ with PriorBand.
• In a 5× budget setting, PriorBand+BO achieved a final error of $0.249 /pm 0.025$ on PD1-ImageNet, outperforming Mobster ($0.410 /pm 0.233$) and Mobster+ESP ($0.336 /pm 0.164$).
• The method demonstrated strong anytime performance, with PriorBand achieving better results than random search and HyperBand even at early stages of the search.

Figure 2: A comparison of naive solutions to Equation 1 on the $3$ -dimensional multi-fidelity Hartmann benchmarks. We compare different versions of HB , utilizing different strengths of the prior distribution $\pi(\cdot)$ : HB with 0% influence of the prior ( HB ), HB with 50% sampling from the pri