[Paper Review] Cost-aware Bayesian Optimization

Eric Hans Lee, Valerio Perrone+2 more|arXiv (Cornell University)|Mar 22, 2020

Machine Learning and Data Classification37 references30 citations

TL;DR

Introduces CArBO, a cost-aware Bayesian optimization method that prioritizes cheap evaluations early and gradually shifts to costlier ones, improving hyperparameter search under a fixed cost budget.

ABSTRACT

Bayesian optimization (BO) is a class of global optimization algorithms, suitable for minimizing an expensive objective function in as few function evaluations as possible. While BO budgets are typically given in iterations, this implicitly measures convergence in terms of iteration count and assumes each evaluation has identical cost. In practice, evaluation costs may vary in different regions of the search space. For example, the cost of neural network training increases quadratically with layer size, which is a typical hyperparameter. Cost-aware BO measures convergence with alternative cost metrics such as time, energy, or money, for which vanilla BO methods are unsuited. We introduce Cost Apportioned BO (CArBO), which attempts to minimize an objective function in as little cost as possible. CArBO combines a cost-effective initial design with a cost-cooled optimization phase which depreciates a learned cost model as iterations proceed. On a set of 20 black-box function optimization problems we show that, given the same cost budget, CArBO finds significantly better hyperparameter configurations than competing methods.

Motivation & Objective

• Motivate and address the mismatch between iteration-budget BO and variable evaluation costs in real-world hyperparameter optimization.
• Develop a black-box cost-aware BO algorithm that minimizes the objective within a cost budget.
• Empirically demonstrate that CArBO outperforms standard EI and EIpu in sequential and batch settings over 20 HPO benchmarks.

Proposed method

• Propose a cost-effective initial design to maximize space coverage within a warm-start budget (minimax fill criterion).
• Introduce cost-cooling (EI-cool) that decreases the influence of the cost model as more budget is spent, transitioning from EIpu to EI.
• Use a warped GP cost model to predict evaluation cost and feed into EI-cool.
• Enable batch BO with fantasies to achieve linear scaling in batch size.
• Provide a default initial design budget of tau_init = tau/8 and demonstrate robustness to this choice.

Experimental results

Research questions

• RQ1 How can Bayesian optimization be adapted to minimize objectives under a fixed cost budget rather than a fixed iteration budget?
• RQ2 Does a two-part strategy (cheap initial design plus cost-cooling) improve cost-efficiency and final objective quality across heterogeneous HPO problems?
• RQ3 How does CArBO compare to EI and EIpu in sequential and batch hyperparameter optimization under real-world cost budgets?
• RQ4 What low-variance cost models best complement cost-aware BO for extrapolating runtimes (e.g., flop-based linear models)?

Key findings

• CArBO significantly outperforms EI and EIpu across 20 HPO benchmarks within the same cost budget.
• Batch CArBO scales linearly with batch size, maintaining strong performance up to batch sizes of 16.
• A cost-effective initial design yields substantially more information than a standard grid under the same budget.
• EI-cool (cost-cooling) effectively transitions from cost-aware EIpu to standard EI, yielding robust improvements in practice.
• Low-variance cost models (e.g., flop-based linear models) can further improve performance, especially in limited data regimes.