[Paper Review] Tune As You Scale: Hyperparameter Optimization For Compute Efficient Training

Abraham J. Fetterman, Ellie Kitanidis+7 more|arXiv (Cornell University)|Jun 13, 2023

Machine Learning and Data Classification4 citations

TL;DR

CARBS is a cost-aware Bayesian optimization algorithm that performs local search around the performance-cost Pareto frontier to enable efficient, automated hyperparameter tuning for large deep learning models. It discovers scaling laws for all hyperparameters, reproduces Chinchilla's model scaling results with less compute, and solves the entire ProcGen benchmark using only PPO tuning—demonstrating superior sample efficiency and robustness compared to baselines.

ABSTRACT

Hyperparameter tuning of deep learning models can lead to order-of-magnitude performance gains for the same amount of compute. Despite this, systematic tuning is uncommon, particularly for large models, which are expensive to evaluate and tend to have many hyperparameters, necessitating difficult judgment calls about tradeoffs, budgets, and search bounds. To address these issues and propose a practical method for robustly tuning large models, we present Cost-Aware Pareto Region Bayesian Search (CARBS), a Bayesian optimization algorithm that performs local search around the performance-cost Pareto frontier. CARBS does well even in unbounded search spaces with many hyperparameters, learns scaling relationships so that it can tune models even as they are scaled up, and automates much of the "black magic" of tuning. Among our results, we effectively solve the entire ProcGen benchmark just by tuning a simple baseline (PPO, as provided in the original ProcGen paper). We also reproduce the model size vs. training tokens scaling result from the Chinchilla project (Hoffmann et al. 2022), while simultaneously discovering scaling laws for every other hyperparameter, via an easy automated process that uses significantly less compute and is applicable to any deep learning problem (not just language models).

Motivation & Objective

• To address the inefficiency and manual effort in hyperparameter tuning for large deep learning models, especially when evaluations are expensive and scaling relationships are non-trivial.
• To automate the tuning process by eliminating manual specification of search bounds and training budgets, reducing bias and irreproducibility in hyperparameter selection.
• To enable robust, scalable hyperparameter optimization in high-dimensional spaces by learning cost-dependent scaling laws for all hyperparameters.
• To reproduce complex scaling behaviors—like those in the Chinchilla project—automatically and with significantly reduced compute.
• To demonstrate that simple baselines, when tuned with CARBS, can achieve state-of-the-art performance across diverse tasks including reinforcement learning, language modeling, and image classification.

Proposed method

• CARBS uses Gaussian process (GP) surrogate models to jointly model both model performance and training cost (in time) as functions of hyperparameters.
• It performs local Bayesian optimization in the neighborhood of the current performance-cost Pareto frontier, improving sample efficiency in high-dimensional spaces.
• The acquisition function employs expected improvement over the Pareto frontier (EI-PF), incorporating cost-awareness to prioritize efficient, high-performing configurations.
• It dynamically learns scaling laws for hyperparameters such as learning rate, batch size, and training tokens, enabling extrapolation to larger models.
• It includes a random cost clamping mechanism to balance exploration of low-cost and high-performance regions, and resampling to reduce noise in observations.
• The algorithm requires only an initial search center for each hyperparameter, avoiding the need for predefined search bounds or budget constraints.

Experimental results

Research questions

• RQ1 Can a Bayesian optimization algorithm efficiently tune deep learning models with many hyperparameters and high evaluation cost, especially when scaling up?
• RQ2 Can a hyperparameter tuner automatically discover scaling laws for all hyperparameters, including training tokens and epochs, without manual intervention?
• RQ3 Does a cost-aware, Pareto-focused local search strategy outperform global or non-cost-aware Bayesian optimization in terms of sample efficiency and robustness?
• RQ4 Can a simple baseline like PPO achieve state-of-the-art performance on challenging benchmarks like ProcGen when tuned with CARBS?
• RQ5 Can CARBS reproduce complex scaling laws—such as those in the Chinchilla project—using significantly less compute than traditional methods?

Key findings

• CARBS achieved the best median performance and lowest variance on the language modeling task, outperforming other tuning algorithms including ASHA and BOHB.
• On the MuJoCo Ant-v4 reinforcement learning task, CARBS produced more observations near the Pareto frontier than other algorithms, indicating superior sample efficiency and surrogate model quality.
• CARBS successfully solved the entire ProcGen benchmark by tuning only the PPO baseline, demonstrating that systematic hyperparameter tuning can unlock breakthrough performance from simple models.
• CARBS reproduced the model size vs. training tokens scaling law from the Chinchilla project (Hoffmann et al., 2022) while simultaneously discovering scaling laws for all other hyperparameters, all within a single automated tuning run.
• The ablation study showed that removing key components—such as Pareto-based acquisition, cost clamping, or resampling—significantly degraded performance, confirming the necessity of each design choice.
• CARBS required no search bounds or budget specifications, making it fully automated and applicable to any deep learning problem, including those with hyperparameters like training duration that are often fixed in practice.