[Paper Review] PASHA: Efficient HPO and NAS with Progressive Resource Allocation

Ondrej Bohdal, Lukas Balles+4 more|arXiv (Cornell University)|Jul 14, 2022

Machine Learning and Data Classification4 citations

TL;DR

PASHA is a progressive resource allocation method for efficient hyperparameter optimization (HPO) and neural architecture search (NAS) that dynamically increases maximum resources during tuning, reducing computational cost while maintaining model performance. It achieves up to 15.5x speedup over ASHA on large datasets like WMT without sacrificing accuracy, outperforming fixed-budget methods and one-epoch baselines in robustness and efficiency.

ABSTRACT

Hyperparameter optimization (HPO) and neural architecture search (NAS) are methods of choice to obtain the best-in-class machine learning models, but in practice they can be costly to run. When models are trained on large datasets, tuning them with HPO or NAS rapidly becomes prohibitively expensive for practitioners, even when efficient multi-fidelity methods are employed. We propose an approach to tackle the challenge of tuning machine learning models trained on large datasets with limited computational resources. Our approach, named PASHA, extends ASHA and is able to dynamically allocate maximum resources for the tuning procedure depending on the need. The experimental comparison shows that PASHA identifies well-performing hyperparameter configurations and architectures while consuming significantly fewer computational resources than ASHA.

Motivation & Objective

• To address the high computational cost of HPO and NAS when training on large datasets with limited resources.
• To improve upon existing multi-fidelity methods like ASHA, which fix maximum resources and may underutilize early performance signals.
• To develop a method that dynamically allocates resources based on observed performance, reducing wasted computation.
• To enable practical, cost-effective HPO and NAS for practitioners with constrained computational budgets.
• To combine efficiently with sample-efficient strategies like Bayesian Optimization for further gains.

Proposed method

• PASHA extends ASHA by introducing a progressive increase in maximum resources allocated to promising configurations over time.
• It starts with a small initial maximum resource level and increases it incrementally as more information becomes available from early evaluations.
• The algorithm uses a multi-fidelity framework with successive halving, but allows the maximum resource level to grow based on performance stability across rungs.
• Resource levels are defined in terms of training steps or gradient updates, enabling fine-grained control and adaptability.
• PASHA determines when to stop allocating more resources based on the stability of configuration rankings across successive rungs.
• It integrates with Bayesian Optimization via BOHB-like setups, combining sample efficiency with progressive resource scaling.

Experimental results

Research questions

• RQ1 Can a progressive resource allocation strategy reduce computational cost in HPO and NAS without degrading final model performance?
• RQ2 How does dynamic adjustment of maximum resources affect the speed and robustness of configuration search compared to fixed-budget methods?
• RQ3 To what extent can PASHA outperform one-epoch baselines and random search in identifying high-performing hyperparameters or architectures?
• RQ4 How does PASHA perform when combined with Bayesian optimization in terms of accuracy and efficiency?
• RQ5 What is the impact of resource granularity and rung count on PASHA’s effectiveness in different benchmark settings?

Key findings

• On the WMT benchmark, PASHA achieved a speedup of 15.5x over ASHA, reducing runtime from 43.7 hours to 2.8 hours, while maintaining competitive accuracy (62.04% vs. 62.72%).
• On ImageNet, PASHA reduced runtime from 7.3 hours (ASHA) to 3.8 hours, achieving 73.37% accuracy, outperforming the one-epoch baseline (63.40%)
• The one-epoch baseline performed poorly on ImageNet (63.40%) but well on WMT (62.36%), indicating its lack of robustness across datasets.
• Random search performed significantly worse than all other methods, with only 33.93% accuracy on WMT, highlighting the need for structured search strategies.
• PASHA’s performance was stable across multiple runs, with low standard deviations (e.g., 2.05 for WMT), indicating consistent configuration selection.
• The method is effective even with limited rungs when resource granularity is high, and its speedup is most pronounced on large-scale datasets with high computational demands.