[Paper Review] Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search

Arber Zela, Aaron Klein+2 more|arXiv (Cornell University)|Jul 18, 2018

Machine Learning and Data ClassificationComputer Science21 references88 citations

TL;DR

The paper proposes BOHB to jointly optimize neural architecture and hyperparameters with progressively increasing budgets, showing competitive CIFAR-10 results within a 3-hour limit and revealing budget-aware interactions between architecture and hyperparameters.

ABSTRACT

While existing work on neural architecture search (NAS) tunes hyperparameters in a separate post-processing step, we demonstrate that architectural choices and other hyperparameter settings interact in a way that can render this separation suboptimal. Likewise, we demonstrate that the common practice of using very few epochs during the main NAS and much larger numbers of epochs during a post-processing step is inefficient due to little correlation in the relative rankings for these two training regimes. To combat both of these problems, we propose to use a recent combination of Bayesian optimization and Hyperband for efficient joint neural architecture and hyperparameter search.

Motivation & Objective

• Motivate joint optimization of architecture and hyperparameters rather than post-hoc tuning.
• Show that short training budgets may not correlate well with long-budget performance.
• Demonstrate an anytime, budget-aware AutoML approach that gradually increases resources.
• Evaluate joint NAS-HP search on CIFAR-10 under a 3-hour constraint.

Proposed method

• Cast neural architecture search as a hyperparameter optimization problem with categorical and conditional hyperparameters.
• Adopt BOHB, a combination of Bayesian optimization and Hyperband, for efficient multi-budget search.
• Define a joint search space with 10 architectural choices and 7 hyperparameters for a multi-branch residual architecture.
• Use successive halving to allocate more compute to promising configurations across budgets.
• Train and evaluate configurations under multiple budgets (e.g., 400s, 1200s, 1h, 3h) to capture budget-aware performance.
• Compare to manually constructed architectures and analyze budget correlations and parameter importance.

Experimental results

Research questions

• RQ1 Can neural architecture search be effectively performed jointly with hyperparameter optimization?
• RQ2 How do short and long training budgets correlate in ranking configurations, and what budget should be used during optimization?
• RQ3 Is the BOHB approach effective under a strict time budget for CIFAR-10?
• RQ4 What architectural and hyperparameter choices are most influential under limited compute budgets?

Key findings

• Joint architecture and hyperparameter search with BOHB yields competitive CIFAR-10 results within a 3-hour budget (test error 3.18%).
• The best performing architecture under 3h is a medium-sized multi-branch residual network (26 2x64d).
• Spearman correlations show strong alignment between adjacent budgets but degrade quickly across larger budget gaps, making short-budget rankings unreliable for long-budget selection.
• Budget-aware analysis (fANOVA) indicates different hyperparameters and architectural choices gain or lose importance as the budget changes, highlighting interaction effects.
• BOHB-based search outperformed several standard architectures under the same optimization pipeline and budget, demonstrating the value of joint optimization.