[Paper Review] B2EA: An Evolutionary Algorithm Assisted by Two Bayesian Optimization Modules for Neural Architecture Search

Hyunghun Cho, Jungwook Shin+1 more|arXiv (Cornell University)|Feb 7, 2022

Machine Learning and Data Classification4 citations

TL;DR

B2EA is a multi-trial Neural Architecture Search (NAS) algorithm that combines Evolutionary Algorithm (EA) with two Bayesian Optimization (BO) surrogate models to improve search efficiency and robustness. It achieves state-of-the-art performance on 14 diverse benchmarks, including both general and cell-based search spaces, with consistent superiority across three difficulty levels of target accuracy.

ABSTRACT

The early pioneering Neural Architecture Search (NAS) works were multi-trial methods applicable to any general search space. The subsequent works took advantage of the early findings and developed weight-sharing methods that assume a structured search space typically with pre-fixed hyperparameters. Despite the amazing computational efficiency of the weight-sharing NAS algorithms, it is becoming apparent that multi-trial NAS algorithms are also needed for identifying very high-performance architectures, especially when exploring a general search space. In this work, we carefully review the latest multi-trial NAS algorithms and identify the key strategies including Evolutionary Algorithm (EA), Bayesian Optimization (BO), diversification, input and output transformations, and lower fidelity estimation. To accommodate the key strategies into a single framework, we develop B2EA that is a surrogate assisted EA with two BO surrogate models and a mutation step in between. To show that B2EA is robust and efficient, we evaluate three performance metrics over 14 benchmarks with general and cell-based search spaces. Comparisons with state-of-the-art multi-trial algorithms reveal that B2EA is robust and efficient over the 14 benchmarks for three difficulty levels of target performance. The B2EA code is publicly available at \url{https://github.com/snu-adsl/BBEA}.

Motivation & Objective

• Address the limitations of weight-sharing NAS in finding very high-performance architectures by developing a multi-trial approach.
• Overcome the high computational cost of traditional multi-trial NAS while maintaining robustness across diverse search spaces.
• Integrate key strategies—diversification, input/output transformations, and conservative early termination—into a unified framework.
• Demonstrate that multi-trial NAS remains essential for discovering top-tier models, especially in general or unconstrained search spaces.
• Develop a scalable, efficient, and robust NAS algorithm suitable for both general and cell-based search spaces.

Proposed method

• Combine Evolutionary Algorithm (EA) with two Bayesian Optimization (BO) surrogate models (GP and RF) to guide search and improve sample efficiency.
• Introduce a mutation step between BO iterations to enhance population diversity and prevent premature convergence.
• Apply input warping to transform the search space and improve modeling of non-stationary performance landscapes.
• Use output transformation to normalize performance metrics, especially beneficial for extremely difficult target tasks.
• Implement a conservative early termination strategy to reduce redundant evaluations while preserving search quality.
• Use a round-robin acquisition function scheme (EI, PI, UCB) to balance exploration and exploitation in BO, avoiding complex ensemble modeling.

Experimental results

Research questions

• RQ1 Can a multi-trial NAS algorithm outperform weight-sharing methods in discovering high-performance architectures, especially in general search spaces?
• RQ2 How do input warping and output transformation affect the performance and robustness of BO-assisted NAS across diverse benchmarks?
• RQ3 What is the impact of using two distinct BO surrogate models (GP and RF) in a cooperative framework compared to single-model approaches?
• RQ4 How does the integration of mutation in an EA-BO hybrid framework improve search diversity and convergence?
• RQ5 To what extent does conservative early termination reduce computational cost without degrading final performance?

Key findings

• B2EA achieved state-of-the-art performance across 14 benchmarks, including NAS-Bench201-ImageNet16-120, with consistent superiority across three difficulty levels of target accuracy.
• The algorithm completed the search within 15 GPU days for the most challenging target in NAS-Bench201-ImageNet16-120, outperforming existing multi-trial and one-shot methods.
• Input warping and output transformation improved performance on difficult targets but were less effective or even harmful on easier benchmarks, indicating context-dependent benefits.
• Mutation was particularly beneficial in high-difficulty scenarios, where it helped maintain diversity and avoid premature convergence.
• The dual-BO surrogate setup (GP and RF) enhanced robustness and sample efficiency, especially when combined with diversification and early termination.
• The conservative early termination strategy significantly reduced redundant evaluations while maintaining high performance, especially in high-complexity search spaces.