[Paper Review] Neural Architecture Generator Optimization

Binxin Ru, Pedro M. Esperança+1 more|arXiv (Cornell University)|Apr 3, 2020

Machine Learning and Data Classification65 references17 citations

TL;DR

This paper proposes Neural Architecture Generator Optimization (NAGO), a novel NAS framework that shifts focus from optimizing individual architectures to optimizing a stochastic network generator. By using a hierarchical, graph-based search space with continuous hyperparameters, NAGO enables efficient Bayesian optimization, yielding lightweight, highly competitive models across six benchmarks, including ImageNet, with state-of-the-art performance.

ABSTRACT

Neural Architecture Search (NAS) was first proposed to achieve state-of-the-art performance through the discovery of new architecture patterns, without human intervention. An over-reliance on expert knowledge in the search space design has however led to increased performance (local optima) without significant architectural breakthroughs, thus preventing truly novel solutions from being reached. In this work we 1) are the first to investigate casting NAS as a problem of finding the optimal network generator and 2) we propose a new, hierarchical and graph-based search space capable of representing an extremely large variety of network types, yet only requiring few continuous hyper-parameters. This greatly reduces the dimensionality of the problem, enabling the effective use of Bayesian Optimisation as a search strategy. At the same time, we expand the range of valid architectures, motivating a multi-objective learning approach. We demonstrate the effectiveness of this strategy on six benchmark datasets and show that our search space generates extremely lightweight yet highly competitive models.

Motivation & Objective

• To address the limitation of existing NAS methods that rely on narrow, expert-designed search spaces, leading to local optima and lack of architectural innovation.
• To reduce the dimensionality of NAS by framing it as a generator hyperparameter optimization problem, enabling more effective use of global optimization strategies.
• To design a highly expressive, hierarchical, graph-based search space capable of generating diverse architectures with minimal continuous hyperparameters.
• To enable multi-objective and multi-fidelity optimization for efficient architecture discovery while maintaining high performance.
• To demonstrate that optimizing the generator, rather than individual architectures, leads to novel, lightweight, and competitive models.

Proposed method

• Proposes a hierarchical, graph-based search space (HNAG) with three levels: top-level cells, mid-level cell structures, and bottom-level atomic operations, each governed by independent graph generators.
• Employs the Watts-Strogatz model for top and bottom-level graph generation with hyperparameters $\bm{\theta}_{top} = [N_t, K_t, P_t]$ and $\bm{\theta}_{bottom} = [N_b, K_b, P_b]$, and the Erdős–Rényi model for mid-level graphs with $\bm{\theta}_{mid} = [N_m, P_m]$.
• Treats the entire architecture as a sample from a stochastic generator, allowing the search to focus on optimizing generator hyperparameters rather than individual architectures.
• Applies Bayesian Optimization (BO) in both multi-fidelity and multi-objective settings to efficiently explore the low-dimensional generator space.
• Uses a dynamic channel computation strategy that preserves a fixed parameter budget, enabling lightweight model generation.
• Employs BOHB and MOBO for efficient hyperparameter search, with evaluation on CIFAR-10, CIFAR-100, and ImageNet under resource constraints.

Experimental results

Research questions

• RQ1 Can NAS be effectively reframed as a generator hyperparameter optimization problem to escape local optima and discover novel architectures?
• RQ2 Can a hierarchical, graph-based search space with continuous hyperparameters significantly reduce the dimensionality of NAS while maintaining architectural expressiveness?
• RQ3 Does optimizing a network generator using Bayesian optimization yield models that are both lightweight and competitive across diverse vision benchmarks?
• RQ4 How does the performance of the proposed NAGO framework compare to existing one-shot and supernet-based NAS methods in terms of accuracy and parameter efficiency?
• RQ5 To what extent can the generator hyperparameters provide interpretable insights into the global structural properties of high-performing architectures?

Key findings

• NAGO achieves state-of-the-art performance on six vision benchmarks, including CIFAR-10, CIFAR-100, and ImageNet, outperforming existing NAS methods.
• The HNAG search space successfully generates a wide variety of architectures with diverse wiring patterns, memory usage, and training times, including DARTS-like structures.
• On ImageNet, the HNAG-BOHB approach achieves superior mean performance compared to [14] and other benchmarks, despite using only 10 BOHB iterations with limited search budgets.
• The method dynamically computes channel counts to preserve a fixed parameter budget, resulting in highly efficient and lightweight models.
• The absence of techniques like DropPath and Auxiliary Towers in the initial training protocol suggests further accuracy gains are possible with optimized training protocols.
• The framework enables interpretable, high-level insights into architecture design by linking generator hyperparameters to global structural properties such as connectivity and operation types.