[Paper Review] Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammars

Simon Schrodi, Danny Stoll+4 more|arXiv (Cornell University)|Nov 3, 2022

Advanced Neural Network Applications5 citations

TL;DR

This paper proposes a unifying framework for constructing hierarchical neural architecture search (NAS) spaces using context-free grammars (CFGs), enabling compact, expressive, and regular search spaces that are orders of magnitude larger than prior work. It introduces BOHNAS, a Bayesian optimization method with a hierarchical kernel, which efficiently searches these vast spaces and achieves state-of-the-art performance, including a 4.99% accuracy gain on ImageNet-16-120.

ABSTRACT

The discovery of neural architectures from simple building blocks is a long-standing goal of Neural Architecture Search (NAS). Hierarchical search spaces are a promising step towards this goal but lack a unifying search space design framework and typically only search over some limited aspect of architectures. In this work, we introduce a unifying search space design framework based on context-free grammars that can naturally and compactly generate expressive hierarchical search spaces that are 100s of orders of magnitude larger than common spaces from the literature. By enhancing and using their properties, we effectively enable search over the complete architecture and can foster regularity. Further, we propose an efficient hierarchical kernel design for a Bayesian Optimization search strategy to efficiently search over such huge spaces. We demonstrate the versatility of our search space design framework and show that our search strategy can be superior to existing NAS approaches. Code is available at https://github.com/automl/hierarchical_nas_construction.

Motivation & Objective

• To address the lack of a unifying framework for designing hierarchical neural architecture search spaces that support full architectural search across all abstraction levels.
• To enable the construction of highly expressive, regular, and constraint-compliant search spaces that are orders of magnitude larger than existing ones.
• To develop an efficient Bayesian optimization strategy capable of navigating exponentially large hierarchical search spaces.
• To empirically validate the framework's versatility across convolutional networks, transformers, and diverse architectural design choices.
• To promote reproducible NAS research by adhering to best practices and releasing code and hyperparameters.

Proposed method

• The paper models neural architectures as function compositions generated by context-free grammars (CFGs), enabling recursive, hierarchical construction of architectures from basic building blocks.
• It enhances CFGs with mechanisms to enforce regularity through closure under substitution and to incorporate user-defined architectural constraints.
• A novel hierarchical kernel is designed for Bayesian optimization (BOHNAS), integrating multiple granularities of architectural components to model complex dependencies across levels.
• The method supports search over non-linear macro topologies, such as skip connections and branching, not limited to linear or cell-based structures.
• The framework is integrated into NAS-Bench-201 and its derivatives to enable controlled evaluation across architectural aspects like macro-architecture and block repetition.
• Extensive ablation and comparison studies are conducted using consistent training protocols, hyperparameters, and random seeds to ensure fair benchmarking.

Experimental results

Research questions

• RQ1 Can a CFG-based framework unify and scale the construction of hierarchical neural architecture search spaces while preserving regularity and supporting complex topologies?
• RQ2 How does the proposed BOHNAS method perform in comparison to existing NAS baselines on large, hierarchical search spaces?
• RQ3 To what extent can the framework uncover novel architectural patterns beyond standard cell-based or linear designs?
• RQ4 How does architectural design choice—such as macro-architecture or block repetition—affect model performance across different datasets and model types?
• RQ5 Can the framework be effectively applied to diverse architectures, including convolutional networks and transformers, under the same search protocol?

Key findings

• The proposed CFG-based search space construction enables the creation of search spaces that are 100s of orders of magnitude larger than those in prior NAS work, while maintaining expressiveness and regularity.
• BOHNAS achieves state-of-the-art performance on 6 out of 8 datasets, including a 4.99% accuracy improvement on ImageNet-16-120 using the NAS-Bench-201 training protocol.
• The method successfully discovers novel architectures, including a found transformer-based classifier head for sentiment analysis, demonstrating its ability to explore beyond standard search space constraints.
• The framework supports effective search over diverse architectures, including convolutional networks and transformers, with consistent performance gains across both.
• Ablation studies confirm the superiority of the hierarchical kernel in BOHNAS over standard kernels (e.g., WL, GCN) and show consistent performance gains across multiple search runs and seeds.
• The work adheres to the NAS best practices checklist, with full code, hyperparameters, and random seeds released, ensuring reproducibility and fair comparison.