[Paper Review] NAS-Bench-101: Towards Reproducible Neural Architecture Search

Chris Ying, Aaron Klein+4 more|arXiv (Cornell University)|Feb 25, 2019

Machine Learning in Materials ScienceMaterials Science253 citations

TL;DR

NAS-Bench-101 provides a public, exhaustive dataset mapping 423k unique CNN architectures to training and evaluation metrics on CIFAR-10, enabling rapid, reproducible NAS benchmarking without retraining models.

ABSTRACT

Recent advances in neural architecture search (NAS) demand tremendous computational resources, which makes it difficult to reproduce experiments and imposes a barrier-to-entry to researchers without access to large-scale computation. We aim to ameliorate these problems by introducing NAS-Bench-101, the first public architecture dataset for NAS research. To build NAS-Bench-101, we carefully constructed a compact, yet expressive, search space, exploiting graph isomorphisms to identify 423k unique convolutional architectures. We trained and evaluated all of these architectures multiple times on CIFAR-10 and compiled the results into a large dataset of over 5 million trained models. This allows researchers to evaluate the quality of a diverse range of models in milliseconds by querying the pre-computed dataset. We demonstrate its utility by analyzing the dataset as a whole and by benchmarking a range of architecture optimization algorithms.

Motivation & Objective

• Provide a publicly available, exhaustive NAS dataset to enable reproducible research.
• Define a compact, expressive search space and exhaustively evaluate architectures.
• Offer a standardized training/evaluation pipeline to ensure fair comparisons.
• Demonstrate the dataset’s utility through analysis of search space properties and algorithm benchmarking.

Proposed method

• Define a 7-node cell graph space with 3 operation labels (3x3 conv, 1x1 conv, 3x3 max-pool) and constrain to V ≤ 7 and at most 9 edges.
• Enumerate ~423k unique graphs using graph isomorphism reduction and a fixed cell encoding.
• Translate graphs to CNN cells with a fixed combine semantics (concatenate at outputs, sum elsewhere).
• Train every architecture on CIFAR-10 with a single, public training pipeline and fixed hyperparameters (cosine LR decay, RMSProp, L2 regularization).
• Repeat training three times per architecture across four epoch budgets (4, 12, 36, 108) to measure variance and enable multi-fidelity benchmarking.

Experimental results

Research questions

• RQ1 Can an exhaustively evaluated NAS search space be used to analyze architectural design choices and search space properties?
• RQ2 How do different NAS optimization algorithms perform when benchmarked on a fixed, public dataset?
• RQ3 What is the impact of architectural operations and cell depth/width on accuracy and training cost within NAS-Bench-101?
• RQ4 To what extent do locality properties of the search space affect NAS algorithm performance?

Key findings

• The dataset maps 423k unique architectures to training/validation/test metrics, enabling rapid lookups without retraining.
• The best architecture achieves a mean test accuracy of 94.32% on CIFAR-10; ResNet-like and Inception-like cells reach 93.12% and 92.95%, respectively.
• Most architectures reach high training accuracy with low validation/test misalignment (validation and test accuracy correlate highly, r=0.999 at 108 epochs).
• Hand-designed cells lie near the Pareto frontier of accuracy versus training cost, indicating topology and operation choice are critical for high performance and efficiency.
• Locality in the search space is strong, with meaningful performance changes concentrated within small edit distances; ~35.4% of the space lies within distance 6 of top graphs.