[Paper Review] NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search

Xuanyi Dong, Yi Yang|arXiv (Cornell University)|Jan 2, 2020

Advanced Neural Network ApplicationsComputer Science290 citations

TL;DR

The paper introduces NAS-Bench-201, a fixed cell-based search space benchmark with 15,625 architectures evaluated across three datasets, plus rich training logs and diagnostic data to enable fair NAS algorithm comparisons and reuse of trained model parameters.

ABSTRACT

Neural architecture search (NAS) has achieved breakthrough success in a great number of applications in the past few years. It could be time to take a step back and analyze the good and bad aspects in the field of NAS. A variety of algorithms search architectures under different search space. These searched architectures are trained using different setups, e.g., hyper-parameters, data augmentation, regularization. This raises a comparability problem when comparing the performance of various NAS algorithms. NAS-Bench-101 has shown success to alleviate this problem. In this work, we propose an extension to NAS-Bench-101: NAS-Bench-201 with a different search space, results on multiple datasets, and more diagnostic information. NAS-Bench-201 has a fixed search space and provides a unified benchmark for almost any up-to-date NAS algorithms. The design of our search space is inspired from the one used in the most popular cell-based searching algorithms, where a cell is represented as a DAG. Each edge here is associated with an operation selected from a predefined operation set. For it to be applicable for all NAS algorithms, the search space defined in NAS-Bench-201 includes all possible architectures generated by 4 nodes and 5 associated operation options, which results in 15,625 candidates in total. The training log and the performance for each architecture candidate are provided for three datasets. This allows researchers to avoid unnecessary repetitive training for selected candidate and focus solely on the search algorithm itself. The training time saved for every candidate also largely improves the efficiency of many methods. We provide additional diagnostic information such as fine-grained loss and accuracy, which can give inspirations to new designs of NAS algorithms. In further support, we have analyzed it from many aspects and benchmarked 10 recent NAS algorithms.

Motivation & Objective

• Provide a fixed, algorithm-agnostic cell-based search space for reproducible NAS evaluations.
• Offer training logs, accuracy, and loss data across multiple datasets for each architecture to avoid redundant training.
• Deliver additional diagnostic information (parameters, FLOPs, latency, fine-grained training dynamics) to inform NAS design.
• Enable benchmarking and fair comparison of diverse NAS algorithms on standardized datasets.

Proposed method

• Define a fixed 4-node densely connected DAG cell with 5 operation options per edge (zeroize, skip, 1x1 conv, 3x3 conv, 3x3 avg pool).
• Assemble cells into a standard macro skeleton with three stages and a residual downsampling block comparable to modern cell-based NAS methods.
• Train and evaluate every architecture candidate (15,625 total) on CIFAR-10, CIFAR-100, and ImageNet-16-120 using a unified training protocol and epochs.
• Provide complete training logs (loss/accuracy per epoch), parameter counts, FLOPs, and latency for each architecture & run.
• Benchmark 10 NAS algorithms (RL, ES, differentiable, HPO) on NAS-Bench-201 to assess speedups and transferability.

Experimental results

Research questions

• RQ1 How does a fixed, comprehensive NAS cell space enable fairer comparison across NAS algorithms?
• RQ2 What is the performance distribution and transferability of architectures across CIFAR-10, CIFAR-100, and ImageNet-16-120 within NAS-Bench-201?
• RQ3 Can NAS algorithms achieve substantial speedups using the benchmark’s precomputed results without retraining architectures?
• RQ4 What diagnostic information can inform the design of more robust and efficient NAS methods?

Key findings

• The NAS-Bench-201 space contains 15,625 candidate cells and supports evaluations across three datasets.
• - A wide range of NAS methods can be benchmarked in a unified, reproducible framework with consistent training setups.
• DARTS variants and parameter-sharing approaches show sensitivity to BN handling and training hyper-parameters within this space.
• Non-parameter-sharing NAS methods (REA, RS, REINFORCE, BOHB) can achieve strong rankings with fine-grained training signals and shorter runs.
• The benchmark reveals generally consistent architecture rankings across datasets and highlights transferability challenges when directly porting architectures between datasets.