[Paper Review] NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search

Arber Zela, Julien Siems+1 more|arXiv (Cornell University)|Jan 28, 2020

Advanced Neural Network Applications29 references61 citations

TL;DR

Introduces NAS-Bench-1Shot1, a benchmarking framework that reuses NAS-Bench-101 to cheaply evaluate one-shot NAS methods, enabling analysis of trajectories, hyperparameter sensitivity, and comparisons with black-box optimizers.

ABSTRACT

One-shot neural architecture search (NAS) has played a crucial role in making NAS methods computationally feasible in practice. Nevertheless, there is still a lack of understanding on how these weight-sharing algorithms exactly work due to the many factors controlling the dynamics of the process. In order to allow a scientific study of these components, we introduce a general framework for one-shot NAS that can be instantiated to many recently-introduced variants and introduce a general benchmarking framework that draws on the recent large-scale tabular benchmark NAS-Bench-101 for cheap anytime evaluations of one-shot NAS methods. To showcase the framework, we compare several state-of-the-art one-shot NAS methods, examine how sensitive they are to their hyperparameters and how they can be improved by tuning their hyperparameters, and compare their performance to that of blackbox optimizers for NAS-Bench-101.

Motivation & Objective

• Provide a general benchmarking framework for one-shot NAS that can instantiate recent variants.
• Enable cheap, anytime evaluation of one-shot NAS methods by reusing NAS-Bench-101 computations.
• Compare state-of-the-art one-shot NAS methods and assess hyperparameter sensitivity and potential improvements via tuning.
• Offer a unified codebase to reproduce and fairly compare one-shot NAS components and discrete NAS optimizers.

Proposed method

• Define a mapping between NAS-Bench-101 search space representations and one-shot NAS representations to query discrete architectures found by one-shot methods.
• Construct three NAS-Bench-1Shot1 search spaces with varying complexity (Search Space 1–3) derived from NAS-Bench-101, including a largest space with 363,648 architectures.
• Provide a general framework for implementing and evaluating one-shot NAS methods (e.g., DARTS, GDAS, PC-DARTS, ENAS, Random WS) within a single codebase.
• Track architecture weights across search epochs and query NAS-Bench-101 for test/validation errors to analyze the full trajectory without retraining each architecture.
• Examine correlations between one-shot model rankings and true NAS-Bench-101 performance.
• Assess robustness to hyperparameters and demonstrate tunability via hyperparameter optimization (BOHB).
• Make open-source implementation available to facilitate reproducibility and fair benchmarking.

Experimental results

Research questions

• RQ1 How well do different one-shot NAS methods perform in an anytime evaluation framework when their architectures are tracked against NAS-Bench-101 evaluations?
• RQ2 What is the correlation between one-shot validation performance and true NAS-Bench-101 test performance across search spaces?
• RQ3 How sensitive are one-shot NAS methods to their hyperparameters, and can tuning improve performance relative to discrete NAS optimizers?
• RQ4 Can a unified framework faithfully compare various one-shot NAS variants on a common codebase without confounding factors?
• RQ5 To what extent can hyperparameter optimization reduce overfitting and improve architecture quality across search spaces?

Key findings

• GDAS offers the best anytime performance across the benchmarks but may prematurely converge to suboptimal minima due to temperature annealing.
• DARTS and PC-DARTS show decreasing one-shot validation error that does not always align with decreasing NAS-Bench-101 test error.
• There is little to no correlation between one-shot validation rankings and NAS-Bench-101 test performance for DARTS, PC-DARTS, GDAS, and Random WS, while ENAS shows some correlation in certain spaces.
• Hyperparameter tuning with BOHB substantially improves outcomes, with best configurations surpassing default settings and sometimes outperforming discrete NAS optimizers.
• Tuning reduces search time (e.g., DARTS tuning example from 45 GPU days to 1 day on 16 GPUs) and demonstrates robust configurations across search spaces.
• Random WS and ENAS generally underperform due to weak correlation between one-shot scaling and true architecture performance.
• The framework enables efficient, fair comparisons and reproducing results, and highlights the potential for hyperparameter-driven improvements in one-shot NAS.