[Paper Review] Random Search and Reproducibility for Neural Architecture Search

Liam Li, Ameet Talwalkar|arXiv (Cornell University)|Feb 20, 2019

Machine Learning and Data Classification47 references253 citations

TL;DR

The paper shows that simple random-search baselines (with early-stopping and weight-sharing) are competitive with state-of-the-art NAS methods on PTB and CIFAR-10, and it analyzes reproducibility issues in NAS research while providing open resources for exact replication.

ABSTRACT

Neural architecture search (NAS) is a promising research direction that has the potential to replace expert-designed networks with learned, task-specific architectures. In this work, in order to help ground the empirical results in this field, we propose new NAS baselines that build off the following observations: (i) NAS is a specialized hyperparameter optimization problem; and (ii) random search is a competitive baseline for hyperparameter optimization. Leveraging these observations, we evaluate both random search with early-stopping and a novel random search with weight-sharing algorithm on two standard NAS benchmarks---PTB and CIFAR-10. Our results show that random search with early-stopping is a competitive NAS baseline, e.g., it performs at least as well as ENAS, a leading NAS method, on both benchmarks. Additionally, random search with weight-sharing outperforms random search with early-stopping, achieving a state-of-the-art NAS result on PTB and a highly competitive result on CIFAR-10. Finally, we explore the existing reproducibility issues of published NAS results. We note the lack of source material needed to exactly reproduce these results, and further discuss the robustness of published results given the various sources of variability in NAS experimental setups. Relatedly, we provide all information (code, random seeds, documentation) needed to exactly reproduce our results, and report our random search with weight-sharing results for each benchmark on multiple runs.

Motivation & Objective

• Ground empirical results in NAS by evaluating simple baselines against leading NAS methods on standard benchmarks (PTB and CIFAR-10).
• Assess whether random search with early-stopping and with weight-sharing can match or exceed state-of-the-art NAS performance within similar compute budgets.
• Identify key meta-hyperparameters that influence NAS performance and demonstrate a simple, reproducible method.
• Open-source all code, seeds, and documentation to enable exact replication of results and study broad reproducibility.

Proposed method

• Frame NAS as a specialized hyperparameter optimization problem and benchmark simple baselines against leading NAS approaches.
• Use random search with early-stopping (ASHA) as a competitive baseline for NAS on CIFAR-10 and PTB.
• Develop a random search with weight-sharing algorithm that trains shared weights across architectures and evaluates multiple sampled architectures.
• Explore meta-hyperparameters (batch size, epochs, network size, number of evaluated architectures, gradient clipping) to improve search quality.
• Provide an exact reproducible experimental setup with fixed seeds and code to enable exact replication, plus repeated runs to study broad reproducibility.

Experimental results

Research questions

• RQ1 How do simple NAS baselines based on random search compare to state-of-the-art NAS methods on standard benchmarks (PTB and CIFAR-10) under similar compute budgets?
• RQ2 Can random search with weight-sharing achieve state-of-the-art or competitive results on PTB and CIFAR-10, and what components drive this performance?
• RQ3 What are the key meta-hyperparameters that most impact random search-based NAS performance?
• RQ4 What are the reproducibility challenges in NAS and how can they be addressed with open materials and multiple runs?

Key findings

• Random search with early-stopping (ASHA) achieves a test perplexity of 56.4 on PTB, comparable to ENAS and better than some other baselines.
• Random search with weight-sharing achieves a test perplexity of 55.5 on PTB, attaining state-of-the-art perplexity among NAS-based results on this benchmark.
• On CIFAR-10, random search with weight-sharing is competitive with DARTS and SNAS, approaching or matching state-of-the-art results under similar budgets.
• The study documents reproducibility gaps in published NAS results due to missing seeds, code, and documentation, and provides open-source materials to enable exact replication.
• Experiments show non-trivial differences across runs for broad reproducibility, highlighting the need for robust reporting and statistics.