[Paper Review] Neural Architecture Search: Insights from 1000 Papers

Colin White, Mahmoud Safari+6 more|arXiv (Cornell University)|Jan 20, 2023

Machine Learning and Data Classification72 citations

TL;DR

This survey organizes NAS research by taxonomy of search spaces, algorithms, speedups, and resources, highlighting rapid progress and practical guidance after 1000+ papers since 2020.

ABSTRACT

In the past decade, advances in deep learning have resulted in breakthroughs in a variety of areas, including computer vision, natural language understanding, speech recognition, and reinforcement learning. Specialized, high-performing neural architectures are crucial to the success of deep learning in these areas. Neural architecture search (NAS), the process of automating the design of neural architectures for a given task, is an inevitable next step in automating machine learning and has already outpaced the best human-designed architectures on many tasks. In the past few years, research in NAS has been progressing rapidly, with over 1000 papers released since 2020 (Deng and Lindauer, 2021). In this survey, we provide an organized and comprehensive guide to neural architecture search. We give a taxonomy of search spaces, algorithms, and speedup techniques, and we discuss resources such as benchmarks, best practices, other surveys, and open-source libraries.

Motivation & Objective

• Provide a structured, up-to-date guide to neural architecture search (NAS).
• Classify NAS by search spaces, strategies, and performance estimation/speedup techniques.
• Discuss resources, benchmarks, and open-source libraries to promote reproducibility.

Proposed method

• Present a taxonomy of NAS search spaces (macro, chain-structured, cell-based, hierarchical).
• Describe architecture encodings and their impact on NAS performance.
• Survey black-box optimization and one-shot NAS strategies, including baselines and common algorithms.
• Summarize speedup techniques and performance estimation methods to accelerate NAS.
• Discuss extensions, applications, and resources (benchmarks, surveys, libraries).

Experimental results

Research questions

• RQ1 What are the main NAS search spaces and how do they differ in expressiveness and search efficiency?
• RQ2 What are the dominant NAS search strategies and how do they compare in terms of robustness and speed?
• RQ3 What speedup and performance-estimation techniques enable practical NAS at scale?
• RQ4 What resources (benchmarks, libraries, best practices) support reproducible NAS research?
• RQ5 How has NAS evolved across domains beyond image classification?

Key findings

• NAS has rapidly evolved, with over 1000 NAS papers in the last two years, outpacing many human-designed architectures on various tasks.
• One-shot and weight-sharing methods substantially speed up NAS and are increasingly robust, while traditional black-box methods remain valuable for certain settings.
• Cell-based search spaces are the most popular due to efficiency and transferability, though they may limit expressiveness; hierarchical spaces offer greater expressiveness at search cost.
• Benchmarks like NAS-Bench-101 have advanced reproducibility and scientific study in NAS; NAS research has broadened from vision to speech, NLP, and beyond.
• Diverse encoding strategies (graph-based, path-based, learned encodings) significantly influence NAS performance and generalization.