[Paper Review] A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions

Pengzhen Ren, Yun Xiao+5 more|arXiv (Cornell University)|Jun 1, 2020

Machine Learning and Data Classification155 references202 citations

TL;DR

This survey reframes NAS by tracing early challenges to modern solutions, proposing a four-pronged optimization framework (modular search space, continuous search, architecture recycling, incomplete training) and outlining future research directions.

ABSTRACT

Deep learning has made breakthroughs and substantial in many fields due to its powerful automatic representation capabilities. It has been proven that neural architecture design is crucial to the feature representation of data and the final performance. However, the design of the neural architecture heavily relies on the researchers' prior knowledge and experience. And due to the limitations of human' inherent knowledge, it is difficult for people to jump out of their original thinking paradigm and design an optimal model. Therefore, an intuitive idea would be to reduce human intervention as much as possible and let the algorithm automatically design the neural architecture. Neural Architecture Search (NAS) is just such a revolutionary algorithm, and the related research work is complicated and rich. Therefore, a comprehensive and systematic survey on the NAS is essential. Previously related surveys have begun to classify existing work mainly based on the key components of NAS: search space, search strategy, and evaluation strategy. While this classification method is more intuitive, it is difficult for readers to grasp the challenges and the landmark work involved. Therefore, in this survey, we provide a new perspective: beginning with an overview of the characteristics of the earliest NAS algorithms, summarizing the problems in these early NAS algorithms, and then providing solutions for subsequent related research work. Besides, we conduct a detailed and comprehensive analysis, comparison, and summary of these works. Finally, we provide some possible future research directions.

Motivation & Objective

• Explain the evolution of Neural Architecture Search (NAS) from early RL/EA methods to modern efficient strategies.
• Identify core challenges in NAS (search space, search efficiency, evaluation) and map them to concrete solutions.
• Propose a unified framework highlighting four major optimization directions and their impact on NAS effectiveness.

Proposed method

• Analyze historical NAS work to identify foundational characteristics and limitations.
• Introduce and synthesize four optimization directions: modular search space, continuous search strategy, neural architecture recycling, and incomplete training.
• Explain how differentiable search (e.g., DARTS) relaxes discrete choices to enable gradient-based optimization.
• Discuss how modular search spaces (cells/blocks) reduce search complexity while maintaining performance.
• Contextualize the evolution of NAS with respect to efficiency, scalability, and broader application domains.

Experimental results

Research questions

• RQ1 What were the main limitations of early NAS approaches, and how have subsequent methods addressed them?
• RQ2 How do modular search spaces and continuous optimization improve NAS efficiency and effectiveness?
• RQ3 What role do neural architecture recycling and incomplete training play in reducing computational burden during NAS?
• RQ4 What future directions and challenges remain for NAS research?

Key findings

• Provides a comprehensive, challenge-to-solution perspective on NAS evolution.
• Highlights four core optimization directions and their practical impact on search efficiency and architecture quality.
• Reviews widespread cell-based search space approaches and their influence on modern NAS design.
• Discusses evaluation challenges and the need for unified benchmarks and future research directions.