[论文解读] Neural Architecture Search: A Survey

Thomas Elsken, Jan Hendrik Metzen+1 more|arXiv (Cornell University)|Aug 16, 2018

Machine Learning and Data Classification参考文献 94被引用 1,401

一句话总结

This survey categorizes neural architecture search (NAS) methods along search space, search strategy, and performance estimation strategies, outlining common designs, algorithms, and future directions. It compares RL, evolutionary, Bayesian, and gradient-based approaches and discusses efficiency techniques for evaluating architectures.

ABSTRACT

Deep Learning has enabled remarkable progress over the last years on a variety of tasks, such as image recognition, speech recognition, and machine translation. One crucial aspect for this progress are novel neural architectures. Currently employed architectures have mostly been developed manually by human experts, which is a time-consuming and error-prone process. Because of this, there is growing interest in automated neural architecture search methods. We provide an overview of existing work in this field of research and categorize them according to three dimensions: search space, search strategy, and performance estimation strategy.

研究动机与目标

• Motivate the study of NAS as a logical step in automating architecture engineering for deep learning.
• Categorize NAS methods by search space, search strategy, and performance estimation strategy.
• Survey historical and current approaches (RL, evolution, BO, gradient-based) and efficiency techniques for performance estimation.

提出的方法

• Review and categorize NAS literature along three dimensions: search space, search strategy, and performance estimation.
• Describe common search spaces (chain-structured, multi-branch with skip connections, and cell-based spaces) and discuss their implications for optimization.
• Summarize diverse search strategies (reinforcement learning, neuro-evolution, Bayesian optimization, gradient-based methods, random/hill-climbing) and their pros/cons.
• Explain performance estimation strategies to reduce computational cost, including low-fidelity proxies, learning curve extrapolation, weight inheritance/morphisms, one-shot models, and their biases.

Figure 1: Abstract illustration of Neural Architecture Search methods. A search strategy selects an architecture $\mathsf{A}$ from a predefined search space $\mathcal{A}$ . The architecture is passed to a performance estimation strategy, which returns the estimated performance of $\mathsf{A}$ to the

实验结果

研究问题

• RQ1 What are the main components and design choices in NAS (search space, search strategy, performance estimation) and how do they interact?
• RQ2 How do different NAS approaches (RL, evolution, BO, gradient-based) compare in terms of efficiency and final performance?
• RQ3 What techniques exist to speed up NAS performance estimation, and what biases do they introduce?
• RQ4 What future directions could broaden NAS beyond image classification and into multi-task, multi-objective, or robust architectures?

主要发现

• NAS methods have outperformed manual architectures on tasks like image classification, object detection, and semantic segmentation in various works.
• Cell-based search spaces reduce search space size and improve transferability across datasets, enabling faster search and reuse of learned designs.
• One-shot and weight-sharing NAS drastically reduce computational resources to a few GPU days but introduce potential biases and correlations that require careful analysis.
• Gradient-based NAS (continuous relaxations) enables efficient optimization but depends on relaxation quality and may require discretization post-optimization.
• RL and evolution offer competitive performance with different trade-offs in anytime performance and architecture sizes; random search is generally outperformed in reported studies.
• There is evidence that lower-fidelity performance estimates speed up search but can bias architecture ranking if fidelity gaps are large.

Figure 3: Illustration of the cell search space. Left: Two different cells, e.g., a normal cell (top) and a reduction cell (bottom) (Zoph et al., 2018 ) . Right: an architecture built by stacking the cells sequentially. Note that cells can also be combined in a more complex manner, such as in multi-