[论文解读] A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions

Pengzhen Ren, Yun Xiao+5 more|arXiv (Cornell University)|Jun 1, 2020

Machine Learning and Data Classification参考文献 155被引用 202

一句话总结

本综述通过追溯早期挑战至现代解决方案来重新定义神经架构搜索（NAS），提出四点优化框架（模块化搜索空间、连续搜索、架构循环再用、不完全训练），并概述未来研究方向。

ABSTRACT

Deep learning has made breakthroughs and substantial in many fields due to its powerful automatic representation capabilities. It has been proven that neural architecture design is crucial to the feature representation of data and the final performance. However, the design of the neural architecture heavily relies on the researchers' prior knowledge and experience. And due to the limitations of human' inherent knowledge, it is difficult for people to jump out of their original thinking paradigm and design an optimal model. Therefore, an intuitive idea would be to reduce human intervention as much as possible and let the algorithm automatically design the neural architecture. Neural Architecture Search (NAS) is just such a revolutionary algorithm, and the related research work is complicated and rich. Therefore, a comprehensive and systematic survey on the NAS is essential. Previously related surveys have begun to classify existing work mainly based on the key components of NAS: search space, search strategy, and evaluation strategy. While this classification method is more intuitive, it is difficult for readers to grasp the challenges and the landmark work involved. Therefore, in this survey, we provide a new perspective: beginning with an overview of the characteristics of the earliest NAS algorithms, summarizing the problems in these early NAS algorithms, and then providing solutions for subsequent related research work. Besides, we conduct a detailed and comprehensive analysis, comparison, and summary of these works. Finally, we provide some possible future research directions.

研究动机与目标

• 阐明神经架构搜索（NAS）从早期的强化学习/进化算法方法发展到现代高效策略的演变。
• 识别 NAS 的核心挑战（搜索空间、搜索效率、评估），并将其映射到具体解决方案。
• 提出一个统一框架，突出四大优化方向及其对 NAS 效果的影响。

提出的方法

• 分析历史 NAS 工作以识别基础特征及局限性。
• 提出并综合四大优化方向：模块化搜索空间、连续搜索策略、神经架构回收/再利用、以及不完全训练。
• 解释可微分搜索（如 DARTS）如何放松离散选择，以实现基于梯度的优化。
• 讨论模块化搜索空间（单元/模块）如何在保持性能的同时降低搜索复杂度。
• 将 NAS 的演变置于效率、可扩展性及更广泛应用领域的背景下进行讨论。

实验结果

研究问题

• RQ1 早期 NAS 方法的主要局限性是什么，后续方法如何应对这些问题？
• RQ2 模块化搜索空间与连续优化如何提升 NAS 的效率与效果？
• RQ3 神经架构回收/再利用和不完全训练在降低 NAS 计算负担方面扮演何种角色？
• RQ4 NAS 研究尚存的未来方向与挑战有哪些？

主要发现

• 提供了一个从挑战到解决方案的全面视角，聚焦 NAS 的演变。
• 突出四大核心优化方向及其对搜索效率与架构质量的实际影响。
• 回顾广泛的基于单元/单元格的搜索空间方法及其对现代 NAS 设计的影响。
• 讨论评估挑战以及对统一基准和未来研究方向的需求。