[论文解读] A Comprehensive Survey on Hardware-Aware Neural Architecture Search

Hadjer Benmeziane, Kaoutar El Maghraoui+4 more|arXiv (Cornell University)|Jan 22, 2021

Advanced Neural Network Applications参考文献 187被引用 56

一句话总结

本综述评估面向硬件的神经网络架构搜索（HW-NAS），按搜索空间、策略、加速以及硬件成本估算对方法进行分类，并讨论挑战与未来方向。

ABSTRACT

Neural Architecture Search (NAS) methods have been growing in popularity. These techniques have been fundamental to automate and speed up the time consuming and error-prone process of synthesizing novel Deep Learning (DL) architectures. NAS has been extensively studied in the past few years. Arguably their most significant impact has been in image classification and object detection tasks where the state of the art results have been obtained. Despite the significant success achieved to date, applying NAS to real-world problems still poses significant challenges and is not widely practical. In general, the synthesized Convolution Neural Network (CNN) architectures are too complex to be deployed in resource-limited platforms, such as IoT, mobile, and embedded systems. One solution growing in popularity is to use multi-objective optimization algorithms in the NAS search strategy by taking into account execution latency, energy consumption, memory footprint, etc. This kind of NAS, called hardware-aware NAS (HW-NAS), makes searching the most efficient architecture more complicated and opens several questions. In this survey, we provide a detailed review of existing HW-NAS research and categorize them according to four key dimensions: the search space, the search strategy, the acceleration technique, and the hardware cost estimation strategies. We further discuss the challenges and limitations of existing approaches and potential future directions. This is the first survey paper focusing on hardware-aware NAS. We hope it serves as a valuable reference for the various techniques and algorithms discussed and paves the road for future research towards hardware-aware NAS.

研究动机与目标

• 推动在多样化硬件平台上实现神经网络架构设计自动化的需求。
• 明确 HW-NAS 的定义及其对资源受限部署的重要性。
• 在关键维度（搜索空间、策略、加速、成本估算）上提供 HW-NAS 的分类体系。
• 识别挑战、局限性及未来研究方向，为后续研究提供指引。

提出的方法

• 根据目标和目标平台对 HW-NAS 工作进行分类。
• 界定 Architecture Search Space 和 Hardware Search Space 及其作用。
• 将 HW-NAS 形式化为一个多目标优化问题（单目标/多目标，带约束）。
• 评审搜索策略，包括强化学习、进化算法、基于梯度与不可微分的技术，以及加速方法。
• 讨论硬件成本估算方法与指标，以及如何进行测量或预测。
• 调查产业界在不同硬件平台上的采用情况与实际考量。

实验结果

研究问题

• RQ1 不同硬件目标的主要 HW-NAS 分类及其目标是什么？
• RQ2 架构搜索空间和硬件搜索空间以及搜索策略如何共同影响 HW-NAS 的效率与实用性？
• RQ3 HW-NAS 中存在哪些硬件成本估算方法，以及在跨平台部署 HW-NAS 驱动的架构时面临哪些挑战？

主要发现

• HW-NAS 使在受限设备上平衡模型精度与硬件效率成为可能。
• 存在 HW-NAS 目标的分类：单一目标固定配置、单一目标多配置、以及多目标。
• 两种核心搜索空间类型是 Architecture Search Space 和 Hardware Search Space，拥有多种子类型及权衡。
• 硬件成本估算与测量至关重要，包含多种策略，包括运行时测量、分析模型和预测技术。
• 本文的调查提出了推进 HW-NAS 研究与部署的挑战、局限性及未来方向。