[Paper Review] A Comprehensive Survey on Hardware-Aware Neural Architecture Search

Hadjer Benmeziane, Kaoutar El Maghraoui+4 more|arXiv (Cornell University)|Jan 22, 2021

Advanced Neural Network ApplicationsComputer Science187 references56 citations

TL;DR

This survey reviews hardware-aware NAS (HW-NAS), categorizing methods by search space, strategy, acceleration, and hardware cost estimation, and discusses challenges and future directions.

ABSTRACT

Neural Architecture Search (NAS) methods have been growing in popularity. These techniques have been fundamental to automate and speed up the time consuming and error-prone process of synthesizing novel Deep Learning (DL) architectures. NAS has been extensively studied in the past few years. Arguably their most significant impact has been in image classification and object detection tasks where the state of the art results have been obtained. Despite the significant success achieved to date, applying NAS to real-world problems still poses significant challenges and is not widely practical. In general, the synthesized Convolution Neural Network (CNN) architectures are too complex to be deployed in resource-limited platforms, such as IoT, mobile, and embedded systems. One solution growing in popularity is to use multi-objective optimization algorithms in the NAS search strategy by taking into account execution latency, energy consumption, memory footprint, etc. This kind of NAS, called hardware-aware NAS (HW-NAS), makes searching the most efficient architecture more complicated and opens several questions. In this survey, we provide a detailed review of existing HW-NAS research and categorize them according to four key dimensions: the search space, the search strategy, the acceleration technique, and the hardware cost estimation strategies. We further discuss the challenges and limitations of existing approaches and potential future directions. This is the first survey paper focusing on hardware-aware NAS. We hope it serves as a valuable reference for the various techniques and algorithms discussed and paves the road for future research towards hardware-aware NAS.

Motivation & Objective

• Motivate the need to automate neural architecture design for diverse hardware platforms.
• Define HW-NAS and its importance for resource-constrained deployment.
• Provide a taxonomy of HW-NAS along key dimensions (search space, strategy, acceleration, cost estimation).
• Identify challenges, limitations, and prospective directions to guide future research.

Proposed method

• Classify HW-NAS works according to goals and target platforms.
• Define Architecture Search Space and Hardware Search Space and discuss their roles.
• Formalize HW-NAS as a multi-objective optimization problem (single/multi-objective, constrained).
• Review search strategies including reinforcement learning, evolutionary algorithms, gradient-based and non-differentiable techniques, and acceleration methods.
• Discuss hardware cost estimation methods and metrics, and how they are measured or predicted.
• Survey industrial adoption and practical considerations across hardware platforms.

Experimental results

Research questions

• RQ1 What are the main HW-NAS categories and their goals for different hardware targets?
• RQ2 How do architecture and hardware search spaces, together with search strategies, affect the efficiency and practicality of HW-NAS?
• RQ3 What methods exist for hardware cost estimation in HW-NAS, and what are the challenges in deploying HW-NAS-driven architectures across platforms?

Key findings

• HW-NAS enables balancing model accuracy with hardware efficiency for constrained devices.
• A taxonomy exists for HW-NAS goals: single target fixed configuration, single target multiple configurations, and multiple targets.
• Two core search space types are Architecture Search Space and Hardware Search Space, with various subtypes and trade-offs.
• Hardware cost estimation and measurement are critical, with multiple strategies including runtime measurements, analytical models, and prediction techniques.
• The survey identifies challenges, limitations, and future directions to advance HW-NAS research and deployment.