[Paper Review] HW-NAS-Bench:Hardware-Aware Neural Architecture Search Benchmark

Chaojian Li, Zhongzhi Yu+7 more|arXiv (Cornell University)|Mar 19, 2021

Advanced Neural Network Applications38 references24 citations

TL;DR

HW-NAS-Bench is the first public dataset for hardware-aware NAS, aggregating measured/estimated hardware-cost data for NAS-Bench-201 and FBNet across six devices to enable accessible, reproducible HW-NAS research.

ABSTRACT

HardWare-aware Neural Architecture Search (HW-NAS) has recently gained tremendous attention by automating the design of DNNs deployed in more resource-constrained daily life devices. Despite its promising performance, developing optimal HW-NAS solutions can be prohibitively challenging as it requires cross-disciplinary knowledge in the algorithm, micro-architecture, and device-specific compilation. First, to determine the hardware-cost to be incorporated into the NAS process, existing works mostly adopt either pre-collected hardware-cost look-up tables or device-specific hardware-cost models. Both of them limit the development of HW-NAS innovations and impose a barrier-to-entry to non-hardware experts. Second, similar to generic NAS, it can be notoriously difficult to benchmark HW-NAS algorithms due to their significant required computational resources and the differences in adopted search spaces, hyperparameters, and hardware devices. To this end, we develop HW-NAS-Bench, the first public dataset for HW-NAS research which aims to democratize HW-NAS research to non-hardware experts and make HW-NAS research more reproducible and accessible. To design HW-NAS-Bench, we carefully collected the measured/estimated hardware performance of all the networks in the search spaces of both NAS-Bench-201 and FBNet, on six hardware devices that fall into three categories (i.e., commercial edge devices, FPGA, and ASIC). Furthermore, we provide a comprehensive analysis of the collected measurements in HW-NAS-Bench to provide insights for HW-NAS research. Finally, we demonstrate exemplary user cases to (1) show that HW-NAS-Bench allows non-hardware experts to perform HW-NAS by simply querying it and (2) verify that dedicated device-specific HW-NAS can indeed lead to optimal accuracy-cost trade-offs. The codes and all collected data are available at https://github.com/RICE-EIC/HW-NAS-Bench.

Motivation & Objective

• Democratize HW-NAS research by lowering hardware expertise requirements.
• Provide a unified, real hardware data-driven benchmark for HW-NAS across multiple search spaces.
• Enable reproducibility and comparison of HW-NAS algorithms on diverse devices.
• Analyze relationships between hardware-cost, energy, latency, and accuracy across architectures.
• Demonstrate practical workflows where non-hardware experts can perform device-specific HW-NAS.

Proposed method

• Construct HW-NAS-Bench by collecting measured/estimated hardware-cost (latency, energy) for all architectures in NAS-Bench-201 and FBNet across six devices.
• Develop a hardware-cost collection pipeline with device-specific compilation and measurement steps.
• Categorize devices into edge commercial GPUs, Raspi, Edge TPU, Pixel 3, ASIC (Eyeriss), and FPGA, and document configurations.
• Analyze correlations between hardware-cost and theoretical metrics (FLOPs, #Params) using Kendall rank correlation.
• Assess cross-device variability and the necessity of device-specific HW-NAS through empirical comparisons.
• Provide example user cases demonstrating querying the dataset and device-specific HW-NAS optimization.

Experimental results

Research questions

• RQ1 How well do traditional hardware-cost proxies (FLOPs, number of parameters) correlate with real hardware-cost (latency, energy) across diverse devices?
• RQ2 How does hardware-cost vary for the same architecture across different hardware devices?
• RQ3 Is device-specific HW-NAS necessary to achieve optimal accuracy-cost trade-offs?
• RQ4 Can HW-NAS researchers (without hardware expertise) leverage a public benchmark to develop effective HW-NAS solutions?
• RQ5 What insights can be drawn about the relationship between accuracy and hardware-cost for NAS-Bench-201 and FBNet architectures?

Key findings

• Real hardware-cost correlates poorly with FLOPs and #Params across several devices, indicating FLOPs-based HW-NAS can be sub-optimal.
• Hardware-cost for the same architectures can differ significantly between devices, sometimes with near-zero correlation.
• Device-specific HW-NAS often yields better accuracy-cost trade-offs than cross-device transfers, underscoring the need for device-tailored optimization.
• HW-NAS-Bench enables non-hardware experts to conduct HW-NAS by querying pre-measured data, reducing barriers to entry.
• Evaluations on NAS-Bench-201 and FBNet spaces show distinct optimal architectures depending on target device, validating device-specific HW-NAS claims.
• The dataset and pipelines facilitate reproducible HW-NAS benchmarking across multiple hardware targets.