[Paper Review] BRP-NAS: Prediction-based NAS using GCNs

Łukasz Dudziak, Thomas Chau+4 more|arXiv (Cornell University)|Jul 16, 2020

Advanced Memory and Neural Computing82 citations

TL;DR

BRP-NAS proposes a graph convolutional network (GCN)-based latency predictor for hardware-aware neural architecture search (NAS), improving sample efficiency by using binary relation learning and iterative data selection. It outperforms prior methods on NAS-Bench-101 and NAS-Bench-201, and introduces LatBench—a large-scale latency benchmark across diverse devices.

ABSTRACT

Neural architecture search (NAS) enables researchers to automatically explore broad design spaces in order to improve efficiency of neural networks. This efficiency is especially important in the case of on-device deployment, where improvements in accuracy should be balanced out with computational demands of a model. In practice, performance metrics of model are computationally expensive to obtain. Previous work uses a proxy (e.g., number of operations) or a layer-wise measurement of neural network layers to estimate end-to-end hardware performance but the imprecise prediction diminishes the quality of NAS. To address this problem, we propose BRP-NAS, an efficient hardware-aware NAS enabled by an accurate performance predictor-based on graph convolutional network (GCN). What is more, we investigate prediction quality on different metrics and show that sample efficiency of the predictor-based NAS can be improved by considering binary relations of models and an iterative data selection strategy. We show that our proposed method outperforms all prior methods on NAS-Bench-101 and NAS-Bench-201, and that our predictor can consistently learn to extract useful features from the DARTS search space, improving upon the second-order baseline. Finally, to raise awareness of the fact that accurate latency estimation is not a trivial task, we release LatBench -- a latency dataset of NAS-Bench-201 models running on a broad range of devices.

Motivation & Objective

• Address the poor accuracy of existing latency predictors in NAS, especially proxy and layer-wise methods.
• Improve sample efficiency in NAS by reducing the number of models that need to be trained to find optimal architectures.
• Develop a more reliable end-to-end latency predictor that generalizes across diverse hardware platforms.
• Enable accurate performance prediction without relying on FLOPS or model size as proxies.
• Promote reproducibility in NAS research by releasing LatBench, a large-scale latency measurement dataset across multiple devices.

Proposed method

• Use a graph convolutional network (GCN) to encode neural architectures as directed graphs, capturing structural dependencies.
• Replace absolute accuracy prediction with a binary relation predictor that ranks two architectures without estimating exact accuracy.
• Implement an iterative data selection strategy that prioritizes high-performing models for predictor training to improve top-K ranking correlation.
• Leverage transfer learning to boost predictor performance across different search spaces.
• Train the GCN-based predictor on a combination of top-K ranked models and randomly sampled models to balance exploration and exploitation.
• Release Eagle, a tool for measuring and predicting model performance across diverse hardware platforms, and LatBench, a latency dataset from real device measurements.

Experimental results

Research questions

• RQ1 Can a GCN-based end-to-end latency predictor outperform proxy metrics and layer-wise estimators in NAS?
• RQ2 Does binary relation learning for accuracy prediction improve sample efficiency compared to absolute accuracy regression?
• RQ3 How does iterative data selection focusing on top-performing models enhance predictor generalization and NAS performance?
• RQ4 To what extent do latency estimates vary across different hardware platforms, and is a single predictor sufficient?
• RQ5 Can a large-scale, real-device latency benchmark improve the reliability and reproducibility of NAS research?

Key findings

• The GCN-based latency predictor significantly outperforms proxy and layer-wise methods on all tested devices, including desktop CPUs, GPUs, embedded GPUs, TPUs, and mobile DSPs.
• BRP-NAS achieves a test accuracy of 94.22% on CIFAR-10 using only 140 trained models on NAS-Bench-101, outperforming prior predictor-based NAS methods.
• The binary relation predictor improves top-K ranking correlation between predicted and ground-truth accuracy, enhancing sample efficiency in NAS.
• LatBench, the released latency benchmark, shows low correlation (r < 0.8) between FLOPS/parameters and actual latency across devices, validating the need for device-specific predictors.
• The predictor generalizes well across diverse hardware, with latency correlation between devices ranging from 0.548 (TPU vs. mobile GPU) to 0.997 (CPU vs. GPU), indicating strong device-specific variation.
• Iterative data selection and binary relation learning reduce the number of required training runs, with BRP-NAS showing increasing performance gains over random search beyond the first iteration on the DARTS search space.