[Paper Review] A Hardware-Aware Framework for Accelerating Neural Architecture Search Across Modalities

Daniel Cummings, Anthony Sarah+4 more|arXiv (Cornell University)|May 19, 2022

Machine Learning and Data Classification4 citations

TL;DR

This paper proposes LINAS, a hardware-aware NAS framework that accelerates multi-objective neural architecture search across modalities by iteratively pairing evolutionary algorithms with lightly trained performance predictors. It achieves faster convergence to Pareto-optimal sub-networks—especially for image classification and recommendation tasks—by reducing validation overhead while maintaining hardware-specific optimization across GPU, CPU, and mobile platforms.

ABSTRACT

Recent advances in Neural Architecture Search (NAS) such as one-shot NAS offer the ability to extract specialized hardware-aware sub-network configurations from a task-specific super-network. While considerable effort has been employed towards improving the first stage, namely, the training of the super-network, the search for derivative high-performing sub-networks is still under-explored. Popular methods decouple the super-network training from the sub-network search and use performance predictors to reduce the computational burden of searching on different hardware platforms. We propose a flexible search framework that automatically and efficiently finds optimal sub-networks that are optimized for different performance metrics and hardware configurations. Specifically, we show how evolutionary algorithms can be paired with lightly trained objective predictors in an iterative cycle to accelerate architecture search in a multi-objective setting for various modalities including machine translation and image classification.

Motivation & Objective

• To address the high computational cost of searching for optimal sub-networks in hardware-aware neural architecture search (NAS), especially when targeting diverse hardware platforms and multiple performance metrics.
• To reduce the reliance on expensive full-validation cycles during NAS by introducing a lightweight predictor-based search loop that accelerates exploration of the architecture space.
• To enable efficient, multi-objective optimization of sub-networks across different modalities—such as image classification, machine translation, and recommendation systems—while remaining hardware-aware.
• To demonstrate that evolutionary algorithms paired with predictive models can outperform standard NAS baselines like random search and NSGA-II in terms of hypervolume and convergence speed.
• To provide a generalizable framework that supports flexible integration of various search algorithms and predictor models across diverse deep learning workloads and hardware configurations.

Proposed method

• The framework uses a pre-trained super-network with weight-sharing to enable efficient sub-network extraction without retraining.
• It employs a two-stage loop: an internal evolutionary algorithm (EA) that explores the architecture space using predictions from a lightweight performance predictor.
• The performance predictor is trained on a small set of sampled sub-networks and predicts key metrics such as accuracy and latency for new architectures.
• The evolutionary algorithm (e.g., NSGA-II, U-NSGA-III) operates on the predicted performance to generate new candidate sub-networks, reducing the need for full validation.
• The process iteratively refines the predictor and evolves the population using multi-objective optimization to converge toward a Pareto-optimal front of sub-networks.
• The framework is modular, allowing interchangeable search algorithms and predictor models, and supports deployment across diverse hardware platforms including GPU, CPU, and mobile devices.

Experimental results

Research questions

• RQ1 Can a lightweight predictor-based evolutionary loop significantly accelerate the search for hardware-optimized sub-networks in a multi-objective NAS setting?
• RQ2 How does the performance of LINAS compare to standard NAS baselines like random search and NSGA-II in terms of hypervolume and convergence speed across different modalities?
• RQ3 To what extent does the choice of evolutionary algorithm or predictor model affect the final quality of the Pareto-optimal sub-network set?
• RQ4 How generalizable is the LINAS framework across different modalities such as image classification, recommendation (NCF), and transformer-based models?
• RQ5 Can LINAS achieve effective hardware-aware optimization without prior knowledge of latency ranges or platform-specific characteristics?

Key findings

• LINAS achieves significantly faster convergence to higher hypervolume values compared to NSGA-II and random search, particularly in image classification tasks (e.g., MobileNetV3 and ResNet50), with a 50% reduction in evaluation count to reach comparable performance.
• The use of Pareto-based multi-objective evolutionary algorithms (e.g., NSGA-II, U-NSGA-III) in the internal loop yields superior performance compared to single-objective methods like MOTPE, which suffer from high evaluation runtimes.
• A population size of 50 in the internal EA loop provides the best trade-off between exploration and convergence speed in the MobileNetV3 search space.
• LINAS demonstrates consistent performance across diverse hardware platforms—including GPU, CPU, and mobile devices—without requiring platform-specific tuning or prior latency knowledge.
• While the benefit is less pronounced in highly constrained search spaces like NCF and Transformers due to a denser distribution of high-performing architectures, LINAS still provides measurable improvement over baselines.
• The framework’s modular design allows for flexible integration of various search algorithms and predictors, with minimal impact on final performance, confirming its generalizability across modalities and hardware targets.