[Paper Review] Single-Path Mobile AutoML: Efficient ConvNet Design and NAS Hyperparameter Optimization

Dimitrios Stamoulis, Ruizhou Ding+5 more|arXiv (Cornell University)|Jul 1, 2019

Advanced Neural Network ApplicationsComputer Science60 references34 citations

TL;DR

This paper proposes Single-Path NAS, a differentiable Neural Architecture Search (NAS) method that reduces search cost to under 3 hours by encoding all candidate operations within a single shared superkernel per layer, eliminating multi-path overhead. It achieves state-of-the-art 75.62% top-1 ImageNet accuracy under ~80ms latency, outperforming prior mobile NAS methods while reducing search cost by up to 5,000× via only 8 epochs of training on TPU-v3.

ABSTRACT

Can we reduce the search cost of Neural Architecture Search (NAS) from days down to only few hours? NAS methods automate the design of Convolutional Networks (ConvNets) under hardware constraints and they have emerged as key components of AutoML frameworks. However, the NAS problem remains challenging due to the combinatorially large design space and the significant search time (at least 200 GPU-hours). In this work, we alleviate the NAS search cost down to less than 3 hours, while achieving state-of-the-art image classification results under mobile latency constraints. We propose a novel differentiable NAS formulation, namely Single-Path NAS, that uses one single-path over-parameterized ConvNet to encode all architectural decisions based on shared convolutional kernel parameters, hence drastically decreasing the search overhead. Single-Path NAS achieves state-of-the-art top-1 ImageNet accuracy (75.62%), hence outperforming existing mobile NAS methods in similar latency settings (~80ms). In particular, we enhance the accuracy-runtime trade-off in differentiable NAS by treating the Squeeze-and-Excitation path as a fully searchable operation with our novel single-path encoding. Our method has an overall cost of only 8 epochs (24 TPU-hours), which is up to 5,000x faster compared to prior work. Moreover, we study how different NAS formulation choices affect the performance of the designed ConvNets. Furthermore, we exploit the efficiency of our method to answer an interesting question: instead of empirically tuning the hyperparameters of the NAS solver (as in prior work), can we automatically find the hyperparameter values that yield the desired accuracy-runtime trade-off? We open-source our entire codebase at: https://github.com/dstamoulis/single-path-nas.

Motivation & Objective

• To drastically reduce the search cost of Neural Architecture Search (NAS) from days to under 3 hours while maintaining high accuracy under mobile latency constraints.
• To improve the accuracy-runtime trade-off in differentiable NAS by treating the Squeeze-and-Excitation path as a fully searchable operation.
• To explore whether NAS hyperparameters can be automatically tuned to achieve a desired accuracy-runtime trade-off, rather than relying on empirical tuning.
• To provide a scalable, efficient framework for NAS-based AutoML services in cloud and mobile environments.

Proposed method

• Proposes a novel single-path differentiable NAS formulation where all candidate operations are encoded within a single shared superkernel per layer, using subset selection of kernel weights instead of multiple parallel paths.
• Introduces a fully searchable Squeeze-and-Excitation (SE) path by integrating it into the single-path superkernel, enabling end-to-end differentiable optimization of attention mechanisms.
• Reduces search cost to only 8 epochs (24 TPU-hours) by avoiding the memory and computation overhead of multi-path supernet training.
• Employs a hyperparameter optimization framework that treats NAS solver hyperparameters (e.g., trade-off weight λ) as variables to be tuned via black-box optimization.
• Uses Bayesian optimization, multi-fidelity optimization, and random search to automatically identify hyperparameters that yield Pareto-optimal models near a target latency of 80ms.
• Automates the NAS pipeline with cloud TPU integration, enabling scalable, parallel, and efficient hyperparameter search across multiple runs.

Experimental results

Research questions

• RQ1 Can we reduce the search cost of NAS from days to under 3 hours while maintaining or improving accuracy under mobile latency constraints?
• RQ2 How does treating the Squeeze-and-Excitation path as a fully searchable operation improve the accuracy-runtime trade-off in differentiable NAS?
• RQ3 Can the hyperparameters of a differentiable NAS solver be automatically tuned to achieve a desired accuracy-runtime trade-off, rather than relying on manual or empirical tuning?
• RQ4 Does low-fidelity function evaluation in hyperparameter optimization provide a representative view of the search space in NAS, or does it lead to suboptimal convergence?

Key findings

• Single-Path NAS achieves a state-of-the-art top-1 ImageNet accuracy of 75.62% under a latency of approximately 80ms on a Pixel 1, outperforming prior mobile NAS methods and manually designed models like MobileNetV3.
• The method reduces the total search cost to only 8 epochs (24 TPU-hours), representing a 5,000× speedup compared to prior NAS methods that require hundreds of GPU-hours.
• Vanilla Bayesian optimization outperforms multi-fidelity Bayesian optimization in hyperparameter tuning, as low-fidelity evaluations do not reliably represent the full objective landscape.
• The use of a single-path superkernel eliminates the need for maintaining multiple paths, reducing memory and computation overhead while maintaining or improving performance.
• Random search shows relatively strong performance in hyperparameter tuning, suggesting that simple sampling methods may be effective in NAS hyperparameter search when combined with efficient search pipelines.
• The study reveals that early, low-cost evaluations in hyperparameter optimization can lead to 'overshooting' beyond the Pareto-optimal region, indicating a risk in approximation-based methods like Hyperband.