[Paper Review] PONAS: Progressive One-shot Neural Architecture Search for Very Efficient Deployment

Sian-Yao Huang, Wei-Ta Chu|arXiv (Cornell University)|Mar 11, 2020

Advanced Neural Network Applications26 references4 citations

TL;DR

PONAS proposes a progressive one-shot neural architecture search method that constructs an accuracy table during supernet training to enable sub-second architecture search for hardware-constrained deployment. By storing validation accuracy of candidate blocks per layer, PONAS allows instant lookup for specialized network selection in ~10 seconds, achieving 75.2% top-1 accuracy on ImageNet with minimal search cost.

ABSTRACT

We achieve very efficient deep learning model deployment that designs neural network architectures to fit different hardware constraints. Given a constraint, most neural architecture search (NAS) methods either sample a set of sub-networks according to a pre-trained accuracy predictor, or adopt the evolutionary algorithm to evolve specialized networks from the supernet. Both approaches are time consuming. Here our key idea for very efficient deployment is, when searching the architecture space, constructing a table that stores the validation accuracy of all candidate blocks at all layers. For a stricter hardware constraint, the architecture of a specialized network can be very efficiently determined based on this table by picking the best candidate blocks that yield the least accuracy loss. To accomplish this idea, we propose Progressive One-shot Neural Architecture Search (PONAS) that combines advantages of progressive NAS and one-shot methods. In PONAS, we propose a two-stage training scheme, including the meta training stage and the fine-tuning stage, to make the search process efficient and stable. During search, we evaluate candidate blocks in different layers and construct the accuracy table that is to be used in deployment. Comprehensive experiments verify that PONAS is extremely flexible, and is able to find architecture of a specialized network in around 10 seconds. In ImageNet classification, 75.2% top-1 accuracy can be obtained, which is comparable with the state of the arts.

Motivation & Objective

• To address the high computational cost of existing NAS methods in searching and specializing models for diverse hardware constraints.
• To enable extremely fast deployment of efficient, hardware-optimized neural networks by decoupling search from costly retraining.
• To develop a scalable, flexible NAS framework that supports rapid customization of models for varying FLOPs and latency constraints.
• To improve training stability and efficiency through a two-stage training scheme for the supernet.

Proposed method

• Proposes a two-stage training scheme: meta training to initialize the supernet and fine-tuning to stabilize and improve performance.
• Constructs an accuracy table during search that records validation accuracy of each candidate block at each layer.
• Transforms the accuracy table into an accuracy loss domain to enable cross-layer comparison of block importance.
• Uses table lookup instead of retraining or evolutionary search to quickly evaluate candidate architectures under hardware constraints.
• Employs progressive search by selecting the best block per layer independently, allowing heterogeneous layer designs.
• Leverages the accuracy loss metric to quantify the importance of different layers and guide efficient specialization.

Experimental results

Research questions

• RQ1 Can a one-shot NAS method achieve sub-10-second architecture search for hardware-constrained models?
• RQ2 Can a pre-constructed accuracy table enable accurate and efficient performance prediction for specialized networks?
• RQ3 Does measuring accuracy loss across layers allow effective ranking of block importance and guide better network specialization?
• RQ4 Can a two-stage training strategy improve convergence speed and stability compared to random initialization in supernet training?

Key findings

• PONAS achieves 75.2% top-1 accuracy on ImageNet, matching state-of-the-art performance with significantly reduced search cost.
• Architecture specialization can be completed in approximately 10 seconds using only table lookup, drastically reducing time compared to prior one-shot NAS methods.
• The accuracy loss metric shows a strong Kendall’s τ correlation of 0.733 with actual validation accuracy, validating its reliability for performance prediction.
• Replacing blocks in more important layers (e.g., those with asymmetric channel dimensions) yields greater accuracy gains, confirming the importance metric’s validity.
• The two-stage training scheme leads to faster convergence and greater stability than training from random initialization, as shown in training curve comparisons.
• The worst+most importance model achieves 73.9% top-1 accuracy with fewer FLOPs (295M) than the worst+least importance model (305M), demonstrating the effectiveness of layer importance-aware specialization.