[Paper Review] HardCoRe-NAS: Hard Constrained diffeRentiable Neural Architecture Search

Niv Nayman, Yonathan Aflalo+2 more|arXiv (Cornell University)|Feb 23, 2021

Advanced Neural Network Applications63 references17 citations

TL;DR

HardCoRe-NAS proposes a differentiable neural architecture search framework that enforces hard latency constraints throughout the search process, using a differentiable search space and a block coordinate stochastic Frank-Wolfe optimization method. It achieves state-of-the-art accuracy on ImageNet while strictly satisfying user-defined latency constraints without post-hoc tuning or accuracy degradation.

ABSTRACT

Realistic use of neural networks often requires adhering to multiple constraints on latency, energy and memory among others. A popular approach to find fitting networks is through constrained Neural Architecture Search (NAS), however, previous methods enforce the constraint only softly. Therefore, the resulting networks do not exactly adhere to the resource constraint and their accuracy is harmed. In this work we resolve this by introducing Hard Constrained diffeRentiable NAS (HardCoRe-NAS), that is based on an accurate formulation of the expected resource requirement and a scalable search method that satisfies the hard constraint throughout the search. Our experiments show that HardCoRe-NAS generates state-of-the-art architectures, surpassing other NAS methods, while strictly satisfying the hard resource constraints without any tuning required.

Motivation & Objective

• To address the limitation of existing constrained NAS methods that enforce resource constraints only softly, leading to constraint violations and accuracy loss.
• To develop a scalable, differentiable NAS method that enforces hard constraints on latency throughout the entire search process, including during discretization.
• To enable efficient and accurate architecture search across diverse hardware platforms with varying latency requirements.
• To maintain high accuracy by preserving pretrained weights in a one-shot model and using knowledge distillation for fast fine-tuning of subnetworks.

Proposed method

• Formulates the NAS problem as a bilevel optimization with hard constraints on expected latency, ensuring strict compliance at all stages.
• Introduces a differentiable search space that supports efficient sampling of subnetworks from a single one-shot model with frozen pretrained weights.
• Employs an accurate analytical formula for estimating the expected latency of each architecture in the search space.
• Uses the block coordinate stochastic Frank-Wolfe (BC-SFW) algorithm to solve the constrained optimization problem, ensuring convergence while maintaining hard constraints.
• Applies a novel projection method for discretizing continuous architecture parameters into discrete architectures, strictly preserving the latency constraint.
• Leverages knowledge distillation from a heavy model to fine-tune extracted subnetworks, reducing training cost to just 15 GPU hours per model.

Experimental results

Research questions

• RQ1 Can a differentiable NAS method enforce hard latency constraints throughout the entire search process, including during architecture discretization?
• RQ2 How does enforcing hard constraints affect the accuracy and constraint satisfaction of the final architecture compared to soft-constrained methods?
• RQ3 Can the proposed method scale efficiently across multiple devices and latency targets with minimal additional training cost?
• RQ4 To what extent does the proposed one-shot training and knowledge distillation scheme reduce the time and cost of generating final models compared to retraining from scratch?
• RQ5 How does the BC-SFW optimization method compare to gradient descent with soft penalties in terms of constraint adherence and convergence speed?

Key findings

• HardCoRe-NAS achieves state-of-the-art top-1 accuracy on ImageNet across all latency levels, outperforming methods like MobileNet V3, TF-NAS, MNAS-NET, and SPNAS.
• The method strictly satisfies the user-defined latency constraint across all tested latency levels (35–60ms), with no violations observed during discretization.
• The argmax projection method commonly used in NAS violates the latency constraint by up to 10%, whereas HardCoRe-NAS’s proposed projection maintains exact compliance.
• The use of knowledge distillation enables model generation in just 15 GPU hours per subnetwork, reducing total cost to 400 + 15N hours, significantly lower than OFA’s 1200 + 25N hours.
• The BC-SFW algorithm converges faster and maintains hard constraints throughout optimization, unlike soft-constrained gradient descent, which is sensitive to hyperparameter tuning and often violates constraints.
• Multi-path sampling improves the correlation between proxy accuracy and final accuracy (ρ = 0.99) and increases the Kendall’s τ by 0.35, indicating more reliable ranking of architectures.