[Paper Review] Single Path One-Shot Neural Architecture Search with Uniform Sampling

Zichao Guo, Xiangyu Zhang+5 more|arXiv (Cornell University)|Mar 31, 2019

Advanced Neural Network Applications58 references246 citations

TL;DR

The paper introduces a single-path one-shot NAS with uniform path sampling to train a stochastic supernet, enabling efficient, flexible architecture search that supports large search spaces and real-world constraints, achieving state-of-the-art results on ImageNet.

ABSTRACT

We revisit the one-shot Neural Architecture Search (NAS) paradigm and analyze its advantages over existing NAS approaches. Existing one-shot method, however, is hard to train and not yet effective on large scale datasets like ImageNet. This work propose a Single Path One-Shot model to address the challenge in the training. Our central idea is to construct a simplified supernet, where all architectures are single paths so that weight co-adaption problem is alleviated. Training is performed by uniform path sampling. All architectures (and their weights) are trained fully and equally. Comprehensive experiments verify that our approach is flexible and effective. It is easy to train and fast to search. It effortlessly supports complex search spaces (e.g., building blocks, channel, mixed-precision quantization) and different search constraints (e.g., FLOPs, latency). It is thus convenient to use for various needs. It achieves start-of-the-art performance on the large dataset ImageNet.

Motivation & Objective

• Renew interest in the one-shot NAS paradigm by addressing training instability and weight coupling issues.
• Propose a single-path supernet with uniform path sampling to decouple architecture search from weight optimization.
• Demonstrate a flexible search framework that supports complex design choices (channels, mixed-precision quantization) and real-world constraints (FLOPs, latency).
• Show state-of-the-art performance on ImageNet in terms of accuracy, memory, and search efficiency.

Proposed method

• Formulate a single-path supernet where each architecture corresponds to one path to reduce weight co-adaptation.
• Train the supernet with a uniform path sampling strategy so every architecture is trained fully and equally.
• Use an evolutionary algorithm for architecture search under hard latency/FLOPs constraints.
• Introduce novel choice blocks for channel number search and mixed-precision quantization search.
• Compare uniform path sampling with path dropout and demonstrate superior stability and performance.

Experimental results

Research questions

• RQ1 Can a single-path, uniformly sampled supernet effectively predict architecture performance without fine-tuning?
• RQ2 Does uniform sampling alleviate weight coupling and training instability compared to path dropout in one-shot NAS?
• RQ3 How well does the approach handle complex search spaces (channels, quantization) and real-world constraints (latency, FLOPs) on large datasets?
• RQ4 Is evolutionary search more effective than random search for selecting architectures within a large space?
• RQ5 What is the comparative efficiency and memory footprint of this method relative to prior NAS methods on ImageNet?

Key findings

• A single-path supernet trained with uniform path sampling is easy to train and fast to search.
• The approach supports rich search spaces including channel count and mixed-precision quantization.
• Evolutionary architecture search outperforms random search in finding high-performing architectures under constraints.
• On ImageNet, the method achieves strong accuracy while satisfying latency/FLOPs constraints and requires less memory during training than some prior methods.
• The method enables multiple constraint-driven searches from the same trained supernet, demonstrating flexibility and efficiency.