[Paper Review] Progressive Neural Architecture Search

Chenxi Liu, Barret Zoph+8 more|arXiv (Cornell University)|Dec 2, 2017

Advanced Neural Network Applications41 references166 citations

TL;DR

PNAS introduces a progressive, surrogate-guided search over CNN cell structures, achieving state-of-the-art accuracy with up to 8x faster compute than prior RL-based NAS methods.

ABSTRACT

We propose a new method for learning the structure of convolutional neural networks (CNNs) that is more efficient than recent state-of-the-art methods based on reinforcement learning and evolutionary algorithms. Our approach uses a sequential model-based optimization (SMBO) strategy, in which we search for structures in order of increasing complexity, while simultaneously learning a surrogate model to guide the search through structure space. Direct comparison under the same search space shows that our method is up to 5 times more efficient than the RL method of Zoph et al. (2018) in terms of number of models evaluated, and 8 times faster in terms of total compute. The structures we discover in this way achieve state of the art classification accuracies on CIFAR-10 and ImageNet.

Motivation & Objective

• Motivate and reduce the computational cost of neural architecture search (NAS) for CNNs compared to RL and EA approaches.
• Propose a progressive, block-wise search over CNN cells combined with a surrogate predictor to guide expansion.
• Demonstrate that the discovered architectures reach state-of-the-art accuracy on CIFAR-10 and ImageNet with lower compute.
• Show that sharing a single cell type and progressively increasing complexity improves search efficiency and transferability.

Proposed method

• Define a hierarchical cell-based search space with B blocks per cell and a fixed set of block operations and inputs.
• Perform progressive search from simple (1-block) cells to deeper (B-block) cells by expanding candidates and predicting performance with a surrogate model.
• Train a population of proxy CNNs built from candidate cells to obtain validation accuracy as supervision.
• Train an ensemble surrogate predictor (MLP or RNN) to rank expanded cells and select top-K for the next generation, updating the predictor with new observed data.
• Construct final CNNs by stacking the best cell type with specified repeats and stride patterns, then train on target datasets (CIFAR-10 and ImageNet).
• Compare efficiency to NAS (RL-based) and random search, reporting model counts, total compute, and top validation accuracy.

Experimental results

Research questions

• RQ1 Can progressive, surrogate-guided search reduce the number of evaluated models in NAS while maintaining or improving accuracy?
• RQ2 Does using a cell-based, progressively complex search space improve efficiency and transferability to larger datasets like ImageNet?
• RQ3 How well do surrogate predictors (MLP vs RNN ensembles) rank promising architectures and generalize to larger, unseen cells?
• RQ4 What is the practical speedup and accuracy trade-off of PNAS compared with NAS, random search, and Hierarchical EA on CIFAR-10 and ImageNet?

Key findings

• PNAS is up to 5x more efficient than the RL NAS method of Zoph et al. (2018) in the number of models evaluated.
• PNAS is up to 8x faster in total compute than the RL NAS method for the same search space.
• The architectures discovered by PNAS achieve state-of-the-art or competitive classification accuracies on CIFAR-10 and ImageNet.
• Using a progressive search with surrogate guidance enables exploring larger, more complex cells than direct full-CNN searches.
• An ML-based surrogate ensemble (especially MLP-ensemble) effectively ranks candidate cells for unseen, larger block counts, improving search efficiency.
• PNASNet-5 outperforms several NAS variants in ImageNet (Mobile and Large settings) while using substantially less compute than AmoebaNet implementations.