[Paper Review] Sample-Efficient Neural Architecture Search by Learning Action Space

Linnan Wang, Saining Xie+3 more|arXiv (Cornell University)|Jun 17, 2019

Machine Learning and Data ClassificationComputer Science66 references37 citations

TL;DR

LaNAS learns latent actions to partition the NAS search space, enabling a Monte Carlo Tree Search that is significantly more sample-efficient than baselines and achieves state-of-the-art accuracy with far fewer evaluations.

ABSTRACT

Neural Architecture Search (NAS) has emerged as a promising technique for automatic neural network design. However, existing MCTS based NAS approaches often utilize manually designed action space, which is not directly related to the performance metric to be optimized (e.g., accuracy), leading to sample-inefficient explorations of architectures. To improve the sample efficiency, this paper proposes Latent Action Neural Architecture Search (LaNAS), which learns actions to recursively partition the search space into good or bad regions that contain networks with similar performance metrics. During the search phase, as different action sequences lead to regions with different performance, the search efficiency can be significantly improved by biasing towards the good regions. On three NAS tasks, empirical results demonstrate that LaNAS is at least an order more sample efficient than baseline methods including evolutionary algorithms, Bayesian optimizations, and random search. When applied in practice, both one-shot and regular LaNAS consistently outperform existing results. Particularly, LaNAS achieves 99.0% accuracy on CIFAR-10 and 80.8% top1 accuracy at 600 MFLOPS on ImageNet in only 800 samples, significantly outperforming AmoebaNet with 33x fewer samples. Our code is publicly available at https://github.com/facebookresearch/LaMCTS.

Motivation & Objective

• Motivate the importance of action space design in NAS and its impact on search efficiency.
• Propose a method to learn latent actions that partition the search space into high- and low-performing regions.
• Integrate learned actions with Monte Carlo Tree Search to efficiently sample architectures.
• Evaluate LaNAS on diverse NAS benchmarks and compare with baseline search methods.

Proposed method

• Model the NAS search space partition at each node with a linear regressor that defines a latent action.
• Recursively partition the space into good and bad regions to form a hierarchical tree.
• Use Monte Carlo Tree Search with UCB to navigate leaves and sample architectures from partitioned regions.
• Evaluate architectures via training from scratch or via a one-shot supernet with masking to estimate performance.
• Optionally integrate with a one-shot NAS supernet to transform to specific architectures by masking.

Experimental results

Research questions

• RQ1 Can learning latent actions to partition the NAS space improve search efficiency for MCTS?
• RQ2 How does LaNAS perform compared to evolutionary, Bayesian, random search, and MCTS baselines across NAS benchmarks?
• RQ3 What is the impact of tree height, initialization samples, and UCB hyper-parameters on search performance?
• RQ4 How does integrating with one-shot NAS affect evaluation cost and final performance?

Key findings

• LaNAS achieves much higher sample efficiency than baseline methods across NAS tasks.
• On CIFAR-10, LaNAS reaches 99.0% accuracy with 800 samples.
• On ImageNet (600 MFLOPS mobile setting), LaNAS reaches 80.8% top-1 accuracy with 800 samples.
• LaNAS requires about 33x fewer samples than AmoebaNet to reach competitive performance.
• LaNAS consistently outperforms baselines and scales to large GPU resources in practice for NAS.