[Paper Review] Breaking the Curse of Space Explosion: Towards Efficient NAS with Curriculum Search

Yong Guo, Yaofo Chen+5 more|arXiv (Cornell University)|Jul 7, 2020

Advanced Neural Network Applications36 citations

TL;DR

The paper introduces Curriculum Neural Architecture Search (CNAS), which progressively enlarges the NAS search space and uses curriculum learning and an operation warmup strategy to improve sampling efficiency and architecture quality, achieving superior results on CIFAR-10 and ImageNet with reduced search cost.

ABSTRACT

Neural architecture search (NAS) has become an important approach to automatically find effective architectures. To cover all possible good architectures, we need to search in an extremely large search space with billions of candidate architectures. More critically, given a large search space, we may face a very challenging issue of space explosion. However, due to the limitation of computational resources, we can only sample a very small proportion of the architectures, which provides insufficient information for the training. As a result, existing methods may often produce suboptimal architectures. To alleviate this issue, we propose a curriculum search method that starts from a small search space and gradually incorporates the learned knowledge to guide the search in a large space. With the proposed search strategy, our Curriculum Neural Architecture Search (CNAS) method significantly improves the search efficiency and finds better architectures than existing NAS methods. Extensive experiments on CIFAR-10 and ImageNet demonstrate the effectiveness of the proposed method.

Motivation & Objective

• Address the space explosion issue in NAS due to extremely large search spaces.
• Propose a curriculum search strategy that solves a sequence of NAS problems with increasing space sizes.
• Preserve and transfer knowledge from simpler to more complex search spaces to improve sampling accuracy.
• Mitigate training instability when introducing new operations via an operation warmup scheme.

Proposed method

• Represent NAS as a reinforcement learning problem with a controller that samples architectures from a DAG-based cell space.
• Define a progressively growing search space by adding candidate operations rather than increasing node count, to ensure smoother space growth.
• Train a super-network with weight sharing to estimate architecture rewards efficiently.
• Introduce a curriculum training algorithm that optimizes architectures stage-by-stage with an entropy term to encourage exploration.
• Apply an operation warmup where new operations are trained under fixed controllers to ensure fair comparison and stable learning.
• Infer final architectures by sampling multiple candidates from the final stage policy and selecting the best on validation data.

Experimental results

Research questions

• RQ1 Can curriculum learning alleviate NAS space explosion by solving a sequence of progressively harder NAS problems?
• RQ2 Does gradual addition of operations (rather than nodes) yield smoother search space growth and better final architectures?
• RQ3 Does an operation warmup strategy stabilize training when new operations are introduced?
• RQ4 How does CNAS perform compared with state-of-the-art NAS methods on CIFAR-10 and ImageNet?

Key findings

• CNAS consistently outperforms fixed-size NAS baselines and CNAS-Node in experiments on CIFAR-10.
• The curriculum approach enables learning transferable knowledge from small to large search spaces, improving final architecture quality.
• Operation warmup significantly stabilizes training and improves search performance.
• CNAS architectures achieve competitive or superior accuracy with substantially lower search costs compared to several NAS methods.
• Final CIFAR-10 architectures transferred to ImageNet show strong performance under mobile settings.
• Code for CNAS is publicly available.
• Experiments demonstrate CNAS’s superiority across multiple benchmarks and comparisons with state-of-the-art models.