[Paper Review] ASAP: Architecture Search, Anneal and Prune

Asaf Noy, Niv Nayman+6 more|arXiv (Cornell University)|Apr 8, 2019

Advanced Neural Network Applications49 references64 citations

TL;DR

ASAP proposes a differentiable, annealable NAS method that gradually prunes inferior operations during search, achieving competitive accuracy with significantly reduced search time.

ABSTRACT

Automatic methods for Neural Architecture Search (NAS) have been shown to produce state-of-the-art network models. Yet, their main drawback is the computational complexity of the search process. As some primal methods optimized over a discrete search space, thousands of days of GPU were required for convergence. A recent approach is based on constructing a differentiable search space that enables gradient-based optimization, which reduces the search time to a few days. While successful, it still includes some noncontinuous steps, e.g., the pruning of many weak connections at once. In this paper, we propose a differentiable search space that allows the annealing of architecture weights, while gradually pruning inferior operations. In this way, the search converges to a single output network in a continuous manner. Experiments on several vision datasets demonstrate the effectiveness of our method with respect to the search cost and accuracy of the achieved model. Specifically, with $0.2$ GPU search days we achieve an error rate of $1.68\%$ on CIFAR-10.

Motivation & Objective

• Reduce neural architecture search time by introducing an annealable, differentiable search space.
• Improve final architecture quality by gradual pruning of weak connections during the search phase.
• Provide theoretical guidance for the annealing schedule and pruning policy to ensure convergence to a strong architecture.
• Demonstrate competitiveness of ASAP against state-of-the-art NAS methods on CIFAR-10 and transferability to other datasets.

Proposed method

• Define a differentiable, annealable search space for mixed operations within NAS cells.
• Use a Gibbs-Boltzmann-like distribution Phi_o(alpha; T) to select operations at each edge, with temperature T that is annealed over time.
• Update architecture weights alpha via gradient descent on validation loss, while updating network weights omega on training loss.
• Gradually prune inferior operations by thresholding Phi_o(alpha; T) with a threshold theta_t that decays over time, enabling continuous pruning.
• Provide a theoretical PAC-style guarantee (0, δ)-PAC) for pruning inferior operations under a chosen schedule (Theorem 2) and a practical exponential annealing schedule.
• Stack learned normal and reduction cells to form the final network, following the DARTS paradigm but with annealing and gradual pruning.

Experimental results

Research questions

• RQ1 Can an annealable, differentiable search space improve NAS efficiency without sacrificing accuracy?
• RQ2 Does gradually pruning connections during search yield faster convergence and better final architectures than end-of-search hard pruning?
• RQ3 How should the annealing schedule and pruning threshold be designed to balance exploration and convergence?
• RQ4 How does ASAP perform on CIFAR-10 and transfer to larger datasets like ImageNet compared to other NAS methods?

Key findings

• ASAP reduces search time to hours and can achieve CIFAR-10 test errors competitive with or better than prior NAS methods (e.g., ASAP-Large: 1.68% test error).
• ASAP-Small, ASAP-Medium, and ASAP-Large achieve 1.99%, 1.75%, and 1.68% test errors on CIFAR-10, respectively, with relatively modest search costs.
• The method enables continuous pruning during search, leading to reduced epoch times and increasing sparsity as search progresses.
• ASAP outperforms several state-of-the-art NAS methods in CIFAR-10 accuracy while maintaining very low search costs (e.g., 0.2 GPU days for certain runs).
• The ASAP architecture, learned on CIFAR-10, transfers effectively to ImageNet and other benchmarks, demonstrating transferability of the searched cells.