[Paper Review] Efficient Neural Architecture Search via Proximal Iterations

Quanming Yao, Ju Xu+2 more|arXiv (Cornell University)|May 30, 2019

Advanced Neural Network Applications55 references10 citations

TL;DR

NASP proposes a novel differentiable neural architecture search method using proximal iterations to optimize discrete architectures directly, eliminating the need for a supernet and second-order approximations. It achieves over 10x speedup and superior performance compared to state-of-the-art methods like DARTS, while maintaining architectural integrity and model complexity control through a constrained optimization framework with a discrete regularizer.

ABSTRACT

Neural architecture search (NAS) recently attracts much research attention because of its ability to identify better architectures than handcrafted ones. However, many NAS methods, which optimize the search process in a discrete search space, need many GPU days for convergence. Recently, DARTS, which constructs a differentiable search space and then optimizes it by gradient descent, can obtain high-performance architecture and reduces the search time to several days. However, DARTS is still slow as it updates an ensemble of all operations and keeps only one after convergence. Besides, DARTS can converge to inferior architectures due to the strong correlation among operations. In this paper, we propose a new differentiable Neural Architecture Search method based on Proximal gradient descent (denoted as NASP). Different from DARTS, NASP reformulates the search process as an optimization problem with a constraint that only one operation is allowed to be updated during forward and backward propagation. Since the constraint is hard to deal with, we propose a new algorithm inspired by proximal iterations to solve it. Experiments on various tasks demonstrate that NASP can obtain high-performance architectures with 10 times of speedup on the computational time than DARTS.

Motivation & Objective

• To address the high computational cost and performance degradation in existing differentiable NAS methods, particularly those relying on expensive supernet training and second-order approximations.
• To reformulate neural architecture search as a constrained optimization problem that maintains differentiability while enforcing discrete architecture selection during training.
• To develop an efficient optimization algorithm based on proximal iterations that enables direct, stable updates to a single active operation per layer, avoiding correlated operation updates.
• To introduce a regularizer that controls model complexity and enables better generalization and search efficiency.
• To achieve faster convergence and higher performance than state-of-the-art methods like DARTS, especially in large search spaces.

Proposed method

• NASP formulates architecture search as a constrained optimization problem with a discrete constraint on operations and a regularizer on model complexity.
• It uses proximal iterations to solve the non-convex, non-smooth optimization problem, enabling stable and efficient updates.
• The proximal step is derived with a closed-form solution that enforces discrete architecture selection (only one operation active per layer), removing the need for second-order approximations.
• The method avoids training a supernet by directly updating only the selected operation, drastically reducing forward and backward pass costs.
• A lazy-update strategy is employed to maintain architecture parameters in [0,1], improving convergence and stability over standard proximal algorithms.
• The approach decouples operations during training, reducing interference and enabling more effective search in larger search spaces.

Experimental results

Research questions

• RQ1 Can a differentiable search method achieve high performance without training a supernet, thereby reducing computational cost?
• RQ2 Can proximal iterations be effectively adapted to enforce discrete architecture selection while maintaining differentiability and optimization efficiency?
• RQ3 Does removing second-order approximations in architecture search lead to faster convergence and better performance?
• RQ4 Can a regularizer on model complexity improve generalization and search stability in NAS?
• RQ5 How does NASP compare in speed and accuracy to state-of-the-art methods like DARTS and concurrent approaches such as GDAS and BayesNAS?

Key findings

• NASP achieves over 10x speedup compared to DARTS on CIFAR-10, with validation accuracy of 70.4% and test accuracy of 69.5%.
• On ImageNet, NASP achieves 71.2% top-1 accuracy with only 33M parameters, outperforming DARTS (71.2% accuracy but higher parameter count) and achieving faster convergence.
• The architecture parameters in NASP (denoted as $\bar{\mathbf{A}}$) are significantly more stable than in DARTS, as shown in the evolution of operation choices over training epochs.
• NASP outperforms concurrent methods like GDAS and BayesNAS in both search efficiency and performance, with 0.1 GPU days search cost on PTB compared to 0.4 for GDAS.
• The ablation study confirms that direct proximal updates fail due to the discrete constraint, and that the lazy-update strategy is essential for performance, outperforming standard proximal algorithms.
• NASP eliminates the need for second-order approximations because the discrete architecture update stabilizes the dynamics of the architecture parameters, making such approximations unnecessary.