[Paper Review] Stabilizing DARTS with Amended Gradient Estimation on Architectural Parameters

Kaifeng Bi, Changping Hu+4 more|arXiv (Cornell University)|Oct 25, 2019

Advanced Neural Network ApplicationsComputer Science46 references45 citations

TL;DR

This paper fixes instability in DARTS by amending the gradient estimation for architectural parameters, reducing the search-retraining gap and enabling much larger search spaces with stable results.

ABSTRACT

DARTS is a popular algorithm for neural architecture search (NAS). Despite its great advantage in search efficiency, DARTS often suffers weak stability, which reflects in the large variation among individual trials as well as the sensitivity to the hyper-parameters of the search process. This paper owes such instability to an optimization gap between the super-network and its sub-networks, namely, improving the validation accuracy of the super-network does not necessarily lead to a higher expectation on the performance of the sampled sub-networks. Then, we point out that the gap is due to the inaccurate estimation of the architectural gradients, based on which we propose an amended estimation method. Mathematically, our method guarantees a bounded error from the true gradients while the original estimation does not. Our approach bridges the gap from two aspects, namely, amending the estimation on the architectural gradients, and unifying the hyper-parameter settings in the search and re-training stages. Experiments on CIFAR10 and ImageNet demonstrate that our approach largely improves search stability and, more importantly, enables DARTS-based approaches to explore much larger search spaces that have not been investigated before.

Motivation & Objective

• Identify causes of instability in differentiable NAS (DARTS) and the gap between super-network validation and sub-network performance.
• Propose an amended gradient estimation for architectural parameters to guarantee bounded error and improved stability.
• Unify search and re-training hyper-parameters to further reduce instability.
• Demonstrate stability and performance gains on CIFAR-10 and ImageNet with larger search spaces.

Proposed method

• Derive the gradient with respect to architectural parameters and identify the inaccurate second-order term in the standard DARTS gradient (g2).
• Replace the problematic inverse-Hessian-based term with an amended approximation g2' that uses the Hessian H but scaled by an amending coefficient eta, ensuring the angle between g2 and g2' is non-obtuse.
• Show that the amended term yields a bounded error relative to the true gradient (angle <= 90 degrees).
• Compute the final architectural gradient by combining g1 (standard) with g2' following second-order DARTS computations.
• Experiment with CIFAR-10 and ImageNet, including consistent hyper-parameter settings between search and retraining to reduce the optimization gap.

Experimental results

Research questions

• RQ1 Does amending the architectural gradient estimation reduce the optimization gap between the super-network and its sub-networks in DARTS?
• RQ2 Can a bounded-error, amended gradient term stabilize search and allow larger, more complex NAS spaces?
• RQ3 How do unified search-retraining hyper-parameters impact the stability and quality of the final architectures?
• RQ4 What are the empirical gains on CIFAR-10 and ImageNet when applying the amended gradient approach?

Key findings

• Amending the architectural gradient (g2') yields more stable search behavior than standard DARTS and improves validation accuracy during search to 91.5% on CIFAR-10 (vs 90.5% for first-order and 91.0% for second-order DARTS).
• With eta = 0.1, the method prevents degeneration into all-skip connections and yields competitive final architectures after convergence.
• Compared to DARTS variants and PC-DARTS, the amended approach achieves lower test error on CIFAR-10 (2.71% with Amended-DARTS S1; 2.60% with Amended-DARTS S2 fixed edges; 2.63% with Amended-DARTS S2 searched edges) versus 6.18% (DARTS first-order) and 5.15% (DARTS second-order).
• Using a larger, more complex search space (S2) with fixed edges reduces CIFAR-10 error further to 2.60% and 2.63% for the two variants, and shows more significant gains when transferred to ImageNet.
• The approach demonstrates stability and competitive accuracy even after long search runs (up to 500 epochs) and enables exploration of large spaces (e.g., 1.9e93 architectures in S2).
• Ablation studies show that removing the amending term or inconsistent hyper-parameters degrades performance (e.g., w/o amending term yields 3.15% error; w/o consistency yields 3.08%).