[Paper Review] DARTS-: Robustly Stepping out of Performance Collapse Without Indicators

Xiangxiang Chu, Xiaoxing Wang+4 more|arXiv (Cornell University)|Sep 2, 2020

Software Engineering Research47 references30 citations

TL;DR

DARTS- introduces an auxiliary skip connection to decouple skip-connection advantage from candidate operations, stabilizing differentiable architecture search without reliance on collapse indicators and achieving state-of-the-art robustness with lower search cost.

ABSTRACT

Despite the fast development of differentiable architecture search (DARTS), it suffers from long-standing performance instability, which extremely limits its application. Existing robustifying methods draw clues from the resulting deteriorated behavior instead of finding out its causing factor. Various indicators such as Hessian eigenvalues are proposed as a signal to stop searching before the performance collapses. However, these indicator-based methods tend to easily reject good architectures if the thresholds are inappropriately set, let alone the searching is intrinsically noisy. In this paper, we undertake a more subtle and direct approach to resolve the collapse. We first demonstrate that skip connections have a clear advantage over other candidate operations, where it can easily recover from a disadvantageous state and become dominant. We conjecture that this privilege is causing degenerated performance. Therefore, we propose to factor out this benefit with an auxiliary skip connection, ensuring a fairer competition for all operations. We call this approach DARTS-. Extensive experiments on various datasets verify that it can substantially improve robustness. Our code is available at https://github.com/Meituan-AutoML/DARTS- .

Motivation & Objective

• Motivate and address performance collapse in differentiable architecture search (DARTS) without relying on handcrafted indicators.
• Propose a process-oriented, indicator-free stabilization method using an auxiliary skip connection.
• Demonstrate robustness and efficiency gains across multiple search spaces and datasets.
• Show compatibility of DARTS- with existing DARTS variants and orthogonal improvements.

Proposed method

• Introduce an auxiliary skip connection between every pair of nodes in a cell to separate the stabilization role from the candidate operation’s contribution.
• Decorrelate the auxiliary skip from architecture parameters by using a decayed, fixed auxiliary path during search and a separate, normalized operation weighting
  α for candidate ops.
• Model output on edge (i,j) as ¯o^{(i,j)}(x)=eta x+^{_o^{(i,j)}}/(sum over o in O) o(x), with a decay schedule reducing  to 0, so the final discretization resembles standard DARTS.
• Discuss convergence and robustness through a perturbation-based lens, showing the auxiliary path reduces the influence of skip-connection bias on weight updates.
• Provide an algorithm (DARTS-) that stacks cells with the auxiliary skip and updates architecture weights and weights in alternating steps across search epochs.

Experimental results

Research questions

• RQ1 Can performance collapse in DARTS be resolved without indicators or external hyper-parameter tuning?
• RQ2 Does decoupling the skip-connection advantage via an auxiliary path yield more robust and fair competition among operations during search?
• RQ3 Can DARTS- improve robustness and reduce search cost while remaining compatible with other DARTS variants?
• RQ4 Are indicator-based collapse signals still necessary or reliable after applying DARTS-?
• RQ5 How does DARTS- perform across diverse search spaces and datasets (CIFAR-10/100, ImageNet, NAS-Bench-201) and in transfer tasks?

Key findings

• DARTS- substantially improves robustness across seven search spaces and three datasets.
• DARTS- achieves state-of-the-art results on several spaces with about 3x reduction in search cost relative to R-DARTS.
• On CIFAR-10, DARTS- yields an average top-1 accuracy of 97.5% and stable results across multiple runs.
• On CIFAR-10/100, DARTS- finds competitive models with higher robustness and lower variability than baselines.
• Direct results on ImageNet show DARTS- derived models achieving 76.2% top-1 without extra task-specific training tricks, and up to 77.8% with augmentation and SE modules.
• NAS-Bench-201 evaluation indicates DARTS- sets new state-of-the-art across multiple runs.
• The method can be combined with P-DARTS and PC-DARTS to remove handcrafted priors and still improve performance (e.g., CIFAR-10 gains of ~0.8%).
• Ablation studies show the method’s performance is robust to decay strategies and initial , and that the auxiliary path is essential for stability.