[Paper Review] Rethinking Architecture Selection in Differentiable NAS

Ruochen Wang, Minhao Cheng+3 more|arXiv (Cornell University)|Aug 10, 2021

Advanced Neural Network ApplicationsComputer Science30 references33 citations

TL;DR

The paper argues that the traditional magnitude-based selection (α) in differentiable NAS can be misleading and introduces a perturbation-based architecture selection (PT) that evaluates each operation’s contribution to the supernet’s performance, yielding consistently better architectures and alleviating robustness issues in DARTS.

ABSTRACT

Differentiable Neural Architecture Search is one of the most popular Neural Architecture Search (NAS) methods for its search efficiency and simplicity, accomplished by jointly optimizing the model weight and architecture parameters in a weight-sharing supernet via gradient-based algorithms. At the end of the search phase, the operations with the largest architecture parameters will be selected to form the final architecture, with the implicit assumption that the values of architecture parameters reflect the operation strength. While much has been discussed about the supernet's optimization, the architecture selection process has received little attention. We provide empirical and theoretical analysis to show that the magnitude of architecture parameters does not necessarily indicate how much the operation contributes to the supernet's performance. We propose an alternative perturbation-based architecture selection that directly measures each operation's influence on the supernet. We re-evaluate several differentiable NAS methods with the proposed architecture selection and find that it is able to extract significantly improved architectures from the underlying supernets consistently. Furthermore, we find that several failure modes of DARTS can be greatly alleviated with the proposed selection method, indicating that much of the poor generalization observed in DARTS can be attributed to the failure of magnitude-based architecture selection rather than entirely the optimization of its supernet.

Motivation & Objective

• Assess whether architecture parameter magnitudes reflect operation strength in differentiable NAS.
• Analyze failure modes of magnitude-based selection (e.g., skip-connection domination).
• Propose and evaluate a perturbation-based architecture selection (PT) that measures each operation’s impact on supernet performance.
• Demonstrate PT’s effectiveness across DARTS, SDARTS, and SGAS and on NAS-Bench-201.

Proposed method

• Define operation strength as discretization accuracy at convergence and show it can be misaligned with α.
• Propose perturbation-based strength measure: remove each operation on an edge and gauge impact on validation accuracy.
• Develop Algorithm 1 (perturbation-based architecture selection) that iterates over edges, selects the best operation by ACC drop, discretizes, and fine-tunes.
• Optionally measure operation importance by removing each operation and observing the ACC drop to reduce computation.
• Apply perturbation-based selection on pretrained supernets of DARTS, SDARTS(rs), and SGAS to derive final architectures.
• Demonstrate that α-free training (uniform α) with PT can match or exceed traditional DARTS performance.

Experimental results

Research questions

• RQ1 Does the magnitude of architecture parameters α reliably indicate the contribution of each operation?
• RQ2 Can a perturbation-based criterion better identify strong operations and stabilize architecture selection across differentiable NAS variants?
• RQ3 How does PT affect robustness issues observed in DARTS and its variants across multiple search spaces?
• RQ4 What is the performance impact of applying PT on CIFAR-10 and NAS-Bench-201 compared to traditional magnitude-based selection?

Key findings

• Perturbation-based selection consistently yields better architectures than magnitude-based selection across DARTS, SDARTS(rs), and SGAS.
• DARTS+PT improves CIFAR-10 test error from 3.00% (DARTS) to 2.61% (avg) and 2.48% (best).
• SDARTS-RS+PT improves to 2.54% (avg) and 2.44% (best) on CIFAR-10.
• On NAS-Bench-201, DARTS baseline reports 45.7% test error, whereas DARTS+PT achieves 11.89% (avg) and 6.20% (PT with fixed α).
• DARTS+PT is able to extract meaningful architectures in spaces where DARTS degenerates (e.g., S2, S4 from Zela et al. 2020).
• Fixing α to uniform weights while using PT yields competitive or better results in some spaces, suggesting α may be unnecessary with PT.