[Paper Review] Stabilizing Differentiable Architecture Search via Perturbation-based Regularization

Xiangning Chen, Cho‐Jui Hsieh|arXiv (Cornell University)|Feb 12, 2020

Adversarial Robustness in Machine LearningComputer Science51 references82 citations

TL;DR

SDARTS introduces perturbation-based regularization (random smoothing and adversarial) to stabilize DARTS, reduces Hessian norm, and improves NAS performance across spaces and datasets.

ABSTRACT

Differentiable architecture search (DARTS) is a prevailing NAS solution to identify architectures. Based on the continuous relaxation of the architecture space, DARTS learns a differentiable architecture weight and largely reduces the search cost. However, its stability has been challenged for yielding deteriorating architectures as the search proceeds. We find that the precipitous validation loss landscape, which leads to a dramatic performance drop when distilling the final architecture, is an essential factor that causes instability. Based on this observation, we propose a perturbation-based regularization - SmoothDARTS (SDARTS), to smooth the loss landscape and improve the generalizability of DARTS-based methods. In particular, our new formulations stabilize DARTS-based methods by either random smoothing or adversarial attack. The search trajectory on NAS-Bench-1Shot1 demonstrates the effectiveness of our approach and due to the improved stability, we achieve performance gain across various search spaces on 4 datasets. Furthermore, we mathematically show that SDARTS implicitly regularizes the Hessian norm of the validation loss, which accounts for a smoother loss landscape and improved performance.

Motivation & Objective

• Motivate instability in DARTS due to sharp validation loss landscapes and discrete projection.
• Propose SDARTS with random smoothing (SDARTS-RS) and adversarial (SDARTS-ADV) formulations to smooth the loss landscape.
• Show that SDARTS implicitly regularizes the Hessian of the validation loss to enhance stability and generalization.
• Demonstrate improved performance of SDARTS on CIFAR-10, ImageNet, and Penn Treebank across multiple search spaces.

Proposed method

• Replace the current architecture weight minimization with a neighborhood-based objective: minimize the training loss over perturbations of the architecture weights.
• SDARTS-RS: w̄(A) = argmin_w E_{δ ~ U([-ε, ε])} L_train(w, A+δ).
• SDARTS-ADV: w̄(A) = argmin_w max_{||δ|| ≤ ε} L_train(w, A+δ).
• Update A by descending ∇_A L_val(w̄(A), A).
• Compute perturbations δ either randomly or via an adversarial PGD procedure (min-max optimization).
• Both variants aim to yield a smoother L_val with respect to A, improving stability and generalization.

Experimental results

Research questions

• RQ1 Can perturbation-based regularization stabilize differentiable architecture search against sharp loss landscapes and projection instability?
• RQ2 Do random smoothing and adversarial perturbations lead to smoother loss landscapes and better generalization in NAS?
• RQ3 Is the Hessian norm of the validation loss implicitly regularized by SDARTS, explaining performance gains?
• RQ4 Do SDARTS variants improve robustness and results across CIFAR-10, ImageNet, and PTB spaces compared to DARTS and other baselines?

Key findings

• SDARTS-RS and SDARTS-ADV produce smoother validation loss landscapes compared to vanilla DARTS, reducing sensitivity to perturbations in architecture weights.
• Both SDARTS variants decrease the Hessian norm (spectral norm) of the validation loss during training, correlating with improved stability.
• SDARTS-RS and SDARTS-ADV outperform DARTS and several regularization baselines across CIFAR-10, CIFAR-100, SVHN, and PTB benchmarks.
• Adapters of SDARTS to PC-DARTS and P-DARTS yield consistent performance gains, and their results on ImageNet transfer are competitive.
• SDARTS-ADV often achieves the best anytime performance, continuing to improve when search epochs extend beyond typical DARTS training.