[Paper Review] Understanding and Robustifying Differentiable Architecture Search

Arber Zela, Thomas Elsken+4 more|arXiv (Cornell University)|Sep 20, 2019

Domain Adaptation and Few-Shot LearningComputer Science47 references166 citations

TL;DR

The paper analyzes why DARTS fails on many NAS benchmarks due to high Hessian curvature in architectural parameters, links curvature to generalization, and proposes robustified DARTS variants with early stopping and inner-objective regularization that improve robustness across multiple search spaces and tasks.

ABSTRACT

Differentiable Architecture Search (DARTS) has attracted a lot of attention due to its simplicity and small search costs achieved by a continuous relaxation and an approximation of the resulting bi-level optimization problem. However, DARTS does not work robustly for new problems: we identify a wide range of search spaces for which DARTS yields degenerate architectures with very poor test performance. We study this failure mode and show that, while DARTS successfully minimizes validation loss, the found solutions generalize poorly when they coincide with high validation loss curvature in the architecture space. We show that by adding one of various types of regularization we can robustify DARTS to find solutions with less curvature and better generalization properties. Based on these observations, we propose several simple variations of DARTS that perform substantially more robustly in practice. Our observations are robust across five search spaces on three image classification tasks and also hold for the very different domains of disparity estimation (a dense regression task) and language modelling.

Motivation & Objective

• Identify NAS benchmarks and search spaces where standard DARTS yields degenerate architectures with poor test performance.
• Characterize the relationship between the Hessian eigenvalues of the validation loss w.r.t. architectural parameters and generalization of discovered architectures.
• Develop practical robustifications of DARTS, including early stopping based on Hessian curvature and regularization of the inner objective, to improve robustness across tasks.
• Demonstrate robustness of the proposed methods across image classification, disparity estimation, and language modeling domains.
• Provide reproducible implementations and scripts to enable adoption of robust DARTS variants.

Proposed method

• Analyze 12 NAS benchmarks spanning four search spaces to observe DARTS failures and degenerate architectures.
• Compute the largest eigenvalue of the Hessian of the validation loss with respect to architectural parameters to study curvature-generalization links.
• Propose early stopping for DARTS when the dominant Hessian eigenvalue increases, to avoid sharp minima in architecture space.
• Regularize the inner objective during the DARTS search via data augmentation (Cutout, ScheduledDropPath) and L2 regularization to reduce Hessian curvature.
• Introduce practical robustifications (DARTS-ES, DARTS-ADA, RobustDARTS) that improve robustness without excessive tuning.
• Validate approaches on image classification (CIFAR-10/100, SVHN), disparity estimation, and Penn Treebank language modeling.

Experimental results

Research questions

• RQ1 What causes standard DARTS to yield degenerate architectures across diverse NAS benchmarks?
• RQ2 How does the curvature of the architectural-parameter loss landscape relate to the generalization of found architectures?
• RQ3 What simple, practical modifications can make DARTS more robust across tasks and search spaces?

Key findings

• Standard DARTS often selects degenerate architectures dominated by skip connections or harmful operations in several spaces.
• A strong correlation exists between the dominant Hessian eigenvalue of the validation loss with respect to architectural parameters and the final architecture’s test error.
• Early stopping based on Hessian curvature (tracking the dominant eigenvalue) substantially improves robustness and reduces search time.
• Regularizing the inner objective via data augmentation and increased L2 regularization lowers Hessian curvature and improves generalization of found architectures.
• Practical robustifications (DARTS-ES, DARTS-ADA, RobustDARTS) achieve better test performance than standard DARTS or random-search baselines across most benchmarks and tasks.
• RobustDARTS remains competitive with original DARTS on the original spaces while outperforming DARTS on other datasets (e.g., CIFAR-100, SVHN).