[論文レビュー] Understanding and Robustifying Differentiable Architecture Search

Arber Zela, Thomas Elsken+4 more|arXiv (Cornell University)|Sep 20, 2019

Domain Adaptation and Few-Shot Learning参考文献 47被引用数 166

ひとこと要約

この論文は、DARTS が多くの NAS ベンチマークで失敗する理由を、建築パラメータの高い Hessian の曲率と関連付け、曲率と汎化性を結びつけ、早期停止と内側目的関数の正則化を用いた頑健化された DARTS 変種を提案して、複数の探索空間とタスクにおける頑健性を向上させる。

ABSTRACT

Differentiable Architecture Search (DARTS) has attracted a lot of attention due to its simplicity and small search costs achieved by a continuous relaxation and an approximation of the resulting bi-level optimization problem. However, DARTS does not work robustly for new problems: we identify a wide range of search spaces for which DARTS yields degenerate architectures with very poor test performance. We study this failure mode and show that, while DARTS successfully minimizes validation loss, the found solutions generalize poorly when they coincide with high validation loss curvature in the architecture space. We show that by adding one of various types of regularization we can robustify DARTS to find solutions with less curvature and better generalization properties. Based on these observations, we propose several simple variations of DARTS that perform substantially more robustly in practice. Our observations are robust across five search spaces on three image classification tasks and also hold for the very different domains of disparity estimation (a dense regression task) and language modelling.

研究の動機と目的

• Identify NAS benchmarks and search spaces where standard DARTS yields degenerate architectures with poor test performance.
• Characterize the relationship between the Hessian eigenvalues of the validation loss w.r.t. architectural parameters and generalization of discovered architectures.
• Develop practical robustifications of DARTS, including early stopping based on Hessian curvature and regularization of the inner objective, to improve robustness across tasks.
• Demonstrate robustness of the proposed methods across image classification, disparity estimation, and language modeling domains.
• Provide reproducible implementations and scripts to enable adoption of robust DARTS variants.

提案手法

• Analyze 12 NAS benchmarks spanning four search spaces to observe DARTS failures and degenerate architectures.
• Compute the largest eigenvalue of the Hessian of the validation loss with respect to architectural parameters to study curvature-generalization links.
• Propose early stopping for DARTS when the dominant Hessian eigenvalue increases, to avoid sharp minima in architecture space.
• Regularize the inner objective during the DARTS search via data augmentation (Cutout, ScheduledDropPath) and L2 regularization to reduce Hessian curvature.
• Introduce practical robustifications (DARTS-ES, DARTS-ADA, RobustDARTS) that improve robustness without excessive tuning.
• Validate approaches on image classification (CIFAR-10/100, SVHN), disparity estimation, and Penn Treebank language modeling.

実験結果

リサーチクエスチョン

• RQ1 What causes standard DARTS to yield degenerate architectures across diverse NAS benchmarks?
• RQ2 How does the curvature of the architectural-parameter loss landscape relate to the generalization of found architectures?
• RQ3 What simple, practical modifications can make DARTS more robust across tasks and search spaces?

主な発見

• Standard DARTS often selects degenerate architectures dominated by skip connections or harmful operations in several spaces.
• A strong correlation exists between the dominant Hessian eigenvalue of the validation loss with respect to architectural parameters and the final architecture’s test error.
• Early stopping based on Hessian curvature (tracking the dominant eigenvalue) substantially improves robustness and reduces search time.
• Regularizing the inner objective via data augmentation and increased L2 regularization lowers Hessian curvature and improves generalization of found architectures.
• Practical robustifications (DARTS-ES, DARTS-ADA, RobustDARTS) achieve better test performance than standard DARTS or random-search baselines across most benchmarks and tasks.
• RobustDARTS remains competitive with original DARTS on the original spaces while outperforming DARTS on other datasets (e.g., CIFAR-100, SVHN).