[论文解读] Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation

Xin Chen, Lingxi Xie+2 more|arXiv (Cornell University)|Apr 29, 2019

Advanced Neural Network Applications参考文献 37被引用 133

一句话总结

P-DARTS 逐步增加搜索网络的深度，以弥合搜索阶段与评估阶段之间的差距，利用搜索空间近似与正则化来保持效率与稳定性；在 CIFAR-10/100 和 ImageNet 上以极低的搜索成本实现了最先进或有竞争力的结果。

ABSTRACT

Recently, differentiable search methods have made major progress in reducing the computational costs of neural architecture search. However, these approaches often report lower accuracy in evaluating the searched architecture or transferring it to another dataset. This is arguably due to the large gap between the architecture depths in search and evaluation scenarios. In this paper, we present an efficient algorithm which allows the depth of searched architectures to grow gradually during the training procedure. This brings two issues, namely, heavier computational overheads and weaker search stability, which we solve using search space approximation and regularization, respectively. With a significantly reduced search time (~7 hours on a single GPU), our approach achieves state-of-the-art performance on both the proxy dataset (CIFAR10 or CIFAR100) and the target dataset (ImageNet). Code is available at https://github.com/chenxin061/pdarts.

研究动机与目标

• Motivate and address the depth gap between NAS search depth and evaluation depth observed in differentiable NAS methods.
• Introduce a progressive search paradigm that grows depth during search to better match evaluation conditions.
• Develop mechanisms to mitigate computational overhead and instability when increasing depth.
• Demonstrate strong performance and transferability of discovered architectures on CIFAR-10/100 and ImageNet.
• Provide practical insights into memory-efficient search and stability-enhancing regularizations.

提出的方法

• Extend DARTS as a base differentiable NAS framework and define a multi-stage search process with increasing depth.
• Use search space approximation to drop less promising operations at stage transitions, reducing memory/compute as depth grows.
• Incorporate search space regularization: operation-level Dropout to reduce skip-connect dominance and architecture refinement to fix a target number of skip connections.
• Progressively deepen the searched network (5 -> 11 -> 17 cells) and shrink candidate operations (8 -> 5 -> 3 -> 2) across stages.
• Train architecture parameters with gradient-based optimization and use first-order DARTS for efficiency; final cell topology chosen by learned weights plus rules.
• Evaluate the discovered architectures via training from scratch on CIFAR-10/100 and ImageNet (mobile setting) to assess transferability.

实验结果

研究问题

• RQ1 Can progressively increasing the search depth during NAS close the performance gap between search-time and evaluation-time architectures?
• RQ2 Does search space approximation effectively maintain performance while dramatically reducing search memory and time?
• RQ3 Does regularization (Dropout on skip-connects and architecture refinement) stabilize the search process and improve final architecture quality?
• RQ4 How well do architectures discovered on CIFAR-10/100 transfer to ImageNet under mobile settings?

主要发现

• P-DARTS achieves 2.50% test error on CIFAR-10 with 3.4M parameters and 0.3 GPU-days of search on CIFAR-10; transfers to CIFAR-100 with strong performance.
• On CIFAR-10/100, CIFAR-10 discovered architectures transfer effectively to ImageNet under mobile settings, outperforming several baselines including DARTS and SNAS in both accuracy and search efficiency.
• Search space approximation dramatically reduces memory usage and makes multi-stage depth-increasing search feasible on standard GPUs (~7 hours on a P100).
• Regularization, including operation-level Dropout and architecture refinement, stabilizes the search and yields architectures with a controlled number of skip connections, improving final accuracy (e.g., 2.58% CIFAR-10 after progressive stages).
• Compared to shallow-search DARTS, the depth-progressive approach discovers cells with deeper connections, correlating with better evaluation performance.