[Paper Review] Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation

Xin Chen, Lingxi Xie+2 more|arXiv (Cornell University)|Apr 29, 2019

Advanced Neural Network Applications37 references133 citations

TL;DR

P-DARTS progressively increases the depth of the searched network to bridge the gap between search and evaluation, using search space approximation and regularization to maintain efficiency and stability; it achieves state-of-the-art or competitive results on CIFAR-10/100 and ImageNet with very low search cost.

ABSTRACT

Recently, differentiable search methods have made major progress in reducing the computational costs of neural architecture search. However, these approaches often report lower accuracy in evaluating the searched architecture or transferring it to another dataset. This is arguably due to the large gap between the architecture depths in search and evaluation scenarios. In this paper, we present an efficient algorithm which allows the depth of searched architectures to grow gradually during the training procedure. This brings two issues, namely, heavier computational overheads and weaker search stability, which we solve using search space approximation and regularization, respectively. With a significantly reduced search time (~7 hours on a single GPU), our approach achieves state-of-the-art performance on both the proxy dataset (CIFAR10 or CIFAR100) and the target dataset (ImageNet). Code is available at https://github.com/chenxin061/pdarts.

Motivation & Objective

• Motivate and address the depth gap between NAS search depth and evaluation depth observed in differentiable NAS methods.
• Introduce a progressive search paradigm that grows depth during search to better match evaluation conditions.
• Develop mechanisms to mitigate computational overhead and instability when increasing depth.
• Demonstrate strong performance and transferability of discovered architectures on CIFAR-10/100 and ImageNet.
• Provide practical insights into memory-efficient search and stability-enhancing regularizations.

Proposed method

• Extend DARTS as a base differentiable NAS framework and define a multi-stage search process with increasing depth.
• Use search space approximation to drop less promising operations at stage transitions, reducing memory/compute as depth grows.
• Incorporate search space regularization: operation-level Dropout to reduce skip-connect dominance and architecture refinement to fix a target number of skip connections.
• Progressively deepen the searched network (5 -> 11 -> 17 cells) and shrink candidate operations (8 -> 5 -> 3 -> 2) across stages.
• Train architecture parameters with gradient-based optimization and use first-order DARTS for efficiency; final cell topology chosen by learned weights plus rules.
• Evaluate the discovered architectures via training from scratch on CIFAR-10/100 and ImageNet (mobile setting) to assess transferability.

Experimental results

Research questions

• RQ1 Can progressively increasing the search depth during NAS close the performance gap between search-time and evaluation-time architectures?
• RQ2 Does search space approximation effectively maintain performance while dramatically reducing search memory and time?
• RQ3 Does regularization (Dropout on skip-connects and architecture refinement) stabilize the search process and improve final architecture quality?
• RQ4 How well do architectures discovered on CIFAR-10/100 transfer to ImageNet under mobile settings?

Key findings

• P-DARTS achieves 2.50% test error on CIFAR-10 with 3.4M parameters and 0.3 GPU-days of search on CIFAR-10; transfers to CIFAR-100 with strong performance.
• On CIFAR-10/100, CIFAR-10 discovered architectures transfer effectively to ImageNet under mobile settings, outperforming several baselines including DARTS and SNAS in both accuracy and search efficiency.
• Search space approximation dramatically reduces memory usage and makes multi-stage depth-increasing search feasible on standard GPUs (~7 hours on a P100).
• Regularization, including operation-level Dropout and architecture refinement, stabilizes the search and yields architectures with a controlled number of skip connections, improving final accuracy (e.g., 2.58% CIFAR-10 after progressive stages).
• Compared to shallow-search DARTS, the depth-progressive approach discovers cells with deeper connections, correlating with better evaluation performance.