[Paper Review] EDAS: Efficient and Differentiable Architecture Search

Hyeong Gwon Hong, Pyunghwan Ahn+1 more|arXiv (Cornell University)|Dec 3, 2019

Advanced Neural Network Applications21 references4 citations

TL;DR

EDAS proposes a differentiable architecture search method that samples only one edge per iteration for relaxation, preserving discrete architecture behavior and eliminating performance degradation after pruning. By minimizing relaxation while enabling deeper networks, EDAS achieves state-of-the-art accuracy on CIFAR-10, CIFAR-100, and ImageNet in just 0.125 GPU days, outperforming DARTS and P-DARTS with significantly reduced search cost.

ABSTRACT

Transferrable neural architecture search can be viewed as a binary optimization problem where a single optimal path should be selected among candidate paths in each edge within the repeated cell block of the directed a cyclic graph form. Recently, the field of differentiable architecture search attempts to relax the search problem continuously using a one-shot network that combines all the candidate paths in search space. However, when the one-shot network is pruned to the model in the discrete architecture space by the derivation algorithm, performance is significantly degraded to an almost random estimator. To reduce the quantization error from the heavy use of relaxation, we only sample a single edge to relax the corresponding variable and clamp variables in the other edges to zero or one. By this method, there is no performance drop after pruning the one-shot network by derivation algorithm, due to the preservation of the discrete nature of optimization variables during the search. Furthermore, the minimization of relaxation degree allows searching in a deeper network to discover better performance with remarkable search cost reduction (0.125 GPU days) compared to previous methods. By adding several regularization methods that help explore within the search space, we could obtain the network with notable performances on CIFAR-10, CIFAR-100, and ImageNet.

Motivation & Objective

• To address the performance drop in differentiable architecture search caused by overparameterized one-shot networks and quantization error after pruning.
• To reduce search cost while maintaining or improving accuracy compared to existing one-shot differentiable NAS methods.
• To improve exploration in the search space by dynamically adjusting architecture parameter rankings during training.
• To enable effective architecture search in deeper networks by alleviating memory and optimization challenges.

Proposed method

• EDAS samples only one edge per iteration for relaxation, while clamping all other edges to perform a single operation based on the highest architecture parameter.
• Architecture parameters are updated via gradient descent, but only the selected edge is relaxed, preserving discrete behavior in the rest of the network.
• A sign-inversion strategy (EDAS-inv) is applied in early epochs to fairly distribute learning experience across candidate operations and improve exploration.
• Regularization techniques are applied to enhance search space exploration and prevent premature convergence.
• The one-shot network is trained end-to-end, and the final architecture is derived by selecting the operation with the highest architecture parameter for each edge.
• The method avoids the depth gap issue by limiting computation per edge, enabling scalability to deeper architectures.

Experimental results

Research questions

• RQ1 Can a differentiable architecture search method maintain high performance after pruning by avoiding overparameterization and quantization error?
• RQ2 How does relaxing only a single edge per iteration affect search efficiency and final architecture quality compared to full relaxation in DARTS?
• RQ3 Can improved exploration of the search space through dynamic architecture parameter ranking lead to better performance?
• RQ4 Does reducing relaxation degree allow effective search in deeper network architectures with minimal computational cost?
• RQ5 Can the proposed method achieve state-of-the-art performance on multiple benchmarks (CIFAR-10, CIFAR-100, ImageNet) with a search cost under 0.125 GPU days?

Key findings

• EDAS achieves a test accuracy of 97.32% on CIFAR-10, outperforming DARTS (97.12%) and P-DARTS (97.25%) with significantly reduced search cost.
• On CIFAR-100, EDAS achieves 90.68% top-1 accuracy, demonstrating its effectiveness on more complex datasets where prior methods did not show improvement.
• The search cost is reduced to 0.125 GPU days, which is approximately 1/12th of the cost of DARTS, enabling rapid architecture discovery for new tasks.
• EDAS-inv, the improved variant with sign-inversion in early epochs, shows better exploration and higher performance than standard EDAS, indicating enhanced search dynamics.
• The derived architectures from EDAS show consistent accuracy improvement over training epochs, confirming the algorithm's effective search capability.
• Architecture parameter rankings in EDAS change dynamically during training, unlike in α-max where rankings stabilize early, indicating superior exploration.