[Paper Review] DropNAS: Grouped Operation Dropout for Differentiable Architecture Search

Weijun Hong, Guilin Li+5 more|arXiv (Cornell University)|Jan 27, 2022

Domain Adaptation and Few-Shot Learning9 references24 citations

TL;DR

DropNAS introduces grouped operation dropout to the differentiable NAS framework (DARTS) to mitigate co-adaptation and the Matthew Effect, achieving state-of-the-art results on CIFAR-10/100 and competitive ImageNet transfer.

ABSTRACT

Neural architecture search (NAS) has shown encouraging results in automating the architecture design. Recently, DARTS relaxes the search process with a differentiable formulation that leverages weight-sharing and SGD where all candidate operations are trained simultaneously. Our empirical results show that such procedure results in the co-adaption problem and Matthew Effect: operations with fewer parameters would be trained maturely earlier. This causes two problems: firstly, the operations with more parameters may never have the chance to express the desired function since those with less have already done the job; secondly, the system will punish those underperforming operations by lowering their architecture parameter, and they will get smaller loss gradients, which causes the Matthew Effect. In this paper, we systematically study these problems and propose a novel grouped operation dropout algorithm named DropNAS to fix the problems with DARTS. Extensive experiments demonstrate that DropNAS solves the above issues and achieves promising performance. Specifically, DropNAS achieves 2.26% test error on CIFAR-10, 16.39% on CIFAR-100 and 23.4% on ImageNet (with the same training hyperparameters as DARTS for a fair comparison). It is also observed that DropNAS is robust across variants of the DARTS search space. Code is available at https://github.com/wiljohnhong/DropNAS.

Motivation & Objective

• Motivate and diagnose the co-adaptation and Matthew Effect problems in weight-sharing differentiable NAS (DARTS).
• Propose Grouped Operation Dropout (DropNAS) to stabilize training and improve architecture parameter learning.
• Show that DropNAS unifies existing differentiable NAS methods under a single framework and yields state-of-the-art results on CIFAR and competitive transfer to ImageNet.

Proposed method

• Analyze co-adaptation by visualizing feature mappings across candidate operations in the DARTS supernet.
• Propose Grouped Operation Dropout that partitions operations into learnable and non-learnable groups and randomly drops a subset during search with a controlled drop path rate.
• Introduce an Alpha-adjust mechanism to prevent passive updates for dropped operations.
• Apply Partial-Decay to restrict weight decay to kept operations during search.
• Train the one-shot model with one-level optimization and use the final epoch to select the architecture while keeping evaluation budgets aligned with DARTS.

Experimental results

Research questions

• RQ1 Can grouped operation dropout alleviate the co-adaptation problem in differentiable NAS frameworks like DARTS?
• RQ2 What drop path rate and grouping strategy yield the best trade-off between stability and search quality across CIFAR and ImageNet-scale tasks?
• RQ3 Do architectures found with DropNAS transfer well to larger datasets (e.g., ImageNet) and across different search spaces?
• RQ4 How do the proposed adjustments (alpha-adjust, partial-decay) contribute to performance and stability? Empirically, how do they compare to standard DARTS, SNAS, and ProxylessNAS?

Key findings

• DropNAS substantially improves/DARTS-like performance on CIFAR-10 and CIFAR-100 compared to one-level DARTS and related methods.
• The best reported drop path rate (r = 3e-5) consistently yields superior stand-alone accuracy and stable supernet training across CIFAR-10 and CIFAR-100.
• Feature clustering analyses show reduced co-adaptation with DropNAS, with parameterized and non-parameterized operation groups behaving more similarly.
• DropNAS architectures transfer to ImageNet with competitive top-1/top-5 results while maintaining reasonable search cost.
• DropNAS is robust to variants of the DARTS search space, maintaining performance when pooling operations or operation sets are modified.
• Ablation studies indicate that alpha-adjust, partial-decay, and grouping each contribute to final performance gains.