[Paper Review] Noisy Differentiable Architecture Search

Xiangxiang Chu, Bo Zhang|arXiv (Cornell University)|May 7, 2020

Semantic Web and Ontologies61 references37 citations

TL;DR

NoisyDARTS injects unbiased Gaussian noise into skip connections during differentiable architecture search to suppress the unfair advantage of skip paths, smoothing the loss landscape and enabling state-of-the-art NAS results with lower search costs.

ABSTRACT

Simplicity is the ultimate sophistication. Differentiable Architecture Search (DARTS) has now become one of the mainstream paradigms of neural architecture search. However, it largely suffers from the well-known performance collapse issue due to the aggregation of skip connections. It is thought to have overly benefited from the residual structure which accelerates the information flow. To weaken this impact, we propose to inject unbiased random noise to impede the flow. We name this novel approach NoisyDARTS. In effect, a network optimizer should perceive this difficulty at each training step and refrain from overshooting, especially on skip connections. In the long run, since we add no bias to the gradient in terms of expectation, it is still likely to converge to the right solution area. We also prove that the injected noise plays a role in smoothing the loss landscape, which makes the optimization easier. Our method features extreme simplicity and acts as a new strong baseline. We perform extensive experiments across various search spaces, datasets, and tasks, where we robustly achieve state-of-the-art results. Our code is available at https://github.com/xiaomi-automl/NoisyDARTS.

Motivation & Objective

• Motivate and address the performance collapse in Differentiable Architecture Search (DARTS) caused by overreliance on skip connections.
• Propose a simple, unbiased noise injection strategy to regularize the gradient flow and stabilize architecture search.
• Theoretically justify the unbiased noise design and its effect on loss landscape smoothing.
• Empirically validate NoisyDARTS across multiple search spaces, datasets, and tasks, achieving state-of-the-art results with reduced search costs.

Proposed method

• Inject additive Gaussian noise into the outputs of skip connections (OFS: Only For Skip; also a broader NFA: Noise For All, option) to regularize gradient flow among candidate operations.
• Formulate the noise as z ~ N(μ, σ^2) added to intermediate features, ensuring E[z] = 0 for an unbiased gradient.
• Maintain the original softmax-based architectural weights α and the DARTS optimization framework, so the search objective remains: minimize the validation loss over α and w.
• Derive design principles for the injected noise: unbiasedness is crucial, variance σ^2 controls exploration and regularization strength, and noise types are largely interchangeable.
• Provide theoretical intuition showing the noise smooths the loss landscape, approximating the noisy objective by a second-order expansion that favors better-winnable operations.

Experimental results

Research questions

• RQ1 Does unbiased additive noise injected into skip connections mitigate the disproportionate advantage of skip paths in DARTS?
• RQ2 Can NoisyDARTS achieve robust and state-of-the-art NAS performance across diverse search spaces and datasets with lower search cost?
• RQ3 How do the unbiasedness and variance of the injected noise influence optimization dynamics and final architecture quality?
• RQ4 Is the NoisyDARTS approach effective when applied to reduced/robust search spaces compared to standard DARTS variants?
• RQ5 How does NoisyDARTS compare to alternative noise-regularization methods (e.g., SDARTS) in terms of efficiency and performance?

Key findings

• NoisyDARTS eliminates the unfair advantage of skip connections, preventing collapse and enabling competitive models.
• Across multiple benchmarks and spaces, NoisyDARTS achieves state-of-the-art results with substantially lower search cost than RobustDARTS.
• Using zero-mean unbiased noise yields robust performance improvements over both CIFAR-10 and ImageNet evaluations.
• Compared with biased noise (or dropout-like alternatives), unbiased noise consistently outperforms in mitigating gradient bias and improving search stability.
• Ablation shows that unbiased, zero-mean noise is essential; biased noise degrades performance and can misguide optimization.