[Paper Review] Bag of Tricks for Neural Architecture Search

Thomas Elsken, Benedikt Staffler+3 more|arXiv (Cornell University)|Jul 8, 2021

Advanced Neural Network Applications57 references4 citations

TL;DR

This paper presents a comprehensive set of practical techniques—such as weight warm-up, normalization layer adjustments, regularization, and proxy training—to improve the stability, efficiency, and final performance of neural architecture search (NAS) methods. The authors demonstrate that these 'bag of tricks' significantly enhance search reliability and accuracy, often outperforming random search and reducing sensitivity to hyperparameters.

ABSTRACT

While neural architecture search methods have been successful in previous years and led to new state-of-the-art performance on various problems, they have also been criticized for being unstable, being highly sensitive with respect to their hyperparameters, and often not performing better than random search. To shed some light on this issue, we discuss some practical considerations that help improve the stability, efficiency and overall performance.

Motivation & Objective

• To address the instability and hyperparameter sensitivity commonly reported in neural architecture search (NAS) methods.
• To improve the reliability and reproducibility of NAS by identifying and sharing widely used but often hidden practical techniques.
• To reduce the performance gap between proxy tasks and real evaluation by enhancing correlation through architectural selection and training pipeline optimization.
• To demonstrate that training pipeline choices (e.g., data augmentation, regularization) can have a larger impact on final accuracy than architecture design itself.

Proposed method

• Implement weight warm-up by training network weights for a significant portion of the search duration before updating architectural parameters, preventing premature convergence.
• Disable learnable parameters in normalization layers (e.g., batch normalization) during architecture search to avoid destabilizing architectural weight updates.
• Apply strong regularization techniques such as drop path, weight decay, and data augmentation to smooth the loss landscape and improve search stability.
• Use lower-fidelity proxy tasks (e.g., smaller datasets, fewer epochs, reduced model depth) to accelerate NAS while maintaining correlation with full-accuracy performance.
• Cache features from fixed backbones in multi-component architectures (e.g., semantic segmentation, object detection) to speed up evaluation during search.
• Extract final discrete architectures from continuous relaxation using performance-aware methods, such as minimizing accuracy drop upon operation removal, rather than simply selecting max-weight operations.

Experimental results

Research questions

• RQ1 How can the instability and hyperparameter sensitivity of gradient-based NAS methods be mitigated in practice?
• RQ2 To what extent do regularization and normalization techniques affect the reliability of architecture search?
• RQ3 How do proxy training tasks compare to full-accuracy evaluation in terms of ranking consistency for searched architectures?
• RQ4 What role do training pipeline factors (e.g., data augmentation, learning rate schedules) play in final model performance compared to architecture choice?
• RQ5 Can architectural performance be improved by re-evaluating candidate architectures with retrained weights or sub-one-shot models?

Key findings

• Weight warm-up significantly improves search stability by preventing architectures that train faster (e.g., those with many skip connections) from dominating the search.
• Batch normalization is often harmful in NAS and should be disabled or replaced with alternatives like synchronized or group normalization to improve training stability.
• Stronger regularization, such as drop path and data augmentation, smooths the loss landscape and enhances search convergence and performance.
• The training pipeline—including data augmentation and learning rate schedules—can have a larger impact on final accuracy than architectural design, with performance differences of up to 3.2% on ImageNet when pipelines are not matched.
• When trained with identical state-of-the-art pipelines, MobileNetV3’s accuracy gain over MobileNetV2 drops from 3.2% to 2.0%, highlighting the importance of fair comparison.
• Using sub-one-shot models to alleviate co-adaptation issues improves the correlation between one-shot model rankings and final retrained architecture performance.