[Paper Review] Neural Architecture Search without Training

Joseph Mellor, Jack Turner+2 more|Edinburgh Research Explorer|Jun 8, 2020

Advanced Neural Network Applications41 references44 citations

TL;DR

The paper introduces NASWOT, a training-free neural architecture search method that scores untrained networks using activation pattern overlap, enabling extremely fast architecture search and competitive results across multiple NAS benchmarks.

ABSTRACT

The time and effort involved in hand-designing deep neural networks is immense. This has prompted the development of Neural Architecture Search (NAS) techniques to automate this design. However, NAS algorithms tend to be slow and expensive; they need to train vast numbers of candidate networks to inform the search process. This could be alleviated if we could partially predict a network's trained accuracy from its initial state. In this work, we examine the overlap of activations between datapoints in untrained networks and motivate how this can give a measure which is usefully indicative of a network's trained performance. We incorporate this measure into a simple algorithm that allows us to search for powerful networks without any training in a matter of seconds on a single GPU, and verify its effectiveness on NAS-Bench-101, NAS-Bench-201, NATS-Bench, and Network Design Spaces. Our approach can be readily combined with more expensive search methods; we examine a simple adaptation of regularised evolutionary search. Code for reproducing our experiments is available at https://github.com/BayesWatch/nas-without-training.

Motivation & Objective

• Motivate reducing the training burden in NAS by predicting final performance from initial network state.
• Develop a training-free scoring metric based on activation patterns in untrained networks.
• Demonstrate a fast NAS algorithm (NASWOT) and show how it complements existing search methods.
• Validate the approach across multiple NAS benchmarks (NAS-Bench-101, NAS-Bench-201, NATS-Bench) and Network Design Spaces (NDS).
• Explore integration of the scoring metric into more expensive search strategies (e.g., regularised evolutionary search).

Proposed method

• Define binary activation codes for ReLU units in untrained networks to identify linear regions.
• Construct a kernel matrix K_H from Hamming distances between activation codes.
• Define a training-free score s = log|K_H| indicating diagonal-leaning kernels for higher predicted accuracy.
• Empirically correlate s with final trained accuracy across NAS spaces using Kendall’s Tau.
• Integrate s into a simple NASWOT search by sampling N architectures and selecting the top-scoring one.
• Illustrate AREA (Assisted Regularised EA) by seeding Regularised EA with the NASWOT score to guide population initialization.

Experimental results

Research questions

• RQ1 Can an untrained network’s activation patterns serve as a reliable proxy for its eventual training accuracy?
• RQ2 Is there a robust, data- and initialization-insensitive scoring metric that predicts final performance across diverse NAS spaces?
• RQ3 How fast can NAS search be when relying on training-free signals, and how does it compare to trained baselines?
• RQ4 Can the training-free score be integrated into existing NAS algorithms to improve efficiency without sacrificing accuracy?
• RQ5 Do training-free approaches generalize to different datasets and network design spaces (NAS-Bench-101, NAS-Bench-201, NATS-Bench, NDS)?

Key findings

• A simple score s = log|K_H|, derived from activation-code-based kernels, correlates positively with final accuracy across NAS spaces.
• NASWOT achieves competitive final accuracy (e.g., CIFAR-10) in seconds on a single GPU, outperforming some weight-sharing methods and approaching non-weight-sharing baselines in speed.
• In ablation studies, the score is robust to mini-batch choice, input data type, initialisation, and scales reasonably with mini-batch size.
• The score increases early in training but stabilizes, with preserved network ranking, supporting its role as a predictive proxy.
• AREA demonstrates how NASWOT’s score can seed evolutionary search to improve efficiency over standard REA in NAS-Bench benchmarks.