[Paper Review] Zero-Cost Proxies for Lightweight NAS

Mohamed S. Abdelfattah, Abhinav Mehrotra+2 more|arXiv (Cornell University)|Jan 20, 2021

Advanced Neural Network Applications38 references65 citations

TL;DR

The paper proposes zero-cost proxies based on pruning-at-initialization to score neural networks with a single minibatch, achieving comparable or better ranking than reduced-training proxies and enabling faster NAS across multiple benchmarks and search algorithms.

ABSTRACT

Neural Architecture Search (NAS) is quickly becoming the standard methodology to design neural network models. However, NAS is typically compute-intensive because multiple models need to be evaluated before choosing the best one. To reduce the computational power and time needed, a proxy task is often used for evaluating each model instead of full training. In this paper, we evaluate conventional reduced-training proxies and quantify how well they preserve ranking between multiple models during search when compared with the rankings produced by final trained accuracy. We propose a series of zero-cost proxies, based on recent pruning literature, that use just a single minibatch of training data to compute a model's score. Our zero-cost proxies use 3 orders of magnitude less computation but can match and even outperform conventional proxies. For example, Spearman's rank correlation coefficient between final validation accuracy and our best zero-cost proxy on NAS-Bench-201 is 0.82, compared to 0.61 for EcoNAS (a recently proposed reduced-training proxy). Finally, we use these zero-cost proxies to enhance existing NAS search algorithms such as random search, reinforcement learning, evolutionary search and predictor-based search. For all search methodologies and across three different NAS datasets, we are able to significantly improve sample efficiency, and thereby decrease computation, by using our zero-cost proxies. For example on NAS-Bench-101, we achieved the same accuracy 4$ imes$ quicker than the best previous result. Our code is made public at: https://github.com/mohsaied/zero-cost-nas.

Motivation & Objective

• Motivate reducing NAS compute by evaluating models without full training.
• Adapt pruning-at-initialization saliency metrics to score whole networks for NAS.
• Compare zero-cost proxies with conventional reduced-training proxies on multiple benchmarks.
• Integrate zero-cost proxies into diverse NAS algorithms to improve sample efficiency.

Proposed method

• Adapt per-parameter saliency metrics (snip, grasp, synflow, fisher, jacob_cov) to score entire networks by aggregating parameter-level scores.
• Evaluate proxies using Spearman rank correlation against final trained accuracy across NAS benchmarks.
• Compare zero-cost proxies to EcoNAS proxies on NAS-Bench-201 and larger benchmarks.
• Explore integration strategies for zero-cost proxies in NAS algorithms: zero-cost warmup and zero-cost move proposal across RAND, RL, AE, and predictor-based searches.

Experimental results

Research questions

• RQ1 Can pruning-at-initialization saliency metrics be aggregated to score whole networks for NAS without training?
• RQ2 Do zero-cost proxies preserve model ranking better than conventional reduced-training proxies across NAS benchmarks?
• RQ3 How can zero-cost proxies be integrated into existing NAS search algorithms to improve sample efficiency?
• RQ4 Which zero-cost metric is robust across diverse NAS spaces and tasks?

Key findings

• Zero-cost proxies using a single minibatch can match or exceed the ranking quality of EcoNAS proxies (e.g., synflow consistently performs well).
• On NAS-Bench-201, synflow achieves Spearman ρ > 0.73 across datasets, with jacob_cov also strong, and a majority-vote ensemble (vote) reaching ρ > 0.8.
• Zero-cost proxies enable significant speedups in NAS across four search algorithms and three benchmarks, up to 4× faster on NAS-Bench-101.
• Zero-cost warmup and move-proposal strategies with synflow substantially improve sample efficiency for RAND, RL, AE, and predictor-based searches.
• Synflow maintains consistent top-model ranking across NAS benchmarks, and its warmup/move proposals yield state-of-the-art results on NAS-Bench-101 and NAS-Bench-201.