[Paper Review] How Powerful are Performance Predictors in Neural Architecture Search?

Colin White, Arber Zela+3 more|arXiv (Cornell University)|Apr 2, 2021

Machine Learning and Data Classification70 references17 citations

TL;DR

This paper presents the first large-scale comparative study of 31 performance predictors in neural architecture search (NAS), evaluating model-based, learning curve extrapolation, zero-cost, and weight-sharing methods across four search spaces and multiple datasets. It finds that predictors from different families are complementary, and combining them significantly improves predictive performance, offering practical recommendations for NAS under varying computational constraints.

ABSTRACT

Early methods in the rapidly developing field of neural architecture search (NAS) required fully training thousands of neural networks. To reduce this extreme computational cost, dozens of techniques have since been proposed to predict the final performance of neural architectures. Despite the success of such performance prediction methods, it is not well-understood how different families of techniques compare to one another, due to the lack of an agreed-upon evaluation metric and optimization for different constraints on the initialization time and query time. In this work, we give the first large-scale study of performance predictors by analyzing 31 techniques ranging from learning curve extrapolation, to weight-sharing, to supervised learning, to "zero-cost" proxies. We test a number of correlation- and rank-based performance measures in a variety of settings, as well as the ability of each technique to speed up predictor-based NAS frameworks. Our results act as recommendations for the best predictors to use in different settings, and we show that certain families of predictors can be combined to achieve even better predictive power, opening up promising research directions. Our code, featuring a library of 31 performance predictors, is available at https://github.com/automl/naslib.

Motivation & Objective

• To compare the performance of diverse performance predictor families in NAS under varying initialization and query time constraints.
• To evaluate the consistency and reliability of predictors across multiple search spaces and datasets, including NAS-Bench-201, NAS-Bench-101, DARTS, and NAS-Bench-NLP.
• To assess how well each predictor accelerates predictor-based NAS frameworks such as Bayesian optimization and evolution strategies.
• To determine whether combining predictors from different families can yield superior predictive performance.
• To provide a reproducible benchmark and open-source library for future research in performance prediction for NAS.

Proposed method

• The study evaluates 31 performance predictors across four search spaces: NAS-Bench-201, NAS-Bench-101, DARTS, and NAS-Bench-NLP, using CIFAR-10, CIFAR-100, ImageNet16-120, and Penn TreeBank.
• Performance is measured using Pearson, Spearman, Kendall Tau, and sparse Kendall Tau correlation metrics on both random and mutation-based architecture sets.
• The authors use a standardized experimental protocol, including hyperparameter tuning and original implementation reuse, to ensure fair comparison across methods.
• Predictors are evaluated under varying initialization and query time budgets to assess trade-offs between setup cost and inference speed.
• The study integrates predictors into Bayesian optimization and predictor-guided evolution workflows to test real-world acceleration potential.
• A comprehensive open-source library of 31 predictors is released via NASLib to enable reproducibility and future benchmarking.

Experimental results

Research questions

• RQ1 How do zero-cost, model-based, learning curve extrapolation, and weight-sharing predictors compare in terms of predictive accuracy across different computational constraints?
• RQ2 Which performance predictors show consistent performance across diverse search spaces and datasets?
• RQ3 Can combining predictors from different families lead to significantly improved predictive power compared to individual methods?
• RQ4 How effective are these predictors in accelerating real NAS frameworks like Bayesian optimization and evolution strategies?
• RQ5 To what extent can published results of performance predictors be reproduced under standardized evaluation conditions?

Key findings

• The study demonstrates that performance predictors from different families—such as zero-cost, model-based, and learning curve extrapolation methods—exhibit complementary strengths and can be combined to achieve significantly higher predictive performance than any single method.
• Zero-cost proxies, particularly those based on gradient and activation statistics (e.g., SNIP, Grad Norm, Fisher), show strong performance with minimal query time, making them ideal for low-latency settings.
• Model-based predictors like MLP and LGBoost achieve high rank correlation (e.g., Spearman >0.7) and are effective when sufficient initialization time is available.
• Learning curve extrapolation methods (e.g., LcSVR, SoTL) achieve high correlation (Spearman >0.93) when given moderate training time, showing strong predictive power with low initialization cost.
• The authors successfully reproduce published results for 15 out of 31 predictors, with differences of less than 0.04 in rank correlation, validating the reproducibility of the benchmark.
• Combining top-performing predictors from different families results in a substantial performance boost, suggesting a promising research direction for future NAS frameworks.