[Paper Review] Evaluating Efficient Performance Estimators of Neural Architectures

Xuefei Ning, Changcheng Tang+5 more|arXiv (Cornell University)|Aug 7, 2020

Adversarial Robustness in Machine LearningComputer Science68 references30 citations

TL;DR

The paper provides a comprehensive framework to evaluate one-shot estimators (OSEs) and zero-shot estimators (ZSEs) for neural architecture search across multiple NAS benchmarks, analyzes their biases and variances, and offers strategies to improve their reliability.

ABSTRACT

Conducting efficient performance estimations of neural architectures is a major challenge in neural architecture search (NAS). To reduce the architecture training costs in NAS, one-shot estimators (OSEs) amortize the architecture training costs by sharing the parameters of one "supernet" between all architectures. Recently, zero-shot estimators (ZSEs) that involve no training are proposed to further reduce the architecture evaluation cost. Despite the high efficiency of these estimators, the quality of such estimations has not been thoroughly studied. In this paper, we conduct an extensive and organized assessment of OSEs and ZSEs on five NAS benchmarks: NAS-Bench-101/201/301, and NDS ResNet/ResNeXt-A. Specifically, we employ a set of NAS-oriented criteria to study the behavior of OSEs and ZSEs and reveal that they have certain biases and variances. After analyzing how and why the OSE estimations are unsatisfying, we explore how to mitigate the correlation gap of OSEs from several perspectives. Through our analysis, we give out suggestions for future application and development of efficient architecture performance estimators. Furthermore, the analysis framework proposed in our work could be utilized in future research to give a more comprehensive understanding of newly designed architecture performance estimators. All codes are available at https://github.com/walkerning/aw_nas.

Motivation & Objective

• Assess the quality of efficient architecture performance estimators (OSEs and ZSEs) for NAS.
• Identify biases, variances, and failure modes across diverse NAS benchmarks.
• Provide practical strategies to mitigate estimation gaps and improve ranking reliability.

Proposed method

• Review and compare a range of OSEs (one-shot estimators) and ZSEs (zero-shot estimators) on five NAS benchmarks.
• Use NAS-oriented criteria (Pearson LC, Kendall's Tau, SpearmanR, P@topK, BR@K, WR@K) to evaluate estimator quality.
• Analyze biases at complexity-level, operation-level, and architecture-level to explain estimation failures.
• Investigate how training, validation data size, and sampling strategies affect o bservations.
• Propose mitigation techniques including variance reduction, de-isomorphic sampling, and sharing extent reduction.

Experimental results

Research questions

• RQ1 How do OSEs and ZSEs perform in ranking architectures across multiple NAS benchmarks?
• RQ2 What are the main biases and variances affecting OSEs and ZSEs, and why do they occur?
• RQ3 Can proposed strategies mitigate correlation gaps and improve ranking quality in OSEs and ZSEs?
• RQ4 Do one-shot training and input data properties influence the effectiveness of ZSEs and OSEs?
• RQ5 What are practical guidelines for applying efficient estimators in NAS practice?

Key findings

• OSEs generally improve with longer training and larger validation data, but biases toward simpler architectures can persist in early training.
• OSEs show variance due to parameter sharing and forgetting effects, and sampling fairness can influence bias.
• Reducing sharing extent and adopting de-isomorphic sampling can improve ranking quality in OSEs, especially in isomorphic spaces.
• Zero-shot estimators (ZSEs) often underperform compared with simple baselines like #Params and #FLOPs, with architecture-level ZSEs showing space-dependent behavior.
• Among ZSEs, relu_logdet and jacob_cov show relative strengths depending on the search space, but many ZSEs exhibit improper biases toward certain architectural traits.
• One-shot training does not generally enhance ZSEs, and gradient-based ZSEs may deteriorate after training due to gradient magnitudes becoming less informative.