[Paper Review] Rethinking Performance Estimation in Neural Architecture Search

Xiawu Zheng, Rongrong Ji+5 more|arXiv (Cornell University)|May 20, 2020

Advanced Neural Network Applications49 references4 citations

TL;DR

This paper proposes Budgeted Performance Estimation (BPE) and Minimum Importance Pruning (MIP) to accelerate Neural Architecture Search (NAS) by systematically identifying and prioritizing the most influential hyper-parameters in performance estimation. By using random forest to rank hyper-parameter importance and iteratively pruning the least critical ones, MIP achieves up to 1,000× speedup in NAS with negligible accuracy drop, enabling efficient search across RL, evolution, random search, and differentiable NAS methods.

ABSTRACT

Neural architecture search (NAS) remains a challenging problem, which is attributed to the indispensable and time-consuming component of performance estimation (PE). In this paper, we provide a novel yet systematic rethinking of PE in a resource constrained regime, termed budgeted PE (BPE), which precisely and effectively estimates the performance of an architecture sampled from an architecture space. Since searching an optimal BPE is extremely time-consuming as it requires to train a large number of networks for evaluation, we propose a Minimum Importance Pruning (MIP) approach. Given a dataset and a BPE search space, MIP estimates the importance of hyper-parameters using random forest and subsequently prunes the minimum one from the next iteration. In this way, MIP effectively prunes less important hyper-parameters to allocate more computational resource on more important ones, thus achieving an effective exploration. By combining BPE with various search algorithms including reinforcement learning, evolution algorithm, random search, and differentiable architecture search, we achieve 1, 000x of NAS speed up with a negligible performance drop comparing to the SOTA

Motivation & Objective

• To address the major bottleneck in NAS—time-consuming performance estimation (PE) under resource constraints.
• To develop a systematic framework for identifying the most influential hyper-parameters in PE to enable efficient and accurate architecture search.
• To reduce computational cost in NAS without sacrificing search quality by focusing resources on the most impactful hyper-parameters.
• To provide a unified, generalizable solution compatible with diverse search strategies including reinforcement learning, evolution, random search, and differentiable NAS.

Proposed method

• Proposes Budgeted Performance Estimation (BPE) as a systematic framework to control training hyper-parameters such as learning rate, batch size, epochs, and image size to estimate relative architecture performance efficiently.
• Introduces Minimum Importance Pruning (MIP), which uses random forest regression to estimate the importance of each hyper-parameter based on performance evaluation across sampled configurations.
• In each iteration, MIP prunes the hyper-parameter with the lowest estimated importance, reducing the search space and reallocating computational resources to more influential parameters.
• The MIP process iteratively reduces the number of hyper-parameters in BPE, focusing computation on the most impactful ones to improve estimation efficiency.
• Trains a random forest predictor on a dataset of sampled configurations and their performance to predict the relative ranking of architectures under the optimized BPE.
• Applies the learned MIP-BPE across multiple NAS algorithms, including reinforcement learning, evolution, random search, and DARTS, achieving consistent speedups.

Experimental results

Research questions

• RQ1 How can performance estimation in NAS be systematically optimized under strict computational budgets?
• RQ2 Which hyper-parameters in the training process most significantly influence the relative ranking of neural architectures during performance estimation?
• RQ3 Can a dynamic pruning strategy based on hyper-parameter importance improve the efficiency of NAS without degrading final model accuracy?
• RQ4 Why do one-shot NAS methods like DARTS show high instability across random seeds despite poor global performance estimation?
• RQ5 To what extent can a learned BPE configuration generalize across different search algorithms and datasets?

Key findings

• The proposed MIP-BPE framework achieves up to 1,000× speedup in NAS across multiple algorithms, including reinforcement learning, evolution, random search, and differentiable search, with only a negligible performance drop.
• The correlation between the relative performance ranking (rs) and the proposed objective function reaches 0.65, confirming the effectiveness of the MIP-BPE in preserving accurate relative architecture rankings.
• The two most critical hyper-parameters—training epoch and number of layers—show high importance and steep performance curves, indicating that fine-tuning these in a small range is sufficient for transfer to new datasets.
• One-shot methods like DARTS exhibit poor global performance estimation (low global rs) but high local rs, indicating they rely on local optimization dynamics rather than global accuracy prediction.
• The study reveals that performance estimation ranking can drastically change under minor training condition variations, highlighting the instability of existing proxy metrics and the need for robust BPE.
• The framework generalizes well: once a BPE is learned on one dataset, it can be effectively transferred to other datasets with minimal retraining, especially when focusing on epoch and layer count.