[Paper Review] Evaluating the Search Phase of Neural Architecture Search

Kaicheng Yu, Christian Sciuto+3 more|arXiv (Cornell University)|Feb 21, 2019

Advanced Neural Network Applications45 references166 citations

TL;DR

The paper evaluates NAS search strategies by comparing them to random architecture search within the same search spaces, revealing that state-of-the-art NAS methods do not significantly outperform random search, and that weight sharing degrades ranking accuracy.

ABSTRACT

Neural Architecture Search (NAS) aims to facilitate the design of deep networks for new tasks. Existing techniques rely on two stages: searching over the architecture space and validating the best architecture. NAS algorithms are currently compared solely based on their results on the downstream task. While intuitive, this fails to explicitly evaluate the effectiveness of their search strategies. In this paper, we propose to evaluate the NAS search phase. To this end, we compare the quality of the solutions obtained by NAS search policies with that of random architecture selection. We find that: (i) On average, the state-of-the-art NAS algorithms perform similarly to the random policy; (ii) the widely-used weight sharing strategy degrades the ranking of the NAS candidates to the point of not reflecting their true performance, thus reducing the effectiveness of the search process. We believe that our evaluation framework will be key to designing NAS strategies that consistently discover architectures superior to random ones.

Motivation & Objective

• Motivate explicit evaluation of NAS search strategies rather than only final task performance.
• Compare NAS search policies (DARTS, NAO, ENAS) with random search in identical search spaces.
• Identify factors affecting search effectiveness, notably weight sharing and search space constraints.
• Provide an evaluation framework and publicly available code to facilitate robust NAS search analysis.

Proposed method

• Define a fair random search baseline that uniformly samples architectures from the same search space as NAS methods.
• Repeat searches with multiple random seeds to reduce randomness and compare mean performances.
• Use reduced search spaces (RNN with n=2 nodes and NASBench-101 CNN space) to obtain ground-truth architecture performances for exhaustive evaluation.
• Measure ranking correspondence between NAS-predicted architectures and their true performance (Kendall Tau) to quantify the impact of weight sharing.
• Train all sampled architectures with identical hyperparameters and epoch budgets to ensure fair comparisons.

Experimental results

Research questions

• RQ1 Do state-of-the-art NAS algorithms (DARTS, NAO, ENAS) outperform random search in the same search spaces?
• RQ2 How does weight sharing affect the correlation between NAS rankings during search and ground-truth performance?
• RQ3 Does constraining or exhaustively evaluating the search space change the relative performance of NAS methods versus random search?
• RQ4 What is the impact of removing weight sharing on NAS performance in reduced search spaces?

Key findings

• In standard search spaces, NAS methods (DARTS, NAO, ENAS) generally do not significantly outperform random search.
• In RNN space, random search often yields the best or near-best perplexity/accuracy, challenging the effectiveness of NAS policies.
• Weight sharing severely biases architecture rankings and reduces the reliability of the search phase, sometimes making NAS no better than random search.
• Removing weight sharing in reduced search spaces substantially improves NAS performance, with NAO and ENAS surpassing random search more consistently.
• In reduced CNN space, NAS methods improve over random search less dramatically, and their rankings are highly affected by weight sharing, especially as space size grows.
• The authors provide a framework and public code to enable fair, seed-robust evaluation of NAS search strategies.