[Paper Review] Semi-Supervised Neural Architecture Search

Renqian Luo, Xu Tan+4 more|arXiv (Cornell University)|Feb 24, 2020

Advanced Neural Network ApplicationsComputer Science45 references43 citations

TL;DR

SemiNAS uses a semi-supervised accuracy predictor trained on a small set of architecture-accuracy pairs plus a large set of unlabeled architectures to guide NAS, reducing cost and improving accuracy across NASBench-101, ImageNet, and TTS tasks.

ABSTRACT

Neural architecture search (NAS) relies on a good controller to generate better architectures or predict the accuracy of given architectures. However, training the controller requires both abundant and high-quality pairs of architectures and their accuracy, while it is costly to evaluate an architecture and obtain its accuracy. In this paper, we propose SemiNAS, a semi-supervised NAS approach that leverages numerous unlabeled architectures (without evaluation and thus nearly no cost). Specifically, SemiNAS 1) trains an initial accuracy predictor with a small set of architecture-accuracy data pairs; 2) uses the trained accuracy predictor to predict the accuracy of large amount of architectures (without evaluation); and 3) adds the generated data pairs to the original data to further improve the predictor. The trained accuracy predictor can be applied to various NAS algorithms by predicting the accuracy of candidate architectures for them. SemiNAS has two advantages: 1) It reduces the computational cost under the same accuracy guarantee. On NASBench-101 benchmark dataset, it achieves comparable accuracy with gradient-based method while using only 1/7 architecture-accuracy pairs. 2) It achieves higher accuracy under the same computational cost. It achieves 94.02% test accuracy on NASBench-101, outperforming all the baselines when using the same number of architectures. On ImageNet, it achieves 23.5% top-1 error rate (under 600M FLOPS constraint) using 4 GPU-days for search. We further apply it to LJSpeech text to speech task and it achieves 97% intelligibility rate in the low-resource setting and 15% test error rate in the robustness setting, with 9%, 7% improvements over the baseline respectively.

Motivation & Objective

• Motivation to reduce the high cost of evaluating architectures in NAS.
• Develop a semi-supervised framework to leverage unlabeled architectures.
• Demonstrate that a semi-supervised predictor can guide NAS with comparable or better accuracy at lower cost.

Proposed method

• Train an encoder-predictor-decoder accuracy predictor on a small set of labeled architecture-accuracy pairs.
• Use the predictor to assign pseudo-labels to a large pool of unlabeled architectures.
• Jointly train the predictor and autoencoder with both true and pseudo-labeled data.
• Iteratively generate new architectures by gradient ascent in the predictor’s embedding space and evaluate only a subset.
• Apply the approach within the NAO framework and show compatibility with conventional and one-shot NAS.
• Compare against baselines like random search, regularized evolution, and standard NAO.

Experimental results

Research questions

• RQ1 Can unlabeled architectures be effectively used to augment NAS training data via a semi-supervised predictor?
• RQ2 Does SemiNAS reduce computational cost while maintaining or improving accuracy relative to conventional and one-shot NAS?
• RQ3 How well does the semi-supervised predictor generalize across NASBench-101, ImageNet, and TTS tasks?

Key findings

• On NASBench-101, SemiNAS with 2000 queries achieves 94.02% test accuracy, surpassing all baselines at the same cost.
• With 300 queries, SemiNAS reaches 93.89% test accuracy, competitive with NAO using 2000 queries.
• SemiNAS for NASBench-101 with 1000 labeled and 10000 unlabeled architectures outperforms baselines in test accuracy and reduces regret.
• On ImageNet under a 600M FLOPS constraint, SemiNAS achieves 23.5% top-1 error, outperforming NAO and ProxylessNAS baselines.
• In TTS on LJSpeech, SemiNAS yields 97% intelligibility (low-resource) and 15% sentence error rate (robustness), with improvements over Transformer TTS and NAO.
• SemiNAS also achieves higher Diagonal Focus Rate (DFR) and MOS than competing TTS models, indicating better attention alignment and perceived quality.