[Paper Review] Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?

Yan Shen, Yu Zheng+3 more|arXiv (Cornell University)|Jun 12, 2020

Domain Adaptation and Few-Shot LearningComputer Science70 references34 citations

TL;DR

arch2vec pre-trains architecture representations with an unsupervised variational graph autoencoder, decoupling representation learning from search and yielding smoother latent spaces that improve NAS efficiency across multiple search spaces and strategies.

ABSTRACT

Existing Neural Architecture Search (NAS) methods either encode neural architectures using discrete encodings that do not scale well, or adopt supervised learning-based methods to jointly learn architecture representations and optimize architecture search on such representations which incurs search bias. Despite the widespread use, architecture representations learned in NAS are still poorly understood. We observe that the structural properties of neural architectures are hard to preserve in the latent space if architecture representation learning and search are coupled, resulting in less effective search performance. In this work, we find empirically that pre-training architecture representations using only neural architectures without their accuracies as labels considerably improve the downstream architecture search efficiency. To explain these observations, we visualize how unsupervised architecture representation learning better encourages neural architectures with similar connections and operators to cluster together. This helps to map neural architectures with similar performance to the same regions in the latent space and makes the transition of architectures in the latent space relatively smooth, which considerably benefits diverse downstream search strategies.

Motivation & Objective

• Motivate unsupervised architecture representation learning for NAS to avoid bias from supervised signals.
• Develop a decoupled architecture representation learning approach (arch2vec).
• Evaluate whether unsupervised pre-training improves downstream NAS efficiency and robustness across diverse spaces and strategies.

Proposed method

• Propose arch2vec, a variational graph isomorphism autoencoder that encodes architectures using only structural information (no accuracies).
• Use Graph Isomorphism Networks (GINs) to obtain node embeddings and a variational decoder to reconstruct adjacency and operation matrices.
• Train with a variational lower bound including reconstruction loss and KL divergence to a Gaussian prior.
• Evaluate downstream NAS performance using RL (REINFORCE) and Bayesian Optimization (DNGO) with pre-trained embeddings.
• Compare against discrete adjacency-based encodings and supervised architecture representation learning on NAS-Bench-101, NAS-Bench-201, and DARTS.
• Provide pre-training performance metrics (reconstruction, validity, uniqueness) and NAS performance (test regret, accuracy) across spaces.

Experimental results

Research questions

• RQ1 Can unsupervised pre-training of architecture representations yield smoother latent spaces than jointly learned supervised representations?
• RQ2 Do arch2vec embeddings improve the efficiency and robustness of NAS when paired with RL and BO strategies across multiple search spaces?
• RQ3 How well do unsupervised embeddings preserve local structural relationships and predict architecture performance?
• RQ4 Is architecture search biased by supervised signals when learning representations jointly with search?

Key findings

• arch2vec outperforms GAE and VGAE in reconstruction, validity, and uniqueness across NAS-Bench-101, NAS-Bench-201, and DARTS.
• arch2vec embeddings yield better predictive performance for architecture performance than supervised representations (lower RMSE and higher Pearson r).
• In arch2vec, latent-space distances correlate monotonically with edit distances of architectures, indicating preserved structural closeness.
• t-SNE visualizations show arch2vec embeddings cover the latent space with clustering of similar accuracies, unlike the supervised counterpart which is more discontinuous.
• Downstream NAS using arch2vec (RL and BO) achieves competitive or superior final performance and robustness compared to baselines, with arch2vec-BO and arch2vec-RL often leading to the best results across datasets.