[Paper Review] Graph HyperNetworks for Neural Architecture Search

Wenjun Zhang, Mengye Ren+1 more|arXiv (Cornell University)|Oct 12, 2018

Advanced Neural Network ApplicationsComputer Science39 references115 citations

TL;DR

Graph HyperNetwork (GHN) generates all weights for a candidate CNN directly from its computation graph, enabling fast NAS by evaluating many architectures with GHN-generated weights and achieving competitive accuracy with ~10x faster search than random baselines.

ABSTRACT

Neural architecture search (NAS) automatically finds the best task-specific neural network topology, outperforming many manual architecture designs. However, it can be prohibitively expensive as the search requires training thousands of different networks, while each can last for hours. In this work, we propose the Graph HyperNetwork (GHN) to amortize the search cost: given an architecture, it directly generates the weights by running inference on a graph neural network. GHNs model the topology of an architecture and therefore can predict network performance more accurately than regular hypernetworks and premature early stopping. To perform NAS, we randomly sample architectures and use the validation accuracy of networks with GHN generated weights as the surrogate search signal. GHNs are fast -- they can search nearly 10 times faster than other random search methods on CIFAR-10 and ImageNet. GHNs can be further extended to the anytime prediction setting, where they have found networks with better speed-accuracy tradeoff than the state-of-the-art manual designs.

Motivation & Objective

• Motivate reducing NAS computational cost by amortizing inner-loop weight optimization.
• Introduce a graph-based representation of architectures to capture topology.
• Develop GHN to predict all weights via a graph neural network plus hypernetwork.
• Demonstrate GHN-enabled NAS efficiency and effectiveness on CIFAR-10 and ImageNet-mobile.
• Extend GHN to anytime prediction to optimize speed-accuracy tradeoffs.

Proposed method

• Encode neural architectures as computation graphs where nodes are operators with weights.
• Use a graph neural network to produce node embeddings that summarize topology.
• Apply a shared hypernetwork to map node embeddings to all node weights via equation w~v=H(hv^(T);φ).
• Train GHN end-to-end by backpropagating training loss through generated weights.
• Adopt forward-backward asynchronous propagation to improve gradient flow over long DAGs.
• Explore architectural motifs by stacking GHNs across repeated modules with shared parameters.

Experimental results

Research questions

• RQ1 Can a graph-based hypernetwork accurately predict weights for unseen architectures from their computation graphs?
• RQ2 Does GHN-trained surrogate signal enable competitive NAS performance with substantially reduced search cost?
• RQ3 What is the correlation between GHN-predicted performance and fully trained performance across architectures?
• RQ4 Can GHN be extended to anytime prediction to improve speed-accuracy tradeoffs?
• RQ5 How do design choices (node count, propagation scheme, stacked architectures) affect GHN effectiveness?

Key findings

• GHN achieves competitive CIFAR-10 results and ImageNet-mobile results with about 10x faster search cost than several random-search baselines.
• On CIFAR-10, GHN Top (F=32) requires 0.42 GPU-days search cost and yields 5.1e6 parameters with 95.7% accuracy; GHN Top-Best (1K, F=32) achieves 97.16% ±0.07% with 0.84 GPU-days and 5.7e6 parameters.
• On ImageNet-mobile, transferring the best CIFAR block via GHN yields Top-1 73.0% and Top-5 91.3% with 0.84 GPU-days search cost and ~6.1e6 parameters.
• Predicted-performance correlation shows GHN yields higher correlation for top architectures (0.68 with Random-100, Top-50 0.48 under a shared-parameter stacked GHN with forward-backward propagation) compared to one-shot and SGD baselines.
• GHN demonstrates strong anytime-prediction performance, finding networks with better speed-accuracy tradeoffs than state-of-the-art manually designed models.
• Ablation studies indicate that increasing node count beyond a point can hurt learning, forward-backward propagation outperforms synchronous schemes, and sharing parameters across stacked GHNs improves performance.