[Paper Review] SMASH: One-Shot Model Architecture Search through HyperNetworks

Andrew Brock, Theodore Lim+2 more|arXiv (Cornell University)|Aug 17, 2017

Machine Learning and Data Classification32 references392 citations

TL;DR

SMASH trains a HyperNet to generate weights for variable-architecture networks, enabling one-shot ranking of many architectures via a single training run. It evaluates architectures by the HyperNet-generated weights to select promising designs for full training.

ABSTRACT

Designing architectures for deep neural networks requires expert knowledge and substantial computation time. We propose a technique to accelerate architecture selection by learning an auxiliary HyperNet that generates the weights of a main model conditioned on that model's architecture. By comparing the relative validation performance of networks with HyperNet-generated weights, we can effectively search over a wide range of architectures at the cost of a single training run. To facilitate this search, we develop a flexible mechanism based on memory read-writes that allows us to define a wide range of network connectivity patterns, with ResNet, DenseNet, and FractalNet blocks as special cases. We validate our method (SMASH) on CIFAR-10 and CIFAR-100, STL-10, ModelNet10, and Imagenet32x32, achieving competitive performance with similarly-sized hand-designed networks. Our code is available at https://github.com/ajbrock/SMASH

Motivation & Objective

• Motivate reducing architectural search cost by avoiding full training of every candidate.
• Propose a flexible memory-bank based encoding to represent diverse network connectivities.
• Introduce a Dynamic HyperNet that maps architecture encodings to weights for the main network.
• Investigate correlation between SMASH proxy performance and truly trained performance across datasets.
• Explore transferability of found architectures across datasets and domains.

Proposed method

• Define a memory-bank view of networks to encode variable depth and connectivity as binary vectors.
• Train a HyperNet to map architecture encodings to the weights of a main network, enabling weight generation conditioned on architecture.
• During SMASH training, sample architectures, generate their weights via the HyperNet, and backpropagate through the whole system.
• Evaluate a set of architectures on a validation set using HyperNet-generated weights to rank them for full training.
• Fix the best-scoring architecture and train it with freely learned weights for final evaluation.

Experimental results

Research questions

• RQ1 Does the SMASH-generated weight proxy correlate with performance of fully trained architectures?
• RQ2 Can SMASH reliably rank a wide range of architectures using a single training run?
• RQ3 How does the architecture search via SMASH transfer to different datasets and domains?
• RQ4 What is the effect of HyperNet capacity on the reliability of SMASH scores?
• RQ5 Can the learned architecture-to-weights mapping be exploited for gradient-like guidance in architecture space?

Key findings

• There is a observed correlation between SMASH scores and true validation performance in CIFAR-100 under certain training configurations.
• Reduced-capacity HyperNets can break the correlation between SMASH scores and true performance.
• When the majority of capacity is in fixed (non-generated) weights, SMASH scoring may not predict architectural performance well.
• AMethods that corrupt the architecture encoding during evaluation degrade validation performance, suggesting the HyperNet learns architecture-conditioned weights.
• Best-performing SMASHv2 architectures achieved competitive results on CIFAR-10/100 and Imagenet32x32 compared to some hand-designed networks and other NAS methods.
• Transfer of found CIFAR-based architectures to STL-10 and ModelNet10 showed mixed results, indicating larger datasets may improve architecture discrimination.