[Paper Review] Hierarchical Representations for Efficient Architecture Search

Hanxiao Liu, Karen Simonyan+3 more|arXiv (Cornell University)|Nov 1, 2017

Evolutionary Algorithms and Applications23 references227 citations

TL;DR

The paper introduces hierarchical architecture representations and demonstrates that evolutionary or random search over these representations can discover competitive neural network cells for image classification, achieving top-1 error of 3.75% on CIFAR-10 and 20.3% top-1 on ImageNet, with efficient search times.

ABSTRACT

We explore efficient neural architecture search methods and show that a simple yet powerful evolutionary algorithm can discover new architectures with excellent performance. Our approach combines a novel hierarchical genetic representation scheme that imitates the modularized design pattern commonly adopted by human experts, and an expressive search space that supports complex topologies. Our algorithm efficiently discovers architectures that outperform a large number of manually designed models for image classification, obtaining top-1 error of 3.6% on CIFAR-10 and 20.3% when transferred to ImageNet, which is competitive with the best existing neural architecture search approaches. We also present results using random search, achieving 0.3% less top-1 accuracy on CIFAR-10 and 0.1% less on ImageNet whilst reducing the search time from 36 hours down to 1 hour.

Motivation & Objective

• Motivate efficient neural architecture search (NAS) by designing a hierarchical, modular representation that mirrors human-driven block design.
• Show that simple search strategies (random/evolutionary) can achieve competitive performance when paired with a powerful hierarchical space.
• Demonstrate scalability and transferability of discovered architectures from CIFAR-10 to ImageNet.
• Provide an efficient distributed search framework to reduce NAS computation time.

Proposed method

• Define flat and hierarchical neural architecture representations.
• Introduce a hierarchical motif-based encoding where lower-level motifs serve as building blocks for higher-level motifs.
• Specify six bottom-level primitives (1x1 conv, 3x3 depthwise conv, 3x3 separable conv, 3x3 max-pool, 3x3 avg-pool, identity) plus a none edge option.
• Use an evolutionary search with mutation operators that modify edges between motifs across hierarchy (including adding, altering, or removing edges).
• Employ asynchronous distributed evolution with a controller performing tournament selection and workers evaluating architecture fitness by training from scratch for a fixed budget.
• Compare evolutionary search to random search to assess the impact of the representation and search strategy.

Experimental results

Research questions

• RQ1 Can hierarchical representations improve the efficiency and effectiveness of NAS compared to flat representations?
• RQ2 How do random search and evolutionary search fare in finding high-performing architectures within the hierarchical space?
• RQ3 What are the performance and transfer capabilities of architectures found on CIFAR-10 when evaluated on ImageNet?
• RQ4 What is the computational efficiency (time, resources) of the proposed asynchronous, distributed NAS framework?

Key findings

• Hierarchical representations enable more parameter-efficient architectures than flat representations at comparable performance.
• Random search over the hierarchical space yields competitive results, confirming the importance of a well-designed search space.
• Evolutionary search over hierarchical representations achieves the best results: CIFAR-10 3.75% ±0.12% (64 channels) and 3.63% ±0.10% (128 channels); ImageNet top-1 20.3% and top-5 5.2%.
• Compared to prior NAS work, the method achieves strong performance with substantially less wall-clock time (e.g., 1 hour for random search over 200 architectures; 1.5 days for 7000-step evolution on 200 GPUs).
• The evolved hierarchical cell contains skip connections and modular motifs, illustrating learned architectural patterns.
• The evolved model with hierarchical cells has around 64M parameters (ImageNet), comparable to Inception-ResNet-v2 and larger than NASNet-A.