[Paper Review] Efficient Architecture Search by Network Transformation

Han Cai, Tianyao Chen+3 more|arXiv (Cornell University)|Jul 16, 2017

Advanced Neural Network Applications323 citations

TL;DR

The paper introduces Efficient Architecture Search (EAS), which uses function-preserving network transformations guided by a reinforcement-learning meta-controller to reuse weights and efficiently explore CNN architectures.

ABSTRACT

Techniques for automatically designing deep neural network architectures such as reinforcement learning based approaches have recently shown promising results. However, their success is based on vast computational resources (e.g. hundreds of GPUs), making them difficult to be widely used. A noticeable limitation is that they still design and train each network from scratch during the exploration of the architecture space, which is highly inefficient. In this paper, we propose a new framework toward efficient architecture search by exploring the architecture space based on the current network and reusing its weights. We employ a reinforcement learning agent as the meta-controller, whose action is to grow the network depth or layer width with function-preserving transformations. As such, the previously validated networks can be reused for further exploration, thus saves a large amount of computational cost. We apply our method to explore the architecture space of the plain convolutional neural networks (no skip-connections, branching etc.) on image benchmark datasets (CIFAR-10, SVHN) with restricted computational resources (5 GPUs). Our method can design highly competitive networks that outperform existing networks using the same design scheme. On CIFAR-10, our model without skip-connections achieves 4.23\% test error rate, exceeding a vast majority of modern architectures and approaching DenseNet. Furthermore, by applying our method to explore the DenseNet architecture space, we are able to achieve more accurate networks with fewer parameters.

Motivation & Objective

• Motivate reducing the computational cost of automatic architecture design by reusing trained networks.
• Propose a framework (EAS) that applies function-preserving transformations to grow or widen networks.
• Leverage a reinforcement learning agent to decide useful transformation actions.
• Demonstrate efficiency and competitiveness on CIFAR-10 and SVHN with limited GPUs.

Proposed method

• Model architecture search as a sequential decision process where the state is the current network and actions are network transformation operations.
• Use Net2WiderNet and Net2DeeperNet as the primary function-preserving transformations to widen or insert layers while preserving functionality.
• Extend Net2Net operations to DenseNet by adapting the transformation to multiple-input paths.
• Employ a bidirectional LSTM encoder to represent the current architecture and multiple actor networks to propose transformation actions.
• Train the RL meta-controller with REINFORCE using a transformed accuracy reward and a moving baseline for variance reduction.
• Experiment with plain CNN spaces and DenseNet spaces on CIFAR-10 and SVHN under resource constraints (5 GPUs).

Experimental results

Research questions

• RQ1 Can function-preserving transformations efficiently explore architecture space without retraining from scratch?
• RQ2 How well can an RL-based meta-controller learn to widen or insert layers to improve validation performance?
• RQ3 Do transformations generalize across plain CNNs and DenseNet-like architectures under limited resources?
• RQ4 What are the performance gains on CIFAR-10 and SVHN compared to baseline architectures and prior auto-design methods?

Key findings

• EAS with 5 GPUs discovers competitive plain CNNs achieving 4.23% test error on CIFAR-10 with augmentation.
• EAS achieves 4.66% test error on CIFAR-10 and 3.44% on CIFAR-10+ for DenseNet spaces with fewer parameters than some baselines.
• The RL-based meta-controller outperforms random search in finding high-performing architectures.
• On SVHN, the top plain CNN from EAS reaches 1.73% test error after training, outperforming many auto-designed models in the same space.
• EAS reduces computational burden by reusing weights and requiring far fewer GPUs than previous large-scale NAS approaches.
• DenseNet exploration via EAS yields 3.44% on CIFAR-10+ with 10.7M parameters, outperforming several DenseNet variants.