[Paper Review] NAT: Neural Architecture Transformer for Accurate and Compact Architectures

Yong Guo, Yin Zheng+5 more|arXiv (Cornell University)|Oct 31, 2019

Advanced Neural Network ApplicationsComputer Science73 citations

TL;DR

NAT treats architecture optimization as an MDP to replace redundant operations with efficient ones, yielding more accurate and compact architectures for hand-crafted and NAS-based models on CIFAR-10 and ImageNet.

ABSTRACT

Designing effective architectures is one of the key factors behind the success of deep neural networks. Existing deep architectures are either manually designed or automatically searched by some Neural Architecture Search (NAS) methods. However, even a well-searched architecture may still contain many non-significant or redundant modules or operations (e.g., convolution or pooling), which may not only incur substantial memory consumption and computation cost but also deteriorate the performance. Thus, it is necessary to optimize the operations inside an architecture to improve the performance without introducing extra computation cost. Unfortunately, such a constrained optimization problem is NP-hard. To make the problem feasible, we cast the optimization problem into a Markov decision process (MDP) and seek to learn a Neural Architecture Transformer (NAT) to replace the redundant operations with the more computationally efficient ones (e.g., skip connection or directly removing the connection). Based on MDP, we learn NAT by exploiting reinforcement learning to obtain the optimization policies w.r.t. different architectures. To verify the effectiveness of the proposed strategies, we apply NAT on both hand-crafted architectures and NAS based architectures. Extensive experiments on two benchmark datasets, i.e., CIFAR-10 and ImageNet, demonstrate that the transformed architecture by NAT significantly outperforms both its original form and those architectures optimized by existing methods.

Motivation & Objective

• Motivate the need to prune non-significant or redundant modules within architectures to improve performance and reduce computation.
• Propose a general architecture optimizer that transforms any given architecture without adding computational cost.
• Cast architecture optimization as an MDP and learn a policy to selectively replace operations with skip connections or null edges.
• Leverage graph convolutional networks to capture adjacency information and guide operation transformations.
• Demonstrate effectiveness on both hand-crafted and NAS-based architectures across CIFAR-10 and ImageNet.

Proposed method

• Model architectures as DAGs with edges categorized as N (null), S (skip), or O (other operations) and define cost ordering c(O)>c(S)>c(N).
• Cast optimization as a one-step Markov decision process and learn a policy to transform β into α with preserved or reduced cost.
• Use a Graph Convolutional Network (GCN) to parameterize the policy and capture local graph structure for edge-level operation decisions.
• Train with policy gradient and entropy regularization to encourage exploration and diverse architecture transformations.
• Employ parameter sharing to train a single NAT over many architectures by building a large shared computation graph.
• Infer optimized architectures by sampling multiple α from the learned policy and selecting the best validation accuracy.

Experimental results

Research questions

• RQ1 Can NAT reliably convert arbitrary architectures into more accurate and/or more compact forms without extra computational cost?
• RQ2 Does NAT provide consistent improvements for both hand-crafted networks (e.g., VGG, ResNet, MobileNet) and NAS-derived models (e.g., DARTS, ENAS, NAONet)?
• RQ3 Does using a GCN-based policy outperform LSTM or random search for architecture transformation?

Key findings

• NAT consistently improves hand-crafted models with comparable compute cost, achieving up to 2.75% Top-1 accuracy gain on ImageNet for VGG-related cases.
• For NAS-based models, NAT reduces parameters by about 20% and yields a 0.6% Top-1 accuracy improvement on ImageNet for certain baselines.
• Across CIFAR-10 and ImageNet, NAT-based architectures outperform their originals and outperform baselines optimized by NAO in most cases.
• Sampling-based policy (GCN) outperforms random search, LSTM, and Maximum-GCN in producing better validated architectures.