[Paper Review] NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm

Xiaoliang Dai, Hongxu Yin+1 more|arXiv (Cornell University)|Nov 6, 2017

Advanced Neural Network Applications42 references51 citations

TL;DR

NeST jointly grows and prunes DNN architectures during training from a sparse seed to synthesize compact, accurate networks, achieving substantial parameter and FLOPs reductions on MNIST and ImageNet.

ABSTRACT

Deep neural networks (DNNs) have begun to have a pervasive impact on various applications of machine learning. However, the problem of finding an optimal DNN architecture for large applications is challenging. Common approaches go for deeper and larger DNN architectures but may incur substantial redundancy. To address these problems, we introduce a network growth algorithm that complements network pruning to learn both weights and compact DNN architectures during training. We propose a DNN synthesis tool (NeST) that combines both methods to automate the generation of compact and accurate DNNs. NeST starts with a randomly initialized sparse network called the seed architecture. It iteratively tunes the architecture with gradient-based growth and magnitude-based pruning of neurons and connections. Our experimental results show that NeST yields accurate, yet very compact DNNs, with a wide range of seed architecture selection. For the LeNet-300-100 (LeNet-5) architecture, we reduce network parameters by 70.2x (74.3x) and floating-point operations (FLOPs) by 79.4x (43.7x). For the AlexNet and VGG-16 architectures, we reduce network parameters (FLOPs) by 15.7x (4.6x) and 30.2x (8.6x), respectively. NeST's grow-and-prune paradigm delivers significant additional parameter and FLOPs reduction relative to pruning-only methods.

Motivation & Objective

• Motivate the need to discover compact DNN architectures without excessive trial-and-error or fixed architectures.
• Propose NeST, a synthesis tool that learns both weights and architectures via growth and pruning.
• Demonstrate that NeST yields accurate yet highly compact models across multiple datasets and seed architectures.
• Show that growth+pruning delivers larger reductions than pruning alone.

Proposed method

• Start from a randomly initialized sparse seed architecture with all neurons connected.
• Apply two sequential phases: gradient-based growth to add connections/neurons/feature maps, then magnitude-based pruning to remove redundancies.
• Use three growth policies: (1) connection growth driven by loss-reduction gradients, (2) neuron growth via bridging gradients and correlated neuron pairs with square-root initialized weights, and (3) feature-map growth for convolutional layers.
• In pruning, apply magnitude-based pruning with batch normalization adjustments and introduce partial-area convolution to prune input regions per kernel.
• Optionally retrain after pruning to recover performance.
• Implement and evaluate in TensorFlow/PyTorch on MNIST and ImageNet with LeNet-, AlexNet-, and VGG-16-based seeds.

Experimental results

Research questions

• RQ1 Can NeST synthesize compact, accurate DNNs starting from diverse sparse seed architectures?
• RQ2 How does gradient-guided growth interact with magnitude-based pruning to reduce parameters and FLOPs compared to pruning alone?
• RQ3 What is the impact of seed size and architecture on final model compactness and accuracy for MNIST and ImageNet?
• RQ4 Does partial-area convolution further reduce FLOPs without accuracy loss in practice?

Key findings

• For LeNet-300-100 on MNIST, NeST reduces parameters by 70.2× and FLOPs by 79.4× (74.3× and 43.7× for LeNet-5).
• For AlexNet and VGG-16 on ImageNet, NeST reduces parameters by 15.7× and 30.2× and FLOPs by 4.6× and 8.6×, respectively.
• Grow-and-prune achieves larger reductions than pruning-only methods across tested architectures.
• NeST enables a wide seed range and yields very compact DNNs with no accuracy loss relative to fully connected baselines in reported cases.