[Paper Review] MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning

Zechun Liu, Haoyuan Mu+5 more|arXiv (Cornell University)|Mar 25, 2019

Domain Adaptation and Few-Shot LearningComputer Science56 references99 citations

TL;DR

MetaPruning trains a PruningNet to predict weights for any pruned structure and uses evolutionary search to find high-performing pruned networks under constraints, enabling efficient AutoML-style channel pruning without fine-tuning. It improves MobileNet and ResNet accuracy over baselines under FLOPs/latency constraints.

ABSTRACT

In this paper, we propose a novel meta learning approach for automatic channel pruning of very deep neural networks. We first train a PruningNet, a kind of meta network, which is able to generate weight parameters for any pruned structure given the target network. We use a simple stochastic structure sampling method for training the PruningNet. Then, we apply an evolutionary procedure to search for good-performing pruned networks. The search is highly efficient because the weights are directly generated by the trained PruningNet and we do not need any finetuning at search time. With a single PruningNet trained for the target network, we can search for various Pruned Networks under different constraints with little human participation. Compared to the state-of-the-art pruning methods, we have demonstrated superior performances on MobileNet V1/V2 and ResNet. Codes are available on https://github.com/liuzechun/MetaPruning.

Motivation & Objective

• Reduce manual hyperparameter tuning in channel pruning by learning a meta network that can generate weights for various pruned structures.
• Enable fast search over pruned architectures without finetuning by predicting weights for pruned networks.
• Incorporate constraints such as FLOPs and latency into the pruning search process.
• Handle shortcut/pruned structures in ResNet-like networks through meta-learning.
• Demonstrate superior accuracy over uniform pruning and prior AutoML-based pruning methods on MobileNet and ResNet.

Proposed method

• Train PruningNet, a meta network, to generate weights for any pruned structure given a network encoding vector (channel counts per layer).
• Use stochastic structure sampling to train PruningNet across a distribution of pruned architectures.
• Construct pruned networks by varying the encoding vector and evaluate their validation accuracy using PruningNet-generated weights, without fine-tuning.
• Apply an evolutionary search over pruned architectures (with constraints) to identify high-performing structures.
• For each layer, crop PruningNet-generated weights to match pruned input/output channels before evaluation.
• Compare against uniform pruning baselines and state-of-the-art channel pruning methods under FLOPs and latency constraints.

Experimental results

Research questions

• RQ1 Can a single trained PruningNet generate weights for a wide range of pruned network structures?
• RQ2 Does evolutionary search with a PruningNet enable finding high-quality pruned architectures under FLOPs/latency constraints without fine-tuning?
• RQ3 How does MetaPruning perform compared to uniform pruning and existing AutoML-based pruning methods on MobileNet and ResNet?
• RQ4 Can MetaPruning effectively prune channels in ResNet-style shortcut blocks?
• RQ5 What is the impact of weight prediction in the PruningNet on pruning performance?

Key findings

• MetaPruning achieves higher accuracy than uniform pruning baselines at the same FLOPs for MobileNet V1, MobileNet V2, and ResNet-50.
• At the same latency, MetaPruning outperforms MobileNet V1/V2 baselines by up to 9.9 percentage points in Top-1 accuracy.
• MetaPruning outperforms ThiNet, CP, and SFP pruning methods on ResNet-50 under comparable FLOPs.
• Compared with AutoML pruning methods AMC and NetAdapt, MetaPruning yields superior accuracy for similar FLOPs.
• The PruningNet can predict weights across many pruned structures, enabling efficient search without retraining during search.