[Paper Review] Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures

Hengyuan Hu, Rui Tao Peng+2 more|arXiv (Cornell University)|Jul 12, 2016

Advanced Neural Network Applications15 references741 citations

TL;DR

The paper proposes Network Trimming, an iterative method that prunes neurons with high APoZ (zero activations) to create smaller, more efficient networks, retraining with weight initialization to preserve or improve accuracy.

ABSTRACT

State-of-the-art neural networks are getting deeper and wider. While their performance increases with the increasing number of layers and neurons, it is crucial to design an efficient deep architecture in order to reduce computational and memory costs. Designing an efficient neural network, however, is labor intensive requiring many experiments, and fine-tunings. In this paper, we introduce network trimming which iteratively optimizes the network by pruning unimportant neurons based on analysis of their outputs on a large dataset. Our algorithm is inspired by an observation that the outputs of a significant portion of neurons in a large network are mostly zero, regardless of what inputs the network received. These zero activation neurons are redundant, and can be removed without affecting the overall accuracy of the network. After pruning the zero activation neurons, we retrain the network using the weights before pruning as initialization. We alternate the pruning and retraining to further reduce zero activations in a network. Our experiments on the LeNet and VGG-16 show that we can achieve high compression ratio of parameters without losing or even achieving higher accuracy than the original network.

Motivation & Objective

• Motivation to design efficient deep architectures amidst growing depth and width of networks.
• Identify redundancy in neurons by analyzing activation sparsity on a large validation set.
• Develop an iterative pruning-retraining loop that preserves performance while reducing parameters.
• Provide practical guidance on choosing layers to trim and thresholding for pruning.

Proposed method

• Measure Average Percentage of Zeros (APoZ) for each neuron on a large validation set.
• Prune neurons with APoZ above a threshold (roughly one standard deviation above the mean APoZ of the target layer).
• Initialize the trimmed network with weights from the ancestor model and retrain (or fine-tune) to recover performance.
• Iteratively prune and retrain across layers to gradually reduce redundancy.
• Train networks from empirical baselines (e.g., VGG-16, LeNet) and evaluate compression vs. accuracy.
• Compare against weight-pruning approaches and emphasize neuron-level pruning for GPU efficiency.

Experimental results

Research questions

• RQ1 Can pruning high-APoZ neurons reduce model size without harming accuracy?
• RQ2 Is an iterative pruning-retraining loop effective for large architectures like VGG-16?
• RQ3 How does weight initialization influence retraining after pruning?
• RQ4 Which layers yield the most benefit when pruned in networks like LeNet and VGG-16?
• RQ5 How does APoZ-based pruning compare to connection-pruning methods in terms of computation and memory efficiency?

Key findings

• Significant redundancy exists in networks, with many neurons having high APoZ (e.g., across VGG-16 layers).
• Iterative pruning of high-APoZ neurons achieves 2–3x parameter compression on LeNet without accuracy loss after retraining.
• Weight initialization from the ancestor model is essential to recover performance after pruning.
• In VGG-16, pruning CONV5-3 and FC6 can reach ~2.59x compression with 2–3% higher Top-1/Top-5 accuracy after retraining.
• Trimming multiple layers can be effective but requires retraining; trimming last conv and FC layers yields substantial parameter reductions with maintained or improved accuracy.
• Trimmed VGG-16 models can outperform the original with fewer parameters and reduced overfitting.