[Paper Review] Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon

Xin Dong, Shangyu Chen+1 more|arXiv (Cornell University)|May 22, 2017

Advanced Neural Network Applications284 citations

TL;DR

Introduces Layer-wise Optimal Brain Surgeon (L-OBS), a layer-wise pruning method using second-order derivatives to prune individual layers with bounded final error and only light retraining. It achieves high compression with maintained accuracy across multiple architectures.

ABSTRACT

How to develop slim and accurate deep neural networks has become crucial for real- world applications, especially for those employed in embedded systems. Though previous work along this research line has shown some promising results, most existing methods either fail to significantly compress a well-trained deep network or require a heavy retraining process for the pruned deep network to re-boost its prediction performance. In this paper, we propose a new layer-wise pruning method for deep neural networks. In our proposed method, parameters of each individual layer are pruned independently based on second order derivatives of a layer-wise error function with respect to the corresponding parameters. We prove that the final prediction performance drop after pruning is bounded by a linear combination of the reconstructed errors caused at each layer. Therefore, there is a guarantee that one only needs to perform a light retraining process on the pruned network to resume its original prediction performance. We conduct extensive experiments on benchmark datasets to demonstrate the effectiveness of our pruning method compared with several state-of-the-art baseline methods.

Motivation & Objective

• Motivate the need for slim, accurate networks suitable for embedded and real-world deployment.
• Propose a layer-wise pruning framework that uses second-order information to identify prune-able parameters per layer.
• Guarantee that the overall network performance drop is bounded by reconstructed per-layer errors.
• Reduce computational burden by focusing Hessian computations on each layer and enabling a light retraining phase after pruning.

Proposed method

• Model the pruning of each layer via a layer-wise error using the pre-activation outputs Z^l and a Taylor expansion of the layer-wise error E^l.
• Compute an inverse Hessian H_l^{-1} for each layer to obtain sensitivities L_q = (Θ_{l[q]})^2 / (2 [H_l^{-1}]_{qq}).
• Prune parameters with smallest sensitivity scores while ensuring the induced layer-wise error stays below a threshold ε.
• Provide a bound on the final network error: ε̃^L ≤ sum over layers of products of subsequent layer norms times √δE^l, ensuring controlled overall degradation.
• Extend the Hessian-based pruning to convolutional layers by vectorizing filters and deriving a block-diagonal Hessian structure that simplifies inversion.
• Offer an iterative version where pruning and light retraining are alternated to achieve higher compression ratios.

Experimental results

Research questions

• RQ1 Can layer-wise second-order pruning achieve high compression while maintaining accuracy without full network retraining?
• RQ2 How can per-layer pruning errors be controlled to bound the overall network performance drop?
• RQ3 What are efficient mechanisms to compute and invert layer-wise Hessians in deep networks to enable scalable pruning?
• RQ4 Does the approach generalize to convolutional layers and modern architectures beyond fully-connected networks?

Key findings

• L-OBS achieves substantial compression with minimal accuracy loss before retraining across multiple models (e.g., LeNet variants, CIFAR-Net, AlexNet, VGG-16, ResNet-50).
• For LeNet-300-100, L-OBS (single stage) attains 7% compression with 3.10% Re-Error, while iterative pruning achieves 1.5–1.8% Re-Error with higher compression.
• For LeNet-5 and CIFAR-Net, L-OBS reduces error after pruning to single-digit percent ranges and requires far fewer retraining iterations than competing methods.
• On AlexNet and VGG-16, L-OBS achieves notable compression (11% for AlexNet, 7.5% for VGG-16) with no substantial drops in accuracy, sometimes with minimal retraining.
• L-OBS enables pruning of ResNet-50 while maintaining top-5 accuracy above 85% at pruning ratios ≥ 45%.
• The method reduces retraining burden compared to baselines like Net-Trim, DNS, and LWC, and supports iterative pruning to reach higher compression with light retraining.