[Paper Review] Dynamic Network Surgery for Efficient DNNs

Yiwen Guo, Anbang Yao+1 more|arXiv (Cornell University)|Aug 16, 2016

Advanced Neural Network Applications16 references140 citations

TL;DR

Introduces dynamic network surgery that prunes and splices connections during training to compress DNNs efficiently without accuracy loss, outperforming prior pruning methods.

ABSTRACT

Deep learning has become a ubiquitous technology to improve machine intelligence. However, most of the existing deep models are structurally very complex, making them difficult to be deployed on the mobile platforms with limited computational power. In this paper, we propose a novel network compression method called dynamic network surgery, which can remarkably reduce the network complexity by making on-the-fly connection pruning. Unlike the previous methods which accomplish this task in a greedy way, we properly incorporate connection splicing into the whole process to avoid incorrect pruning and make it as a continual network maintenance. The effectiveness of our method is proved with experiments. Without any accuracy loss, our method can efficiently compress the number of parameters in LeNet-5 and AlexNet by a factor of $\bm{108} imes$ and $\bm{17.7} imes$ respectively, proving that it outperforms the recent pruning method by considerable margins. Code and some models are available at https://github.com/yiwenguo/Dynamic-Network-Surgery.

Motivation & Objective

• Motivate the need for compressing deep neural networks for mobile/deployable use.
• Propose a dynamic pruning and splicing framework to continually maintain network structure.
• Reduce model size significantly while preserving or improving accuracy.
• Demonstrate substantial compression gains on LeNet variants and AlexNet without loss in performance.

Proposed method

• Represent networks with weight matrices Wk and binary masks Tk indicating active connections.
• Solve a constrained pruning problem by alternating updates of Wk and Tk using SGD on L(Wk ⊙ Tk).
• Use a discriminative function hk(·) to determine which connections are pruned, kept, or spliced based on magnitude thresholds (a_k and b_k).
• Incorporate splicing to re-activate previously pruned connections when they become important again.
• Update rule for Wk follows Wk ← Wk − β ∂L(Wk ⊙ Tk)/∂Wk to allow partial backpropagation through masked weights.
• Control pruning/splicing frequency with a stochastic update probability σ(iter) to stabilize convergence.
• Prune and splice separately for convolutional and fully connected layers to mitigate vanishing gradients and learning inefficiency.

Experimental results

Research questions

• RQ1 Can dynamic pruning with on-the-fly splicing achieve higher compression than prior pruning methods without accuracy loss?
• RQ2 How does the proposed method perform on standard networks (LeNet-5, LeNet-300-100, AlexNet) compared to Han et al.’s pruning?
• RQ3 What are effective criteria for determining parameter importance to guide pruning and splicing?

Key findings

• Dynamic network surgery achieves substantial compression on standard models with no accuracy loss (e.g., LeNet-5 reduced by 108×, AlexNet by 17.7×).
• The method often preserves or slightly improves accuracy despite aggressive pruning.
• Layer-wise comparisons show the approach consistently outperforms Han et al.’s pruning across layers in AlexNet.
• The technique requires significantly fewer training iterations than competing pruning methods (e.g., 700K vs. 960K+ for AlexNet).
• Compression is achieved by retaining a small fraction of parameters with masks dynamically updated through pruning and splicing.