[Paper Review] Network Decoupling: From Regular to Depthwise Separable Convolutions

Jianbo Guo, Yuxi Li+3 more|arXiv (Cornell University)|Aug 16, 2018

Sparse and Compressive Sensing Techniques28 references69 citations

TL;DR

The paper proves that regular convolutions can be exactly expressed as a sum of depthwise separable convolutions and introduces Network Decoupling (ND), a training-free deploy-time method to accelerate CNNs by converting layers to depthwise separable forms, achieving notable speedups with minimal accuracy loss and synergizing with other zero-data methods.

ABSTRACT

Depthwise separable convolution has shown great efficiency in network design, but requires time-consuming training procedure with full training-set available. This paper first analyzes the mathematical relationship between regular convolutions and depthwise separable convolutions, and proves that the former one could be approximated with the latter one in closed form. We show depthwise separable convolutions are principal components of regular convolutions. And then we propose network decoupling (ND), a training-free method to accelerate convolutional neural networks (CNNs) by transferring pre-trained CNN models into the MobileNet-like depthwise separable convolution structure, with a promising speedup yet negligible accuracy loss. We further verify through experiments that the proposed method is orthogonal to other training-free methods like channel decomposition, spatial decomposition, etc. Combining the proposed method with them will bring even larger CNN speedup. For instance, ND itself achieves about 2X speedup for the widely used VGG16, and combined with other methods, it reaches 3.7X speedup with graceful accuracy degradation. We demonstrate that ND is widely applicable to classification networks like ResNet, and object detection network like SSD300.

Motivation & Objective

• Analyze the mathematical relationship between regular and depthwise separable convolutions.
• Derive a closed-form, data-free tensor decomposition to decouple regular convolutions.
• Introduce Network Decoupling (ND) as a training-free deployment-time optimization.
• Demonstrate ND’s speedups on VGG16, ResNet, and SSD300 and its complementarity with other methods.

Proposed method

• Model regular convolution as a 4D tensor W and compare with depthwise separable (DW+PW or PW+DW) factorization.
• Prove that any regular convolution can be expressed as a sum of depthwise separable convolutions with K terms (K ≤ k_h k_w).
• Introduce exact network decoupling (ND) and the decoupling rank K as a measure of coupling redundancy.
• Propose an approximated network decoupling by using the top-T components (T ≤ K) for practical speedups.
• Show ND is data-free and complementary to other training-free methods (channel/spatial decomposition, channel pruning).
• Provide guidance on choosing DW+PW vs PW+DW based on energy distribution across top singular vectors.

Experimental results

Research questions

• RQ1 Can regular convolutions be exactly represented as a combination of depthwise separable convolutions without increasing computation?
• RQ2 What is the decoupling rank K and how does it relate to the redundancy in convolutional kernels?
• RQ3 Can we achieve meaningful speedups without training data, and how does ND interact with other zero-data acceleration methods?
• RQ4 How does ND perform on common architectures (VGG16, ResNet, SSD300) and with fine-tuning?

Key findings

• Regular convolutions can be losslessly expanded to a sum of depthwise separable convolutions without increasing computational cost (for DW+PW or PW+DW).
• The decoupling rank K is bounded by k_h k_w and reflects coupling redundancy; smaller K yields larger potential speedup.
• An approximated ND using top-T components achieves substantial speedups with controllable accuracy loss; e.g., top-4 components capture most energy in many layers.
• ND yields ~1.8x speedup on VGG16 (single method) and up to ~3.7x when combined with other training-free methods, with modest accuracy degradation.
• ND applied to ResNet18 gives ~1.5x speedup with ~1.4% top-5 drop; combining ND with other methods further improves acceleration.
• ND applied to SSD300 with VGG16 backbone yields ~2.1x speedup with minor mAP loss; combining with other methods improves to ~2.0x speedup with small mAP loss.
• Extremely decoupled networks can be fine-tuned to recover accuracy, achieving larger speedups (e.g., 3.9x on VGG16 with 10 epochs) and modest accuracy changes, outperforming some tuning-free baselines.