[Paper Review] Group Sparsity: The Hinge Between Filter Pruning and Decomposition for Network Compression

Yawei Li, Shuhang Gu+3 more|Lirias (KU Leuven)|Mar 19, 2020

Sparse and Compressive Sensing TechniquesEngineering54 references18 citations

TL;DR

This paper introduces a unified framework for neural network compression using group sparsity as a hinge between filter pruning and low-rank decomposition. By applying group sparsity regularization to a learnable sparsity-inducing matrix, the method dynamically switches between pruning (via column sparsity) and decomposition (via row sparsity), achieving state-of-the-art accuracy-compression trade-offs on CIFAR-10, CIFAR-100, and ImageNet benchmarks with joint, end-to-end optimization.

ABSTRACT

In this paper, we analyze two popular network compression techniques, i.e. filter pruning and low-rank decomposition, in a unified sense. By simply changing the way the sparsity regularization is enforced, filter pruning and low-rank decomposition can be derived accordingly. This provides another flexible choice for network compression because the techniques complement each other. For example, in popular network architectures with shortcut connections (e.g. ResNet), filter pruning cannot deal with the last convolutional layer in a ResBlock while the low-rank decomposition methods can. In addition, we propose to compress the whole network jointly instead of in a layer-wise manner. Our approach proves its potential as it compares favorably to the state-of-the-art on several benchmarks.

Motivation & Objective

• To unify filter pruning and low-rank decomposition under a single optimization framework for neural network compression.
• To address the limitation of filter pruning in residual networks, where pruning the last convolutional layer in a ResBlock disrupts skip connections.
• To enable joint compression of the entire network rather than layer-wise compression, improving overall model efficiency.
• To leverage the complementary strengths of pruning (parameter reduction) and decomposition (computation savings) through a flexible, unified mechanism.
• To achieve state-of-the-art performance in accuracy and FLOP reduction across multiple benchmarks.

Proposed method

• Introduce a learnable sparsity-inducing matrix A that acts as a hinge between pruning and decomposition by enabling group sparsity on its columns or rows.
• Apply group sparsity regularization to the columns of A to reduce output channels, equivalent to filter pruning via matrix product W × A^c.
• Apply group sparsity regularization to the rows of A to reduce inner dimensions, enabling low-rank decomposition with two lightweight convolutions W^r and A^r.
• Use proximal gradient descent with adaptive learning rate, layer balancing, and regularization factor annealing to solve the optimization problem.
• Employ knowledge distillation during fine-tuning to preserve accuracy during compression.
• Optimize the entire network jointly instead of layer-wise, improving global compression efficiency.

Experimental results

Research questions

• RQ1 Can filter pruning and low-rank decomposition be unified under a single optimization framework?
• RQ2 How can group sparsity in a sparsity-inducing matrix enable dynamic switching between pruning and decomposition modes?
• RQ3 Does joint, end-to-end compression of the entire network outperform layer-wise compression strategies?
• RQ4 Can the proposed method overcome the limitations of filter pruning in residual networks, particularly in layers with skip connections?
• RQ5 What is the impact of regularization type and annealing on compression performance and generalization?

Key findings

• On CIFAR-10, the proposed method achieves a Top-1 error rate of 6.37% at 50% compression ratio, outperforming SSS (25.82%) and other SOTA methods.
• For ResNet-164 on CIFAR-100, the method with regularization annealing achieves better performance than CGES and SSS, even without distillation.
• On ImageNet-2012, the method achieves state-of-the-art trade-off between Top-1 error rate and FLOP compression ratio, outperforming methods like KSE and SSS.
• The ablation study shows that ℓ₁ regularization with T=0.005 and α=0.01 yields the best performance, with minimal sensitivity to hyperparameter changes.
• The method reduces Top-1 error by 0.41% and 1.51% compared to prior work on VGG and DenseNet, respectively, under similar compression ratios.
• FLOP and parameter reduction are significantly better than KSE at all compression levels, with the proposed method achieving lower FLOPs at comparable error rates.