[Paper Review] Learning Structured Sparsity in Deep Neural Networks

Wei Wen, Chunpeng Wu+3 more|arXiv (Cornell University)|Aug 12, 2016

Domain Adaptation and Few-Shot Learning18 references468 citations

TL;DR

The paper introduces Structured Sparsity Learning (SSL), a group Lasso-based regularization method that learns compact, hardware-friendly structures (filters, channels, shapes, and depth) in DNNs, achieving substantial practical speedups with preserved or improved accuracy.

ABSTRACT

High demand for computation resources severely hinders deployment of large-scale Deep Neural Networks (DNN) in resource constrained devices. In this work, we propose a Structured Sparsity Learning (SSL) method to regularize the structures (i.e., filters, channels, filter shapes, and layer depth) of DNNs. SSL can: (1) learn a compact structure from a bigger DNN to reduce computation cost; (2) obtain a hardware-friendly structured sparsity of DNN to efficiently accelerate the DNNs evaluation. Experimental results show that SSL achieves on average 5.1x and 3.1x speedups of convolutional layer computation of AlexNet against CPU and GPU, respectively, with off-the-shelf libraries. These speedups are about twice speedups of non-structured sparsity; (3) regularize the DNN structure to improve classification accuracy. The results show that for CIFAR-10, regularization on layer depth can reduce 20 layers of a Deep Residual Network (ResNet) to 18 layers while improve the accuracy from 91.25% to 92.60%, which is still slightly higher than that of original ResNet with 32 layers. For AlexNet, structure regularization by SSL also reduces the error by around ~1%. Open source code is in https://github.com/wenwei202/caffe/tree/scnn

Motivation & Objective

• Reduce computation cost by learning compact DNN structures from larger models.
• Produce hardware-friendly structured sparsity to enable efficient DNN acceleration.
• Investigate regularization of multiple DNN structures (filters, channels, shapes, depth) with a single framework.
• Demonstrate accuracy preservation or improvement when learning compressed structures.

Proposed method

• Formulate a generic SSL objective combining data loss with standard regularization and a layer-wise grouped sparsity term using Group Lasso.
• Apply group Lasso to different structural groups: filters, channels, filter shapes, and network depth.
• Derive practical variants such as 2D-filter-wise sparsity and shape-wise sparsity to align with GEMM-based implementations.
• Incorporate depth regularization with shortcut connections to avoid breaking information flow when entire layers are pruned.
• Evaluate SSL on MNIST (LeNet, MLP), CIFAR-10 (ConvNet, ResNet), and ImageNet (AlexNet) using off-the-shelf libraries.

Experimental results

Research questions

• RQ1 Can SSL learn compact, structured sparsity patterns without sacrificing accuracy on standard benchmarks?
• RQ2 How do different structured groups (filters, channels, shapes, depth) contribute to actual computation reductions?
• RQ3 What are the practical speedups achievable on CPU and GPU with off-the-shelf libraries when using SSL?
• RQ4 Can SSL be integrated with existing acceleration techniques (e.g., GEMM-based implementations) to maximize speedups?

Key findings

• SSL achieves on average 5.1× speedups on CPU and 3.1× on GPU for AlexNet’s convolutional layers compared to baseline using off-the-shelf libraries.
• SSL can reduce ResNet depth from 20 to 14 layers with lower error (SSL-14) than the original 20-layer ResNet, and 18/32-layer variants reach 7.40%–7.51% error on CIFAR-10.
• On CIFAR-10, SSL regularization can reduce AlexNet’s error by about 1% in some configurations and enable smaller, faster models.
• In MNIST, SSL learns compact structures with comparable error to baselines while significantly reducing filters and channels (e.g., LeNet results show notable FLOP reductions with similar accuracy).
• 2D-filter-wise sparsity reduces FLOP by 30%–40% in AlexNet without accuracy loss, and shape-wise sparsity can achieve similar reductions, illustrating effective alignment with GEMM-based computation.