[Paper Review] The Power of Sparsity in Convolutional Neural Networks

Soravit Changpinyo, Mark Sandler+1 more|arXiv (Cornell University)|Feb 21, 2017

Advanced Neural Network Applications38 references90 citations

TL;DR

The paper introduces a sparse random connectivity approach between convolutional filters in CNNs, showing it can achieve similar or better accuracy than dense networks at high compression, and explores incremental training to densify connections over time.

ABSTRACT

Deep convolutional networks are well-known for their high computational and memory demands. Given limited resources, how does one design a network that balances its size, training time, and prediction accuracy? A surprisingly effective approach to trade accuracy for size and speed is to simply reduce the number of channels in each convolutional layer by a fixed fraction and retrain the network. In many cases this leads to significantly smaller networks with only minimal changes to accuracy. In this paper, we take a step further by empirically examining a strategy for deactivating connections between filters in convolutional layers in a way that allows us to harvest savings both in run-time and memory for many network architectures. More specifically, we generalize 2D convolution to use a channel-wise sparse connection structure and show that this leads to significantly better results than the baseline approach for large networks including VGG and Inception V3.

Motivation & Objective

• Motivate reducing CNN size and compute for deployment on resource-limited devices.
• Investigate whether fixed sparse channel-wise connections can substitute dense connections without substantial accuracy loss.
• Compare sparse random connectivity to depth multiplier at various compression rates across multiple architectures.
• Explore incremental training where connections are progressively activated to accelerate training and enable growth.
• Provide practical guidance for hardware-aware network compression via fixed sparse patterns.

Proposed method

• Generalize 2D convolution to channel-wise sparse connection structures with fixed spatial connectivity across the feature maps.
• Define sparse random connectivity where each output channel connects to only a fraction of input channels, preserving spatial convolution structure.
• Introduce incremental training by starting with a very sparse network and progressively densifying connections over training.
• Compare sparse random connectivity against depth multiplier at multiple compression levels on MNIST, CIFAR-10, and ImageNet (Inception-V3 and VGG-16n).
• Evaluate performance using parameters (Params) and multiply-adds (MAdds) as primary metrics, with accuracy measured as P@1 for ImageNet experiments.

Experimental results

Research questions

• RQ1 Does a fixed, channel-wise sparse connectivity pattern in CNNs preserve accuracy relative to dense architectures at high compression?
• RQ2 How does sparse random connectivity compare to depth multiplier in terms of parameter efficiency and accuracy across networks of varying sizes?
• RQ3 Can incremental training that densifies sparse connections during training speed up training while preserving or improving final accuracy?
• RQ4 What are the practical implications for hardware-friendly CNN design when using fixed sparse connectivity patterns?

Key findings

• Sparse random connectivity often matches or exceeds the accuracy of dense convolutions with the same parameter budget, especially at high compression.
• On Inception-V3, sparse networks achieve substantially fewer parameters and maintain competitive accuracy across a wide range of sparsity levels (e.g., up to 77% P@1 at 5.70 B MAdds and 22 M Params).
• For Inception-V3, mixed configurations (e.g., 0.50/0.01 sparsity) reach 40.3 P@1 with 90k params and 43.0 MAdds, while higher sparsity (0.003) yields 46.1 P@1 with 158k params and 82.0 MAdds.
• On ImageNet with Inception-V3 and VGG-16n, sparse patterns enable dramatic reductions in parameters (e.g., from tens of millions to hundreds of thousands) while retaining or surpassing the accuracy of denser counterparts at comparable compute budgets.
• Incremental training, where connections are added over time, allows the sparse networks to catch up or surpass full dense models in some settings and accelerates early training phases due to reduced computation early on.