[Paper Review] CondenseNet: An Efficient DenseNet using Learned Group Convolutions

Gao Huang, Shichen Liu+2 more|arXiv (Cornell University)|Nov 25, 2017

Advanced Neural Network Applications44 references95 citations

TL;DR

CondenseNet introduces learned group convolutions to prune redundant DenseNet connections, enabling regular group convolutions for fast inference with substantially fewer FLOPs and parameters while maintaining accuracy.

ABSTRACT

Deep neural networks are increasingly used on mobile devices, where computational resources are limited. In this paper we develop CondenseNet, a novel network architecture with unprecedented efficiency. It combines dense connectivity with a novel module called learned group convolution. The dense connectivity facilitates feature re-use in the network, whereas learned group convolutions remove connections between layers for which this feature re-use is superfluous. At test time, our model can be implemented using standard group convolutions, allowing for efficient computation in practice. Our experiments show that CondenseNets are far more efficient than state-of-the-art compact convolutional networks such as MobileNets and ShuffleNets.

Motivation & Objective

• Promote feature re-use while reducing computational cost in DenseNets by learning which input features are essential for each layer.
• Develop a pruning mechanism that automatically induces group-level sparsity during training to enable efficient group convolutions at test time.
• Investigate architecture-level changes (IGR growth, full dense connectivity) to further boost efficiency without sacrificing accuracy.
• Evaluate CondenseNet against mobile- and server-oriented compact CNNs on CIFAR-10/100 and ImageNet.
• Provide an end-to-end training and deployment pipeline that relies on standard group convolutions for efficient inference.

Proposed method

• Introduce learned group convolution (LGC) by splitting 1x1 filter outputs into G groups and learning input feature groupings during training.
• Use a condensation process with a condensation factor C to iteratively prune 1x1 filter weights in each group via a group-lasso regularizer that encourages same sparsity pattern within a group.
• Replace pruned weights with a masking mechanism during training to avoid sparse matrix operations and maintain GPU efficiency.
• After training, convert the sparsified layer into a regular group convolution via an index layer that rearranges features for efficient deployment.
• Optionally adopt architecture tweaks: exponentially increasing growth rate (IGR) and fully dense connectivity across blocks to enhance feature reuse and efficiency.
• Train with cosine learning rate schedule and standard DenseNet-like blocks, adapted for learned group convolutions.

Experimental results

Research questions

• RQ1 How can learned group convolutions prune redundant layer connections without harming accuracy compared to DenseNets?
• RQ2 What are the efficiency gains (in FLOPs and parameters) when replacing dense connectivity with learned group convolutions on CIFAR-10/100 and ImageNet?
• RQ3 Do architectural changes like IGR growth and full dense connectivity improve performance and efficiency across datasets?
• RQ4 Can CondenseNet match or exceed the efficiency of MobileNets and ShuffleNets at comparable accuracy on ImageNet?
• RQ5 Does the learned pruning strategy maintain stable connectivity patterns across different training runs?

Key findings

• CondenseNet achieves substantially lower FLOPs than DenseNets at similar accuracy, e.g., CondenseNet with 275M FLOPs achieves ~29% top-1 error on ImageNet, comparable to MobileNet with roughly double compute.
• On CIFAR-10/100, CondenseNet variants require approximately 8x fewer parameters and FLOPs to reach comparable accuracy with dense networks and pruning baselines.
• Learned group convolutions significantly improve efficiency over traditional DenseNet-like architectures and outperform several state-of-the-art compact CNNs at similar accuracy levels.
• CondenseNet’s light variant (condense-light) performs on par with NASNet-A on CIFAR-10 in terms of accuracy while using far fewer parameters and FLOPs.
• Inference time on ARM shows CondenseNet (G=C=8) with 274M FLOPs delivers about 2x faster runtime than MobileNet at similar accuracy; with 529M FLOPs it achieves competitive top-1 accuracy.
• Ablation studies indicate higher group count G and condensation factors >1 improve efficiency and accuracy, with stable global connectivity patterns across runs.