[Paper Review] Memory Bounded Deep Convolutional Networks

Maxwell D. Collins, Pushmeet Kohli|arXiv (Cornell University)|Dec 3, 2014

Advanced Neural Network Applications24 references128 citations

TL;DR

This paper proposes sparsity-inducing regularizers for deep convolutional neural networks (CNNs) that reduce model size and memory usage by encouraging sparse connectivity in convolutional and fully connected layers. Using stochastic gradient descent, the method achieves up to a 4× reduction in memory (e.g., from 233 MB to 58 MB on AlexNet) with minimal accuracy loss, enabling efficient deployment on resource-constrained devices and enabling memory-efficient ensembles.

ABSTRACT

In this work, we investigate the use of sparsity-inducing regularizers during training of Convolution Neural Networks (CNNs). These regularizers encourage that fewer connections in the convolution and fully connected layers take non-zero values and in effect result in sparse connectivity between hidden units in the deep network. This in turn reduces the memory and runtime cost involved in deploying the learned CNNs. We show that training with such regularization can still be performed using stochastic gradient descent implying that it can be used easily in existing codebases. Experimental evaluation of our approach on MNIST, CIFAR, and ImageNet datasets shows that our regularizers can result in dramatic reductions in memory requirements. For instance, when applied on AlexNet, our method can reduce the memory consumption by a factor of four with minimal loss in accuracy.

Motivation & Objective

• To address the high memory and runtime costs of deploying large deep CNNs on resource-constrained devices like mobile platforms.
• To reduce model complexity without sacrificing accuracy by promoting sparse connectivity in convolutional and fully connected layers.
• To enable the construction of accurate ensembles of deep networks within fixed memory budgets.
• To develop a regularization method compatible with standard stochastic gradient descent training pipelines.
• To empirically validate the trade-off between model sparsity, memory usage, and classification accuracy across MNIST, CIFAR-10, and ImageNet.

Proposed method

• Applies sparsity-inducing regularizers—specifically ℓ₀ and ℓ₁ penalties—during CNN training to encourage non-zero weights to be minimized.
• Uses a greedy search strategy to determine optimal layer-wise sparsity distributions that meet a target memory budget.
• Implements regularization via modified weight updates in stochastic gradient descent, ensuring compatibility with existing deep learning frameworks.
• Applies ℓ₀ constraints per layer to enforce a desired number of non-zero parameters, enabling precise control over model size.
• Employs bagging (bootstrap resampling) to train multiple sparse models in an ensemble, with each model constrained to a shared memory budget.
• Uses storage format optimization (Appendix A) to estimate actual memory usage based on sparsity and data types.

Experimental results

Research questions

• RQ1 Can sparsity-inducing regularizers significantly reduce the memory footprint of trained CNNs without substantial accuracy loss?
• RQ2 How does sparsity regularization compare to traditional methods like weight decay or network pruning in terms of memory efficiency and accuracy?
• RQ3 Can sparse models be effectively used to build ensembles within a fixed memory budget, improving generalization?
• RQ4 How does sparsity affect model performance under limited training data, particularly in terms of generalization?
• RQ5 To what extent can sparse models maintain high accuracy while reducing parameter count and memory usage on standard vision benchmarks?

Key findings

• On the ImageNet dataset, the proposed method reduced memory usage from 233 MB (baseline) to 58 MB (sparse model) with only a 1.7% drop in top-1 accuracy (59.3% to 55.60%).
• For AlexNet, the method achieved a 4× reduction in memory consumption (from 233 MB to 58 MB) while maintaining competitive accuracy.
• On CIFAR-10, ensembles of sparse models trained under a fixed parameter budget achieved higher accuracy than a single dense model, with 3 models reaching 77.40% top-1 accuracy.
• As training data size decreased, sparser models outperformed dense models due to improved generalization, confirming the regularization benefit of sparsity.
• The method enables memory-efficient ensembling: for example, a 3-model ensemble with 143,540 total non-zero parameters achieved 77.40% accuracy, demonstrating that sparsity allows more models within a fixed memory budget.
• Sparsity regularization via ℓ₀ constraints allowed the construction of models with up to 70% fewer non-zero parameters while maintaining high performance, especially under data scarcity.