[Paper Review] DSD: Regularizing Deep Neural Networks with Dense-Sparse-Dense Training Flow.

Song Han, Jeff Pool+7 more|arXiv (Cornell University)|Jul 15, 2016

Domain Adaptation and Few-Shot LearningComputer Science22 references70 citations

TL;DR

DSD is a three-phase training flow—Dense-Sparse-Dense—that regularizes deep neural networks by first training a dense network, then pruning unimportant connections under sparsity constraints, and finally retraining the full dense network after reinitializing pruned weights. This method improves accuracy across CNNs, RNNs, and LSTMs on image classification, speech recognition, and caption generation without increasing inference cost.

ABSTRACT

Modern deep neural networks have a large number of parameters, making them very powerful machine learning systems. A critical issue for training such large networks on large-scale data-sets is to prevent overfitting while at the same time providing enough model capacity. We propose DSD, a dense-sparse-dense training flow, for regularizing deep neural networks. In the first D step, we train a dense network to learn which connections are important. In the S step, we regularize the network by pruning the unimportant connections and retrain the network given the sparsity constraint. In the final D step, we increase the model capacity by freeing the sparsity constraint, re-initializing the pruned parameters, and retraining the whole dense network. Experiments show that DSD training can improve the performance of a wide range of CNN, RNN and LSTMs on the tasks of image classification, caption generation and speech recognition. On the Imagenet dataset, DSD improved the absolute accuracy of AlexNet, GoogleNet, VGG-16, ResNet50, ResNet-152 and SqueezeNet by a geo-mean of 2.1 points (Top-1) and 1.4 points (Top-5). On the WSJ’92 and WSJ’93 dataset, DSD improved DeepSpeech2 WER by 0.53 and 1.08 points. On the Flickr-8K dataset, DSD improved the NeuralTalk BLEU score by 2.0 points. DSD training flow produces the same model architecture and doesn’t incur any inference overhead.

Motivation & Objective

• To address overfitting in large deep neural networks trained on large-scale datasets while preserving model capacity.
• To develop a regularization technique that improves generalization without increasing model complexity or inference cost.
• To explore whether structured pruning and subsequent retraining can enhance performance across diverse architectures and tasks.
• To design a training flow that maintains the same final model architecture while improving generalization through dynamic sparsity.

Proposed method

• In the first D step, a fully connected (dense) network is trained to identify important connections through standard backpropagation.
• In the S step, unimportant connections are pruned based on magnitude or importance scores, and the remaining sparse network is fine-tuned under a sparsity constraint.
• In the final D step, the sparsity constraint is removed, pruned weights are reinitialized, and the full dense network is retrained from scratch.
• The method applies to various architectures including CNNs, RNNs, and LSTMs, and is compatible with standard training pipelines.
• The training flow is end-to-end differentiable and does not alter the final model architecture or introduce inference overhead.

Experimental results

Research questions

• RQ1 Can a dense-sparse-dense training flow improve generalization in deep neural networks across diverse architectures and tasks?
• RQ2 Does structured pruning followed by retraining enhance performance without increasing model size or inference cost?
• RQ3 How does DSD compare to standard training in terms of accuracy gains on image classification, speech recognition, and caption generation?
• RQ4 To what extent does the DSD flow improve performance on benchmark datasets like ImageNet, WSJ, and Flickr-8K?

Key findings

• On ImageNet, DSD improved the top-1 accuracy of AlexNet, GoogleNet, VGG-16, ResNet50, ResNet-152, and SqueezeNet by a geometric mean of 2.1 points.
• On the same dataset, DSD improved top-5 accuracy by a geometric mean of 1.4 points across the same models.
• On the WSJ’92 and WSJ’93 datasets, DSD reduced the WER of DeepSpeech2 by 0.53 and 1.08 points, respectively.
• On the Flickr-8K dataset, DSD improved the NeuralTalk BLEU score by 2.0 points.
• The DSD training flow maintains the same final model architecture and incurs no additional inference overhead.