[Paper Review] DSD: Dense-Sparse-Dense Training for Deep Neural Networks

Song Han, Jeff Pool+10 more|arXiv (Cornell University)|Jul 15, 2016

Advanced Neural Network Applications23 references143 citations

TL;DR

DSD training alternates dense, sparse, and re-dense phases to regularize networks and achieve better optimization, improving accuracy/WER/BLEU across CNNs, RNNs, and LSTMs without inference overhead.

ABSTRACT

Modern deep neural networks have a large number of parameters, making them very hard to train. We propose DSD, a dense-sparse-dense training flow, for regularizing deep neural networks and achieving better optimization performance. In the first D (Dense) step, we train a dense network to learn connection weights and importance. In the S (Sparse) step, we regularize the network by pruning the unimportant connections with small weights and retraining the network given the sparsity constraint. In the final D (re-Dense) step, we increase the model capacity by removing the sparsity constraint, re-initialize the pruned parameters from zero and retrain the whole dense network. Experiments show that DSD training can improve the performance for a wide range of CNNs, RNNs and LSTMs on the tasks of image classification, caption generation and speech recognition. On ImageNet, DSD improved the Top1 accuracy of GoogLeNet by 1.1%, VGG-16 by 4.3%, ResNet-18 by 1.2% and ResNet-50 by 1.1%, respectively. On the WSJ'93 dataset, DSD improved DeepSpeech and DeepSpeech2 WER by 2.0% and 1.1%. On the Flickr-8K dataset, DSD improved the NeuralTalk BLEU score by over 1.7. DSD is easy to use in practice: at training time, DSD incurs only one extra hyper-parameter: the sparsity ratio in the S step. At testing time, DSD doesn't change the network architecture or incur any inference overhead. The consistent and significant performance gain of DSD experiments shows the inadequacy of the current training methods for finding the best local optimum, while DSD effectively achieves superior optimization performance for finding a better solution. DSD models are available to download at https://songhan.github.io/DSD.

Motivation & Objective

• Motivate the need for better optimization in large deep networks to avoid overfitting and local minima.
• Introduce a three-phase training flow (dense → sparse → re-dense) to regularize and enhance capacity.
• Demonstrate consistent accuracy and performance gains across CNNs, RNNs, and LSTMs on diverse tasks.

Proposed method

• Train a dense network to learn weights and identify important connections during the first Dense phase.
• Prune low-magnitude weights to induce sparsity and retrain under a fixed sparse mask in the Sparse phase.
• Restore pruned connections, reinitialize them to zero, and retrain the full dense network in the final Dense phase.
• Use a single hyper-parameter (sparsity ratio) for the S phase and keep other hyper-parameters unchanged.
• Demonstrate that the final dense network maintains the same architecture and inference cost as the original model.
• Optionally iterate the Dense-Sparse-Dense cycle for additional gains.

Experimental results

Research questions

• RQ1 Does introducing a sparsity-constrained phase (S) during training improve optimization and generalization beyond standard dense training?
• RQ2 Can re-densifying after pruning (D phase) increase model capacity and achieve better minima without increasing inference cost?
• RQ3 Is the DSD approach effective across diverse architectures (CNNs, RNNs, LSTMs) and tasks (image classification, captioning, speech recognition)?

Key findings

• DSD improves Top-1 error on ImageNet models: GoogLeNet by 1.1%, VGG-16 by 4.3%, ResNet-18 by 1.2%, and ResNet-50 by 1.1%.
• DSD improves NeuralTalk BLEU scores by over 1.7 on Flickr-8K.
• DSD improves DeepSpeech and DeepSpeech2 WER by 2.0% and 1.1% on WSJ’93, respectively.
• Across experiments, DSD yields consistent performance gains over conventional training without inference overhead.
• DSD enables pruning to 25–50% sparsity in the S step with modest pruning in practice while improving accuracy.
• DSD results are robust across CNNs, RNNs, and LSTMs and across tasks including image classification, caption generation, and speech recognition.