[Paper Review] Sparse Networks from Scratch: Faster Training without Losing Performance

Tim Dettmers, Luke Zettlemoyer|arXiv (Cornell University)|Jul 10, 2019

Advanced Neural Network Applications38 references191 citations

TL;DR

The paper introduces sparse momentum to train fully sparse networks from random initialization, achieving dense performance with up to 5.61x training speedups on CIFAR-10 and competitive results on ImageNet.

ABSTRACT

We demonstrate the possibility of what we call sparse learning: accelerated training of deep neural networks that maintain sparse weights throughout training while achieving dense performance levels. We accomplish this by developing sparse momentum, an algorithm which uses exponentially smoothed gradients (momentum) to identify layers and weights which reduce the error efficiently. Sparse momentum redistributes pruned weights across layers according to the mean momentum magnitude of each layer. Within a layer, sparse momentum grows weights according to the momentum magnitude of zero-valued weights. We demonstrate state-of-the-art sparse performance on MNIST, CIFAR-10, and ImageNet, decreasing the mean error by a relative 8%, 15%, and 6% compared to other sparse algorithms. Furthermore, we show that sparse momentum reliably reproduces dense performance levels while providing up to 5.61x faster training. In our analysis, ablations show that the benefits of momentum redistribution and growth increase with the depth and size of the network. Additionally, we find that sparse momentum is insensitive to the choice of its hyperparameters suggesting that sparse momentum is robust and easy to use.

Motivation & Objective

• Motivate sparse learning as a way to accelerate training without sacrificing accuracy.
• Develop and validate a practical algorithm (sparse momentum) that maintains sparsity throughout training.
• Demonstrate state-of-the-art sparse performance on MNIST, CIFAR-10, and ImageNet.
• Quantify speedups and analyze robustness to hyperparameters and network depth.

Proposed method

• Use exponentially smoothed gradients (momentum) to measure layer and weight efficiency.
• Prune a fraction of weights with smallest magnitude after each epoch.
• Redistribute pruned weights across layers proportional to mean momentum magnitude per layer.
• Grow new weights in missing connections with highest momentum magnitude.
• Compare against compression and other sparse-training methods across datasets.
• Evaluate with dense baselines to assess retention of dense performance and compute speedups.

Experimental results

Research questions

• RQ1 Can sparse networks trained from random initialization achieve dense performance without re-training?
• RQ2 Does momentum-based redistribution and growth improve learning efficiency and accuracy as networks scale in depth and size?
• RQ3 What are the speedups and weight requirements to match dense performance across MNIST, CIFAR-10, and ImageNet?
• RQ4 How robust is sparse momentum to prune rate and momentum hyperparameters?

Key findings

• Sparse momentum achieves dense-equivalent performance on MNIST, CIFAR-10, and ImageNet with far fewer weights (e.g., 5–10% for several models).
• On CIFAR-10, sparsity enables 2.74x–5.61x speedups for sparse convolution (up to 5.61x for WRNs).
• For ImageNet, sparse momentum attains better Top-1 accuracy than several sparse baselines, with 10–20% weights achieving competitive results.
• Momentum redistribution and growth become more critical as networks deepen (e.g., ResNet-50 on ImageNet).
• The method is robust to prune rates (0.2–0.5) and momentum values (0.7–0.9), requiring minimal hyperparameter tuning.