[Paper Review] Why Random Pruning Is All We Need to Start Sparse

Advait Gadhikar, Sohum Mukherjee+1 more|arXiv (Cornell University)|Oct 5, 2022

Music and Audio Processing4 citations

TL;DR

This paper demonstrates that randomly pruned neural networks—specifically, randomly masked (ER) networks—can serve as effective, computationally efficient starting points for training sparse models, achieving performance comparable to dense networks and state-of-the-art lottery ticket pruning. Theoretically, it proves that a random network wider by a logarithmic factor in inverse sparsity can approximate any target network, enabling 'sparse-to-sparse' training that avoids dense initialization and reduces computational cost.

ABSTRACT

Random masks define surprisingly effective sparse neural network models, as has been shown empirically. The resulting sparse networks can often compete with dense architectures and state-of-the-art lottery ticket pruning algorithms, even though they do not rely on computationally expensive prune-train iterations and can be drawn initially without significant computational overhead. We offer a theoretical explanation of how random masks can approximate arbitrary target networks if they are wider by a logarithmic factor in the inverse sparsity $1 / \log(1/ ext{sparsity})$. This overparameterization factor is necessary at least for 3-layer random networks, which elucidates the observed degrading performance of random networks at higher sparsity. At moderate to high sparsity levels, however, our results imply that sparser networks are contained within random source networks so that any dense-to-sparse training scheme can be turned into a computationally more efficient sparse-to-sparse one by constraining the search to a fixed random mask. We demonstrate the feasibility of this approach in experiments for different pruning methods and propose particularly effective choices of initial layer-wise sparsity ratios of the random source network. As a special case, we show theoretically and experimentally that random source networks also contain strong lottery tickets.

Motivation & Objective

• To explain why random pruning at initialization performs surprisingly well in practice despite lacking iterative pruning or retraining.
• To provide a theoretical foundation for the universal approximation capability of random sparse networks under controlled overparameterization.
• To demonstrate that sparse-to-sparse training—starting from a randomly pruned network—can match or exceed performance of dense-to-sparse training.
• To show that strong lottery tickets (SLTs) can be found directly within sparse random source networks, reducing search cost.
• To propose optimal layer-wise sparsity ratios for random source networks that improve efficiency and performance.

Proposed method

• Theoretical analysis proves that a random 3-layer network can approximate any target network if its width exceeds the target by a factor of O(1/log(1/sparsity)), establishing a necessary overparameterization condition.
• Derives a lower bound on required width for random 1-hidden-layer networks, showing that logarithmic overparameterization is necessary for universal approximation.
• Proposes 'sparse-to-sparse' training: initializing with a fixed random sparse mask (ER) and applying pruning algorithms like IMP or continuous sparsification.
• Introduces layerwise sparsity ratios for the initial random network that improve performance and efficiency, validated across multiple architectures and datasets.
• Empirically evaluates the method on ImageNet, GCNs, algorithmic data, and tabular data using iterative magnitude pruning (IMP) and other methods.
• Demonstrates that strong lottery tickets (SLTs) exist within sparse random networks, enabling initialization-free training.

Experimental results

Research questions

• RQ1 Can random sparse networks approximate any target network, and what level of overparameterization is required?
• RQ2 Why does random pruning at initialization outperform more complex pruning schemes in practice?
• RQ3 Is it possible to train sparse models effectively without starting from a dense network?
• RQ4 Do strong lottery tickets exist within sparse random source networks, and can they be found efficiently?
• RQ5 What are optimal layer-wise sparsity ratios for initial random networks to maximize performance and efficiency?

Key findings

• A random network wider by a factor of O(1/log(1/sparsity)) can approximate any target network, providing theoretical justification for the effectiveness of random pruning.
• The required overparameterization is necessary for 3-layer random networks, explaining performance degradation at very high sparsity levels.
• Sparse-to-sparse training using a fixed random mask achieves comparable or better performance than dense-to-sparse training on CIFAR-10, ImageNet, and GCNs.
• On ImageNet with ResNet-50, starting from a 50% sparse random network still enables training of an 80% sparse model with 71.73% accuracy, matching dense initialization.
• In GCN experiments, starting from 40% sparse initial networks yielded 70% sparse models with 81.9% accuracy, only slightly below the 83.33% of dense initialization.
• On algorithmic data and tabular data, sparse-to-sparse training achieved identical performance to dense-to-sparse training, confirming robustness across domains.