[Paper Review] SNIP: Single-shot Network Pruning based on Connection Sensitivity

Namhoon Lee, Thalaiyasingam Ajanthan+1 more|arXiv (Cornell University)|Oct 4, 2018

Advanced Neural Network Applications29 references363 citations

TL;DR

SNIP identifies important network connections before training by measuring how pruning a connection would affect loss, then prunes to a target sparsity and trains the resulting sparse network, achieving near-original accuracy with extreme sparsity across various architectures.

ABSTRACT

Pruning large neural networks while maintaining their performance is often desirable due to the reduced space and time complexity. In existing methods, pruning is done within an iterative optimization procedure with either heuristically designed pruning schedules or additional hyperparameters, undermining their utility. In this work, we present a new approach that prunes a given network once at initialization prior to training. To achieve this, we introduce a saliency criterion based on connection sensitivity that identifies structurally important connections in the network for the given task. This eliminates the need for both pretraining and the complex pruning schedule while making it robust to architecture variations. After pruning, the sparse network is trained in the standard way. Our method obtains extremely sparse networks with virtually the same accuracy as the reference network on the MNIST, CIFAR-10, and Tiny-ImageNet classification tasks and is broadly applicable to various architectures including convolutional, residual and recurrent networks. Unlike existing methods, our approach enables us to demonstrate that the retained connections are indeed relevant to the given task.

Motivation & Objective

• Motivate pruning large networks to reduce memory and computation without substantial performance loss.
• Propose a data-dependent saliency criterion that identifies structurally important connections before training.
• Enable pruning in a single shot at initialization, removing the need for pretraining and iterative prune–train cycles.
• Demonstrate robustness of the method across diverse architectures and datasets.

Proposed method

• Define binary connectivity indicators c and weight vector w to model pruning as sparsity control.
• Compute connection sensitivity s_j as the normalized magnitude of the derivative of the loss with respect to c_j: s_j = |g_j(w; D)| / sum_k |g_k(w; D)|, where g_j = ∂L(c ⊙ w; D)/∂c_j |_{c=1}.
• Retain the top-kappa connections by setting c_j = 1 for the kappa largest s_j and 0 otherwise.
• Prune once at initialization by solving min_w L(c ⊙ w; D) with the pruned mask, then train the sparse network in the standard way.
• Initialization is performed with variance-scaled weights to ensure consistent gradient signals across architectures.
• Use a mini-batch of data to compute saliency, with flexibility to accumulate saliency over batches or use a validation/whole-data set if memory allows.
• Algorithm SNIP proceeds in four steps: compute s_j on a mini-batch, derive the pruning mask from s_j, optimize w under the mask, and finally apply the mask to the trained weights.

Experimental results

Research questions

• RQ1 Can a data-dependent saliency criterion identify important connections prior to training?
• RQ2 How much sparsity can be achieved without noticeable loss in accuracy across different architectures and datasets?
• RQ3 Is pruning at initialization robust to architecture type (CNNs, residuals, RNNs) and initialization schemes?
• RQ4 Does the method reveal that retained connections are genuinely task-relevant when inspected with respect to input data?
• RQ5 How does using a mini-batch for saliency computation affect pruning outcomes and final performance?

Key findings

• SNIP yields extremely sparse models with virtually the same accuracy as reference networks on MNIST, CIFAR-10, and Tiny-ImageNet across varied architectures.
• Pruning levels up to 98% for LeNet-300-100 and 99% for LeNet-5-Caffe still achieve comparable or better accuracy than dense baselines.
• The method generalizes to convolutional, residual, and recurrent networks without architecture-specific pruning schedules or pretraining.
• Saliency-based pruning reveals that retained connections align with discriminative input features, indicating genuine task relevance.
• Performance remains competitive with or superior to many existing pruning methods, while requiring no additional hyperparameters or pretraining.