[Paper Review] On Iterative Neural Network Pruning, Reinitialization, and the Similarity of Masks

M. Paganini, Jessica Zosa Forde|arXiv (Cornell University)|Jan 14, 2020

Domain Adaptation and Few-Shot Learning40 references4 citations

TL;DR

This paper investigates iterative neural network pruning with weight rewinding, showing that multiple high-performing, sparse sub-networks (winning tickets) can emerge depending on the pruning technique. It demonstrates that magnitude-based unstructured pruning with rewinding produces connectivity patterns similar to structured pruning, and that weight stability correlates strongly with performance, even without late resetting.

ABSTRACT

We examine how recently documented, fundamental phenomena in deep learning models subject to pruning are affected by changes in the pruning procedure. Specifically, we analyze differences in the connectivity structure and learning dynamics of pruned models found through a set of common iterative pruning techniques, to address questions of uniqueness of trainable, high-sparsity sub-networks, and their dependence on the chosen pruning method. In convolutional layers, we document the emergence of structure induced by magnitude-based unstructured pruning in conjunction with weight rewinding that resembles the effects of structured pruning. We also show empirical evidence that weight stability can be automatically achieved through apposite pruning techniques.

Motivation & Objective

• To investigate whether multiple high-performing, sparse sub-networks (winning tickets) exist within the same over-parametrized network, independent of pruning method.
• To analyze how different iterative pruning techniques affect the connectivity structure and learning dynamics of pruned models.
• To evaluate whether weight rewinding leads to structural and performance advantages over fine-tuning in pruned networks.
• To assess the role of weight stability in determining the success of iterative pruning and its correlation with model performance.
• To compare the effectiveness of unstructured vs. structured pruning, and random vs. magnitude-based pruning, in identifying trainable sparse sub-networks.

Proposed method

• Empirically compare iterative pruning techniques on small architectures (LeNet, AlexNet, VGG-11) using MNIST and CIFAR-10.
• Apply magnitude-based unstructured pruning combined with weight rewinding to preserve initial weight distributions across pruning iterations.
• Use structured pruning along input channels to compare connectivity patterns with unstructured pruning + rewinding.
• Measure weight stability by computing the absolute difference in weight values between consecutive pruning iterations.
• Evaluate performance via test accuracy and analyze generalization behavior across different pruning methods.
• Visualize and compare the similarity of pruned network masks (i.e., connectivity patterns) across different pruning procedures.

Experimental results

Research questions

• RQ1 Do different pruning techniques yield distinct, high-performing sub-networks (i.e., multiple winning tickets) within the same over-parametrized network?
• RQ2 Does weight rewinding after each pruning iteration produce connectivity structures similar to structured pruning, particularly in terms of input feature selection?
• RQ3 How does the choice of pruning method affect the structural similarity of the final masks and the stability of weights during pruning?
• RQ4 Is there a correlation between weight stability during pruning and the final model performance?
• RQ5 Do different pruning techniques lead to models that make different errors on held-out test sets, suggesting utility in ensemble methods?

Key findings

• Multiple high-performing, sparse sub-networks (winning tickets) can be found within the same over-parametrized network, depending on the pruning technique used, indicating non-uniqueness of winning tickets.
• Magnitude-based unstructured pruning with weight rewinding produces connectivity patterns that closely resemble those from structured pruning along input dimensions, suggesting an input feature selection effect.
• Weight stability during pruning correlates strongly with final model performance, and can be achieved through appropriate pruning techniques even without late resetting.
• The 'hybrid' pruning method (combining structured and unstructured pruning) exhibits the highest weight stability, with minimal ordinality changes in weights across pruning iterations.
• Random structured pruning outperforms random unstructured pruning, indicating that networks are more robust to random channel/unit removal than to random connection removal.
• Different pruning techniques lead to models that make different errors on test sets, implying that ensembling such models could improve generalization.