[Paper Review] Lottery Jackpots Exist in Pre-trained Models

Yuxin Zhang, Mingbao Lin+3 more|arXiv (Cornell University)|Apr 18, 2021

Advanced Neural Network Applications48 references4 citations

TL;DR

This paper introduces 'lottery jackpots'—high-performing, sparse sub-networks that exist within unexpanded, pre-trained models like VGGNet-19 and ResNet-50 without requiring weight fine-tuning. By leveraging magnitude-based mask initialization and a novel short restriction mechanism for efficient mask search, the method achieves state-of-the-art accuracy with up to 90% parameter reduction in under 5 training epochs on ImageNet, significantly reducing pruning cost compared to existing methods.

ABSTRACT

Network pruning is an effective approach to reduce network complexity with acceptable performance compromise. Existing studies achieve the sparsity of neural networks via time-consuming weight training or complex searching on networks with expanded width, which greatly limits the applications of network pruning. In this paper, we show that high-performing and sparse sub-networks without the involvement of weight training, termed "lottery jackpots", exist in pre-trained models with unexpanded width. Furthermore, we improve the efficiency for searching lottery jackpots from two perspectives. Firstly, we observe that the sparse masks derived from many existing pruning criteria have a high overlap with the searched mask of our lottery jackpot, among which, the magnitude-based pruning results in the most similar mask with ours. Consequently, our searched lottery jackpot removes 90% weights in ResNet-50, while it easily obtains more than 70% top-1 accuracy using only 5 searching epochs on ImageNet. In compliance with this insight, we initialize our sparse mask using the magnitude-based pruning, resulting in at least 3x cost reduction on the lottery jackpot searching while achieving comparable or even better performance. Secondly, we conduct an in-depth analysis of the searching process for lottery jackpots. Our theoretical result suggests that the decrease in training loss during weight searching can be disturbed by the dependency between weights in modern networks. To mitigate this, we propose a novel short restriction method to restrict change of masks that may have potential negative impacts on the training loss. Our code is available at https://github.com/zyxxmu/lottery-jackpots.

Motivation & Objective

• To investigate whether high-performing, sparse sub-networks (lottery jackpots) exist in pre-trained models without weight fine-tuning.
• To reduce the computational cost of searching for such sub-networks, which is typically high in existing methods.
• To improve the efficiency and convergence of lottery jackpot search by introducing better initialization and a dynamic restriction mechanism for mask updates.
• To demonstrate that lottery jackpots in pre-trained models can match or exceed the performance of state-of-the-art weight-training-based pruning methods.

Proposed method

• Proposes a new method, SR-popup, to efficiently search for lottery jackpots in pre-trained models by initializing the sparse mask using magnitude-based pruning, which shows high overlap with the final jackpot mask.
• Introduces a short restriction mechanism that limits weight swapping during mask search to prevent negative impacts on training loss, based on theoretical analysis of weight dependency in deep networks.
• Employs a dynamic schedule for restricting the number of weight swaps per iteration, starting with high exploration and gradually increasing restriction to improve convergence and reduce oscillation.
• Uses a relaxed mask formulation with a differentiable relaxation to enable end-to-end training of the mask while preserving the original pre-trained weights.
• Leverages the edge-popup algorithm as a base framework but accelerates it through improved initialization and restriction, reducing search cost by at least 3×.
• Validates the method on CIFAR-10 and ImageNet using VGGNet-19 and ResNet-50, demonstrating high performance with minimal retraining.

Figure 1 : Training/Search cost v.s. top-1 accuracy of ResNet-50 [ 21 ] with a sparse rate of $90\%$ on ImageNet [ 22 ] . Search epoch differs from training epoch in that it only trains the mask for indicating the removal or preserve of weights, without modifying the weight value. Our method can qui

Experimental results

Research questions

• RQ1 Do high-performing, sparse sub-networks (lottery jackpots) exist in unexpanded, pre-trained models without requiring weight fine-tuning?
• RQ2 Can the search cost for such lottery jackpots be significantly reduced while maintaining or improving performance?
• RQ3 How does the choice of initial mask (e.g., magnitude-based pruning) affect the efficiency and outcome of lottery jackpot search?
• RQ4 What is the impact of restricting weight swaps during mask search, and how should the restriction schedule be designed for optimal convergence?
• RQ5 Can a theoretically grounded mechanism mitigate the negative effects of weight dependency during the mask search process?

Key findings

• A lottery jackpot with only 10% of the original parameters in VGGNet-19 achieves the same performance as the full model on CIFAR-10 without any weight modification.
• The proposed method reduces the cost of lottery jackpot search by at least 3× compared to baseline edge-popup, while achieving comparable or better performance.
• Magnitude-based pruning produces sparse masks with the highest overlap to the final lottery jackpot mask, justifying its use as an effective initialization strategy.
• The proposed short restriction mechanism leads to faster convergence and reduced oscillation during mask search, outperforming random and inverse scheduling variants.
• On ImageNet, the method achieves over 70% top-1 accuracy with 90% sparsity in ResNet-50 using only 5 training epochs, demonstrating high efficiency.
• Ablation studies confirm that both the magnitude-based initialization and the dynamic restriction schedule are essential components for optimal performance.