[Paper Review] Progressive Skeletonization: Trimming more fat from a network at initialization

Pau de Jorge, Amartya Sanyal+4 more|arXiv (Cornell University)|Jun 16, 2020

Advanced Neural Network ApplicationsComputer Science33 references40 citations

TL;DR

The paper introduces FORCE and two progressive pruning schemes (Iterative SNIP and FORCE) to prune networks at initialization, achieving very high sparsity while preserving trainability and often outperforming SNIP/GRASP, especially at extreme sparsity. It shows the ability to prune up to 99.5% of parameters on some setups.

ABSTRACT

Recent studies have shown that skeletonization (pruning parameters) of networks extit{at initialization} provides all the practical benefits of sparsity both at inference and training time, while only marginally degrading their performance. However, we observe that beyond a certain level of sparsity (approx $95\%$), these approaches fail to preserve the network performance, and to our surprise, in many cases perform even worse than trivial random pruning. To this end, we propose an objective to find a skeletonized network with maximum {\em foresight connection sensitivity} (FORCE) whereby the trainability, in terms of connection sensitivity, of a pruned network is taken into consideration. We then propose two approximate procedures to maximize our objective (1) Iterative SNIP: allows parameters that were unimportant at earlier stages of skeletonization to become important at later stages; and (2) FORCE: iterative process that allows exploration by allowing already pruned parameters to resurrect at later stages of skeletonization. Empirical analyses on a large suite of experiments show that our approach, while providing at least as good a performance as other recent approaches on moderate pruning levels, provides remarkably improved performance on higher pruning levels (could remove up to $99.5\%$ parameters while keeping the networks trainable). Code can be found in https://github.com/naver/force.

Motivation & Objective

• Motivate pruning at initialization to gain train-time and inference-time sparsity benefits without costly dense training.
• Introduce FORCE as a saliency criterion that captures post-pruning trainability.
• Propose two progressive pruning procedures (Iterative SNIP and FORCE) to maximize FORCE.
• Demonstrate robustness and effectiveness across datasets and architectures, including very high sparsity levels.

Proposed method

• Define FORCE as the saliency after pruning: g(θ̄) = ∂L(θ̄)/∂c evaluated at c = ĉ, which equals (∂L(θ̄)/∂θ̄) ∘ θ (Eq. 5).
• Formulate FORCE objective: maxc S(θ, c) = sum_{i in supp(c)} |θ_i ∂L(θ ∘ c)/∂(θ ∘ c)| (Eq. 6).
• Propose Iterative SNIP: progressively prune by solving pruned-SNIP objective while keeping previously kept connections (c_t+1 respects c_t).
• Propose FORCE: iterative pruning that allows pruned weights to resurrect, sparsifying progressively while recomputing FORCE saliency on a sparsified network.
• Compare to SNIP and GRASP, highlighting that SNIP/GRASP rely on pre-pruning saliency and may fail at high sparsity; FORCE optimizes post-pruning saliency.
• Use an exponential sparsity schedule k_t to control pruning steps (Eq. 8).
• Provide theoretical insight that intermediate masks from Iterative SNIP approximate local minima under small pruning steps.

Experimental results

Research questions

• RQ1 Can pruning at initialization yield trainable sub-networks at extreme sparsity levels (e.g., >95%)?
• RQ2 Does maximizing foresight connection sensitivity (FORCE) enable better transfer of pruning decisions as sparsity increases?
• RQ3 Do iterative pruning strategies (Iterative SNIP and FORCE) outperform one-shot saliency methods like SNIP/GRASP, especially at high sparsity?
• RQ4 What is the trade-off between exploration (FORCE) and exploitation (Iterative SNIP) in finding trainable sparse networks?

Key findings

• FORCE and Iterative SNIP substantially outperform SNIP and GRASP at high sparsity across CIFAR-10, CIFAR-100, Tiny ImageNet, and ImageNet.
• For CIFAR-10 with ResNet50 and VGG19, extreme sparsity (up to 99.9%) still yields trainable networks and accuracies well above random; FORCE achieves notably higher accuracy than alternatives.
• On ImageNet, FORCE prunes up to 90% sparsity with VGG19 (Top-1 70.2, Top-5 89.5) and 95% sparsity (Top-1 65.8, Top-5 86.8); for ResNet50 at 90% sparsity, Top-1 64.9 and Top-5 86.5 (95% sparsity: Top-1 59.0, Top-5 82.3).
• FORCE can prune up to 99.5% of parameters on some architectures while keeping networks trainable, a level of sparsity where prior methods deteriorate or underperform random pruning.
• Iterative SNIP is generally robust and can approach FORCE performance with enough iterations, but FORCE often yields better results due to allowing restoration of previously pruned weights (exploration).
• Pruning cost and efficiency: FORCE/Iter SNIP are cheaper than GRASP-MB and can achieve competitive or superior accuracy with fewer computational overhead compared to Hessian-based GRASP.