[Paper Review] Pruning neural networks without any data by iteratively conserving synaptic flow

Hidenori Tanaka, Daniel Kunin+2 more|arXiv (Cornell University)|Jun 9, 2020

Advanced Memory and Neural Computing256 citations

TL;DR

The paper introduces a data-agnostic pruning method, SynFlow, that preserves synaptic flow to avoid layer-collapse and achieve maximal critical compression without training or data access.

ABSTRACT

Pruning the parameters of deep neural networks has generated intense interest due to potential savings in time, memory and energy both during training and at test time. Recent works have identified, through an expensive sequence of training and pruning cycles, the existence of winning lottery tickets or sparse trainable subnetworks at initialization. This raises a foundational question: can we identify highly sparse trainable subnetworks at initialization, without ever training, or indeed without ever looking at the data? We provide an affirmative answer to this question through theory driven algorithm design. We first mathematically formulate and experimentally verify a conservation law that explains why existing gradient-based pruning algorithms at initialization suffer from layer-collapse, the premature pruning of an entire layer rendering a network untrainable. This theory also elucidates how layer-collapse can be entirely avoided, motivating a novel pruning algorithm Iterative Synaptic Flow Pruning (SynFlow). This algorithm can be interpreted as preserving the total flow of synaptic strengths through the network at initialization subject to a sparsity constraint. Notably, this algorithm makes no reference to the training data and consistently competes with or outperforms existing state-of-the-art pruning algorithms at initialization over a range of models (VGG and ResNet), datasets (CIFAR-10/100 and Tiny ImageNet), and sparsity constraints (up to 99.99 percent). Thus our data-agnostic pruning algorithm challenges the existing paradigm that, at initialization, data must be used to quantify which synapses are important.

Motivation & Objective

• Motivate and formalize why pruning at initialization suffers from layer-collapse.
• Show that synaptic saliency is conserved across neurons and layers under common activations.
• Explain why larger layers receive smaller average scores, leading to collapse in gradient-based methods.
• Develop a data-agnostic pruning algorithm that achieves Maximal Critical Compression.
• Demonstrate through experiments that SynFlow matches or exceeds state-of-the-art pruning without using training data.

Proposed method

• Define synaptic saliency as a Hadamard product of gradients and parameters, and show neuron- and network-wise conservation laws.
• Prove that gradient-based scores exhibit a conservation property, explaining layer-size dependencies.
• Introduce Iterative Synaptic Flow Pruning (SynFlow) based on a data-agnostic loss that yields positive, conserved scores.
• Prove that iterative, positive, conservative scoring satisfies Maximal Critical Compression under global masking.
• Provide an algorithmic pseudocode for SynFlow and discuss computational costs (100 pruning iterations).
• Empirically compare SynFlow to SNIP, GraSP, and magnitude/random pruning across multiple models and datasets.

Experimental results

Research questions

• RQ1 Can highly sparse trainable subnetworks be identified at initialization without training or data?
• RQ2 Why do gradient-based single-shot pruning methods tend to cause layer-collapse, and how can this be mitigated?
• RQ3 Can a data-agnostic pruning method reach Maximal Critical Compression while avoiding layer-collapse?
• RQ4 How does iterative evaluation of a pruning score influence the preservation of network trainability at initialization?

Key findings

• SynFlow consistently outperforms baselines and data-dependent pruning methods in high compression regimes without using data.
• Conservation laws of synaptic saliency hold neuron-wise and network-wise at initialization, explaining why layer-collapse occurs for some scores.
• An inverse relationship between layer size and average layer score explains why large layers are pruned first by gradient-based methods.
• Iterative, positive, conservative scoring guarantees Maximal Critical Compression under global masking (no layer-collapse).
• SynFlow achieves state-of-the-art pruning performance across 12 model/dataset combinations using data-agnostic pruning.