[Paper Review] On the Predictability of Pruning Across Scales

Jonathan S. Rosenfeld, Jonathan Frankle+2 more|arXiv (Cornell University)|Jun 18, 2020

Adversarial Robustness in Machine Learning18 references17 citations

TL;DR

This paper proposes a predictive scaling law for iteratively magnitude-pruned neural networks, showing that generalization error follows a functional form invariant across width, depth, and pruning density. The key contribution is a mathematically interpretable scaling law that enables accurate error prediction across diverse architectures and datasets—including ImageNet and ResNets—enabling analytical optimization of pruned model size under error constraints.

ABSTRACT

We show that the error of iteratively magnitude-pruned networks empirically follows a scaling law with interpretable coefficients that depend on the architecture and task. We functionally approximate the error of the pruned networks, showing it is predictable in terms of an invariant tying width, depth, and pruning level, such that networks of vastly different pruned densities are interchangeable. We demonstrate the accuracy of this approximation over orders of magnitude in depth, width, dataset size, and density. We show that the functional form holds (generalizes) for large scale data (e.g., ImageNet) and architectures (e.g., ResNets). As neural networks become ever larger and costlier to train, our findings suggest a framework for reasoning conceptually and analytically about a standard method for unstructured pruning.

Motivation & Objective

• To address the lack of principled guidance in choosing optimal network width, depth, and pruning density for minimal parameter count under error constraints.
• To identify a functional form that predicts the generalization error of pruned networks across architectural and dataset scales.
• To uncover an invariant that enables error-preserving interchangeability among width, depth, and pruning density.
• To develop a scalable, data-efficient method for predicting pruned network performance without exhaustive retraining.
• To enable analytical optimization of pruned models by formulating a joint scaling law for depth, width, dataset size, and pruning density.

Proposed method

• Proposes a functional form for error prediction as a function of pruning level, derived from iterative magnitude pruning (IMP) on a single network.
• Introduces an invariant that relates depth, width, and pruning density such that error remains unchanged when these are adjusted proportionally.
• Extends the single-network scaling law into a joint scaling law by incorporating depth (l), width (w), dataset size (d), and pruning density (p), using the invariant to unify scaling across these dimensions.
• Employs a power-law functional form: ε ∝ (l^φ w^ψ d - j p'^γ)^(-1/γ), where coefficients are interpretable and fitted to small-scale data.
• Uses only a small number of pruned network evaluations to fit the scaling law, enabling efficient transfer to new architectures and datasets.
• Applies the fitted scaling law to solve an optimization problem: minimizing parameter count subject to a target error threshold, without further training.

Experimental results

Research questions

• RQ1 Can the generalization error of iteratively magnitude-pruned networks be predicted across vastly different widths, depths, and pruning levels?
• RQ2 Is there a mathematical invariant that allows interchangeable adjustments of width, depth, and pruning density without changing error?
• RQ3 Does the proposed scaling law generalize to large-scale datasets like ImageNet and architectures like ResNets?
• RQ4 Can the scaling law be used to analytically determine the minimal parameter-count network that satisfies a given error constraint?
• RQ5 How accurate is the scaling law in predicting error across orders of magnitude in network scale and dataset size?

Key findings

• The generalization error of IMP-pruned networks follows a predictable power-law scaling form across width, depth, and pruning density, with interpretable coefficients.
• An invariant exists that allows error-preserving interchangeability among depth, width, and pruning level, enabling unified scaling across architectural hyperparameters.
• The joint scaling law accurately predicts error across orders of magnitude in depth, width, dataset size, and pruning density, including on ImageNet and ResNet architectures.
• The model achieves prediction accuracy within 25% of the true optimal parameter count when optimizing for minimal size under error constraints.
• The functional form enables analytical optimization of pruned models without additional training, solving the minimal parameter-count problem under error constraints.
• The scaling law is data-efficient to fit, requiring only a small number of pruned network evaluations to calibrate for new architectures and datasets.