[Paper Review] WoodFisher: Efficient Second-Order Approximation for Neural Network Compression

Sidak Pal Singh, Dan Alistarh|arXiv (Cornell University)|Apr 29, 2020

Advanced Neural Network ApplicationsComputer Science48 references36 citations

TL;DR

WoodFisher introduces an efficient inverse-Hessian approximation via the empirical Fisher and Woodbury identity to enable second-order based pruning. It yields state-of-the-art one-shot and competitive gradual pruning for CNNs on ImageNet and CIFAR10.

ABSTRACT

Second-order information, in the form of Hessian- or Inverse-Hessian-vector products, is a fundamental tool for solving optimization problems. Recently, there has been significant interest in utilizing this information in the context of deep neural networks; however, relatively little is known about the quality of existing approximations in this context. Our work examines this question, identifies issues with existing approaches, and proposes a method called WoodFisher to compute a faithful and efficient estimate of the inverse Hessian. Our main application is to neural network compression, where we build on the classic Optimal Brain Damage/Surgeon framework. We demonstrate that WoodFisher significantly outperforms popular state-of-the-art methods for one-shot pruning. Further, even when iterative, gradual pruning is considered, our method results in a gain in test accuracy over the state-of-the-art approaches, for pruning popular neural networks (like ResNet-50, MobileNetV1) trained on standard image classification datasets such as ImageNet ILSVRC. We examine how our method can be extended to take into account first-order information, as well as illustrate its ability to automatically set layer-wise pruning thresholds and perform compression in the limited-data regime. The code is available at the following link, https://github.com/IST-DASLab/WoodFisher.

Motivation & Objective

• Motivate whether second-order information can be accurate and scalable for neural networks.
• Develop an efficient method to estimate inverse Hessian information suitable for large models.
• Apply the method to neural network compression within the Optimal Brain Damage/Surgeon framework.
• Demonstrate one-shot and gradual pruning performance improvements over state-of-the-art methods.

Proposed method

• Approximate the Hessian with the empirical Fisher and use the Woodbury identity to iteratively update the inverse-Fisher estimate.
• Recursively update the empirical Fisher as _hat_{n+1} = F_hat_n + 1/N grad(l_{n+1}) grad(l_{n+1})^T with a dampening term mbda I_d.
• Compute the inverse via Woodberry updates: F_hat_{n+1}^{-1} = F_hat_n^{-1} - (F_hat_n^{-1} grad(l_{n+1}) grad(l_{n+1})^T F_hat_n^{-1}) / (N + grad(l_{n+1})^T F_hat_n^{-1} grad(l_{n+1})).
• Introduce block-wise (chunked) approximations to scale to large models, yielding O(m c d) runtime with block size c and total d parameters.
• Define pruning statistic e_q = w_q^2 / (2 [H^{-1}]_{qq}) to rank parameters for removal, and use this to perform layer-wise or global pruning (joint vs independent WoodFisher).
• Extend to include first-order (gradient) terms and discuss limited-data regime pruning and automatic layer-wise sparsity thresholds.

Experimental results

Research questions

• RQ1 Can second-order approximations (via inverse-Hessian information) be accurate and scalable for modern neural networks?
• RQ2 Is the empirical Fisher a practical and faithful surrogate for the Hessian in large-scale pruning tasks?
• RQ3 Can WoodFisher-based pruning outperform magnitude-based and diagonal-Fisher baselines in one-shot and gradual pruning settings?
• RQ4 Does joint (global) sparsity targeting improve compression performance over layer-wise pruning?
• RQ5 Can WoodFisher be extended to limited-data scenarios and incorporate first-order information to prune before full convergence?

Key findings

• WoodFisher yields significantly better one-shot pruning performance than magnitude pruning and diagonal-Fisher baselines on ResNet-20/CIFAR-10 and ResNet-50/ImageNet.
• Joint WoodFisher (global sparsity targeting) often outperforms independent (layer-wise) WoodFisher, especially at higher sparsity levels.
• Using chunked block-wise approximations maintains practical efficiency while preserving pruning quality, with larger block sizes improving accuracy.
• WoodFisher outperforms state-of-the-art pruning approaches in gradual pruning scenarios, sometimes matching or exceeding top methods with re-training.
• Empirical evidence suggests the local quadratic model built from WoodFisher closely predicts loss changes along pruning directions, supporting the approximation quality.