[Paper Review] M-FAC: Efficient Matrix-Free Approximations of Second-Order Information

Elias Frantar, Eldar Kurtic+1 more|arXiv (Cornell University)|Jul 7, 2021

Stochastic Gradient Optimization Techniques35 references4 citations

TL;DR

This paper introduces M-FAC, a matrix-free method for efficiently computing Inverse-Hessian-Vector Products (IHVPs) under the empirical Fisher approximation, using linear-time algorithms that avoid storing full Hessian matrices. The approach enables state-of-the-art results in neural network pruning and optimization with significantly reduced computational cost compared to existing second-order methods.

ABSTRACT

Efficiently approximating local curvature information of the loss function is a key tool for optimization and compression of deep neural networks. Yet, most existing methods to approximate second-order information have high computational or storage costs, which can limit their practicality. In this work, we investigate matrix-free, linear-time approaches for estimating Inverse-Hessian Vector Products (IHVPs) for the case when the Hessian can be approximated as a sum of rank-one matrices, as in the classic approximation of the Hessian by the empirical Fisher matrix. We propose two new algorithms as part of a framework called M-FAC: the first algorithm is tailored towards network compression and can compute the IHVP for dimension $d$, if the Hessian is given as a sum of $m$ rank-one matrices, using $O(dm^2)$ precomputation, $O(dm)$ cost for computing the IHVP, and query cost $O(m)$ for any single element of the inverse Hessian. The second algorithm targets an optimization setting, where we wish to compute the product between the inverse Hessian, estimated over a sliding window of optimization steps, and a given gradient direction, as required for preconditioned SGD. We give an algorithm with cost $O(dm + m^2)$ for computing the IHVP and $O(dm + m^3)$ for adding or removing any gradient from the sliding window. These two algorithms yield state-of-the-art results for network pruning and optimization with lower computational overhead relative to existing second-order methods. Implementations are available at [9] and [17].

Motivation & Objective

• To address the high computational and storage costs of second-order optimization and pruning methods in deep neural networks.
• To enable exact, efficient computation of Inverse-Hessian-Vector Products (IHVPs) under the empirical Fisher approximation without storing large matrices.
• To develop algorithms that scale linearly with model dimension d and are practical for large-scale DNNs.
• To improve network pruning accuracy and optimization convergence using curvature-aware updates.
• To validate the effectiveness of the proposed methods on standard benchmarks including CIFAR-10, GLUE, and SQuADv2.

Proposed method

• The method uses the Woodbury-Sherman-Morrison formula to recursively update the inverse Fisher matrix as a sum of rank-one components from gradient outer products.
• For the static algorithm, precomputation costs O(dm²) and IHVP computation costs O(dm), with O(m) per-element query cost for diagonal entries.
• The dynamic algorithm maintains a sliding window of m recent gradients, enabling incremental updates with O(dm + m²) cost for IHVP and O(dm + m³) for adding/removing gradients.
• The algorithms avoid explicit matrix storage by operating only on vectors and scalars, ensuring linear time and space complexity in model dimension d.
• The approach leverages the fact that the empirical Fisher matrix is a sum of rank-one matrices, enabling exact inversion via recursive updates.
• The method is applied to both network pruning (via Optimal Brain Surgeon) and preconditioned SGD in optimization.

Experimental results

Research questions

• RQ1 Can we compute IHVPs efficiently under the empirical Fisher approximation without storing full Hessian matrices?
• RQ2 Can matrix-free, linear-time algorithms for IHVPs achieve state-of-the-art performance in network pruning and optimization?
• RQ3 How accurate is the sliding window approximation of the inverse Hessian in dynamic optimization settings?
• RQ4 Does the proposed method outperform existing second-order methods in terms of computational efficiency and model performance?
• RQ5 Can the method be applied effectively to large-scale models like BERT without significant hyperparameter tuning?

Key findings

• M-FAC achieves state-of-the-art pruning performance on CIFAR-10 with ResNet20, matching or exceeding prior methods while reducing computational cost.
• On the SQuADv2 benchmark, M-FAC outperforms HuggingFace's Adam baseline, achieving 52.18 F1 score (vs. 49.99) on BERT-tiny and 61.35 F1 (vs. 58.13) on BERT-mini.
• On GLUE, M-FAC matches or exceeds the performance of tuned AdamW baselines on BERT-tiny, achieving 83.4% SST-2 accuracy (vs. 83.2%) and 81.9% MRPC F1 (vs. 81.1%).
• Cosine similarity between dynamic and static IHVP directions remains high (>0.95) throughout training, validating the sliding window approximation.
• The method achieves O(dm) cost for IHVP computation and O(m) per-element query cost, enabling efficient pruning and optimization.
• The implementation is available at [9] and [17], supporting practical deployment in real-world deep learning pipelines.