[Paper Review] The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networks

Xin Yu, Thiago Serra+2 more|arXiv (Cornell University)|Mar 9, 2022

Brain Tumor Detection and Classification4 citations

TL;DR

The Combinatorial Brain Surgeon (CBS) proposes a novel pruning method that jointly selects weights for removal based on their combined impact, minimizing loss increase through a Mixed-Integer Quadratic Programming (MIQP) formulation. It outperforms magnitude-based and existing Hessian-based methods—especially at high sparsity (e.g., 95–98%)—by leveraging weight cancellation effects and a single-pass weight update, achieving state-of-the-art accuracy on MNIST, CIFAR-10, and ImageNet benchmarks.

ABSTRACT

Neural networks tend to achieve better accuracy with training if they are larger -- even if the resulting models are overparameterized. Nevertheless, carefully removing such excess parameters before, during, or after training may also produce models with similar or even improved accuracy. In many cases, that can be curiously achieved by heuristics as simple as removing a percentage of the weights with the smallest absolute value -- even though magnitude is not a perfect proxy for weight relevance. With the premise that obtaining significantly better performance from pruning depends on accounting for the combined effect of removing multiple weights, we revisit one of the classic approaches for impact-based pruning: the Optimal Brain Surgeon(OBS). We propose a tractable heuristic for solving the combinatorial extension of OBS, in which we select weights for simultaneous removal, as well as a systematic update of the remaining weights. Our selection method outperforms other methods under high sparsity, and the weight update is advantageous even when combined with the other methods.

Motivation & Objective

• Address the limitation of existing pruning methods that evaluate weight removals independently, missing synergistic effects between pruned weights.
• Formulate a combinatorial pruning problem that jointly optimizes the impact of removing multiple weights to minimize loss increase.
• Develop a tractable heuristic for solving the NP-hard combinatorial pruning problem using local search and semi-greedy initialization.
• Decouple weight selection from weight update to improve scalability and enable efficient single-pass updates of unpruned weights.
• Demonstrate that joint impact consideration leads to significantly better performance at high sparsity levels, especially in overparameterized models.

Proposed method

• Formulates the Combinatorial Brain Surgeon (CBS) problem as a Mixed-Integer Quadratic Program (MIQP), modeling the loss increase from removing a set of weights using a quadratic approximation of the loss function.
• Uses the functional Taylor expansion of the loss around trained weights, assuming a local minimum and neglecting third-order terms, to derive a quadratic approximation of loss change.
• Applies a local search algorithm to iteratively improve the selection of weights to prune by exploring neighboring solutions that reduce the estimated loss increase.
• Introduces a randomized extension of magnitude-based pruning to generate diverse initial weight sets for the local search, enhancing exploration of the solution space.
• Proposes a single-pass, systematic weight update rule (CBS-U) that efficiently adjusts unpruned weights after pruning, inspired by Optimal Brain Surgeon (OBS), but generalized to multiple simultaneous removals.
• Decouples weight selection from weight update to avoid scalability bottlenecks and enables efficient integration with existing pruning pipelines.

Experimental results

Research questions

• RQ1 Can joint consideration of multiple weight removals lead to better pruning performance than sequential or independent pruning?
• RQ2 Does accounting for the cancellation effect between pruned weights—where their combined impact on loss is smaller than individual impacts—improve model accuracy at high sparsity?
• RQ3 Can a tractable heuristic be designed to solve the NP-hard combinatorial pruning problem without relying on expensive Hessian computation?
• RQ4 How does the proposed single-pass weight update (CBS-U) compare to existing Hessian-based update strategies in terms of accuracy and efficiency?
• RQ5 Does the method maintain or improve performance across diverse architectures and datasets, especially at extreme sparsity levels (e.g., 95–98%)?

Key findings

• CBS-S (combinatorial weight selection) achieves a 17.20% improvement in accuracy over magnitude pruning (MP) at 98% sparsity on ImageNet with MobileNet, increasing accuracy from 10.12% to 15.49%.
• At 95% sparsity on CIFAR-10 with ResNet20, CBS-S improves accuracy from 17.61% (MP) to 32.63%, a 17.82% gain, demonstrating strong performance at extreme sparsity.
• CBS-U (single-pass weight update) improves accuracy by up to 24.39% at 60% sparsity on MobileNet, showing that efficient weight updates significantly enhance performance.
• On MNIST with MLPNet, CBS-S achieves 93.96% accuracy at 95% sparsity, outperforming MP (83.64%) and WF-S (83.16%) by over 10 percentage points.
• CBS-S and CBS-U together yield state-of-the-art results across all benchmarks, with consistent improvements over magnitude pruning and WoodFisher at high sparsity levels.
• The method is robust across architectures: it achieves the best or second-best results on all tested models (MLPNet, ResNet20, CIFARNet, MobileNet) across multiple sparsity levels.