[Paper Review] What is the State of Neural Network Pruning?

Davis Blalock, Jose Javier Gonzalez Ortiz+2 more|arXiv (Cornell University)|Mar 6, 2020

Advanced Neural Network Applications57 references78 citations

TL;DR

A meta-analysis of neural network pruning across 81 papers reveals severe benchmarking fragmentation and proposes ShrinkBench to enable standardized evaluations and fair comparisons of pruning methods.

ABSTRACT

Neural network pruning---the task of reducing the size of a network by removing parameters---has been the subject of a great deal of work in recent years. We provide a meta-analysis of the literature, including an overview of approaches to pruning and consistent findings in the literature. After aggregating results across 81 papers and pruning hundreds of models in controlled conditions, our clearest finding is that the community suffers from a lack of standardized benchmarks and metrics. This deficiency is substantial enough that it is hard to compare pruning techniques to one another or determine how much progress the field has made over the past three decades. To address this situation, we identify issues with current practices, suggest concrete remedies, and introduce ShrinkBench, an open-source framework to facilitate standardized evaluations of pruning methods. We use ShrinkBench to compare various pruning techniques and show that its comprehensive evaluation can prevent common pitfalls when comparing pruning methods.

Motivation & Objective

• Synthesize the neural network pruning literature to identify practical lessons and common pitfalls.
• Assess how pruning methods compare across architectures, datasets, and evaluation metrics.
• Recommend best practices and tools to enable standardized, fair evaluations of pruning methods.
• Provide a concrete open-source framework to support standardized pruning research.

Proposed method

• Aggregate results from 81 pruning papers to characterize practices and findings.
• Catalog common impediments such as dataset/architecture fragmentation and metric inconsistency.
• Define a standardized evaluation framework and metrics for pruning methods.
• Introduce ShrinkBench as an open-source library for standardized pruning evaluation.
• Demonstrate ShrinkBench baselines and show how standardized evaluation mitigates misinterpretations.

Experimental results

Research questions

• RQ1 What are the common practices and results across recent neural network pruning literature?
• RQ2 How do benchmarking practices hinder fair comparisons between pruning methods?
• RQ3 What concrete remedies and tools can improve reproducibility and comparability in pruning research?
• RQ4 How does ShrinkBench help ensure standardized evaluations and fair method comparisons?

Key findings

• Pruning can significantly compress models with little or no loss of accuracy, and some methods even improve accuracy at small compression levels.
• Many methods outperform random pruning and global pruning is often more effective than layerwise pruning for fixed parameters, though results vary by dataset/architecture.
• Sparse models can outperform dense models with the same parameter count, and sometimes outperform the original unpruned model.
• There is substantial fragmentation in datasets, architectures, and metrics, making direct, controlled comparisons between pruning methods difficult.
• A large portion of papers report results on limited dataset/architecture pairings and provide few operating points, hindering trend identification.
• Confounding variables such as initial model quality, data augmentation, optimization choices, pruning schedules, and libraries significantly affect outcomes, complicating fair comparisons.
• ShrinkBench is proposed as a library to standardize pruning evaluations and mitigate these issues.