[Paper Review] AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Rates

Ning Liu, Xiaolong Ma+4 more|arXiv (Cornell University)|Jul 6, 2019

Advanced Neural Network Applications41 references35 citations

TL;DR

AutoCompress presents an automatic framework for DNN structured pruning using ADMM-based pruning plus a purification step and a guided heuristic search (enhanced over DRL) to achieve ultra-high weight/FLOPs reductions with minimal accuracy loss.

ABSTRACT

Structured weight pruning is a representative model compression technique of DNNs to reduce the storage and computation requirements and accelerate inference. An automatic hyperparameter determination process is necessary due to the large number of flexible hyperparameters. This work proposes AutoCompress, an automatic structured pruning framework with the following key performance improvements: (i) effectively incorporate the combination of structured pruning schemes in the automatic process; (ii) adopt the state-of-art ADMM-based structured weight pruning as the core algorithm, and propose an innovative additional purification step for further weight reduction without accuracy loss; and (iii) develop effective heuristic search method enhanced by experience-based guided search, replacing the prior deep reinforcement learning technique which has underlying incompatibility with the target pruning problem. Extensive experiments on CIFAR-10 and ImageNet datasets demonstrate that AutoCompress is the key to achieve ultra-high pruning rates on the number of weights and FLOPs that cannot be achieved before. As an example, AutoCompress outperforms the prior work on automatic model compression by up to 33x in pruning rate (120x reduction in the actual parameter count) under the same accuracy. Significant inference speedup has been observed from the AutoCompress framework on actual measurements on smartphone. We release all models of this work at anonymous link: http://bit.ly/2VZ63dS.

Motivation & Objective

• Motivate automatic hyperparameter determination for structured pruning to reduce weights/FLOPs with minimal accuracy loss.
• Integrate multiple structured pruning schemes (e.g., filter, column) for higher compression.
• Leverage ADMM-based structured pruning as the core solver and add a purification step.
• Replace DRL-based hyperparameter search with an experience-guided heuristic search to enable high pruning rates.

Proposed method

• Adopts ADMM-based structured weight pruning as the core optimization engine.
• Includes a purification step to remove additional weight via column-/filter-wise thresholds while preserving structure.
• Employs a four-step generic automatic process: action sampling, quick action evaluation, decision making, and actual pruning.
• Uses enhanced simulated annealing with guided search to determine pruning actions across layers.
• Allows progressive rounds of pruning with approximately 2x reduction per round to reach ultra-high compression.
• Provides an automatic framework that combines filter pruning with column pruning for better hardware-compatibility and performance.

Experimental results

Research questions

• RQ1 Can automatic selection of per-layer pruning rates and pruning scheme combinations achieve higher compression while preserving accuracy?
• RQ2 Does combining filter pruning with column pruning and using ADMM-based pruning outperform DRL-based or manual hyperparameter methods?
• RQ3 Is heuristic search (guided by prior experience) more effective than DRL for high-rate structured pruning?
• RQ4 What is the impact of the purification step on further weight/FLOPs reduction without accuracy loss?

Key findings

• AutoCompress outperforms prior automatic model compression methods by up to 33x in pruning rate (up to 120x parameter reduction) at similar accuracy.
• Combined structured pruning (filter + column pruning) yields larger weight/FLOPs reductions than filter pruning alone.
• Enhanced SA-based hyperparameter determination surpasses DRL-based approaches and manual tuning in achieving higher pruning rates.
• Phase II purification further reduces weights without accuracy loss by removing low-magnitude columns/filters and their dependent channels.
• On CIFAR-10 and ImageNet, AutoCompress achieves ultra-high pruning rates and measurable inference speedups on mobile hardware.