[Paper Review] CP-ViT: Cascade Vision Transformer Pruning via Progressive Sparsity Prediction

Zhuoran Song, Yihong Xu+4 more|arXiv (Cornell University)|Mar 9, 2022

CCD and CMOS Imaging SensorsEngineering20 citations

TL;DR

CP-ViT introduces cascade pruning for Vision Transformers by progressively predicting and pruning uninformative patches and heads, using cumulative scores and layer-aware pruning to reduce FLOPs with minimal accuracy loss, with or without finetuning.

ABSTRACT

Vision transformer (ViT) has achieved competitive accuracy on a variety of computer vision applications, but its computational cost impedes the deployment on resource-limited mobile devices. We explore the sparsity in ViT and observe that informative patches and heads are sufficient for accurate image recognition. In this paper, we propose a cascade pruning framework named CP-ViT by predicting sparsity in ViT models progressively and dynamically to reduce computational redundancy while minimizing the accuracy loss. Specifically, we define the cumulative score to reserve the informative patches and heads across the ViT model for better accuracy. We also propose the dynamic pruning ratio adjustment technique based on layer-aware attention range. CP-ViT has great general applicability for practical deployment, which can be applied to a wide range of ViT models and can achieve superior accuracy with or without fine-tuning. Extensive experiments on ImageNet, CIFAR-10, and CIFAR-100 with various pre-trained models have demonstrated the effectiveness and efficiency of CP-ViT. By progressively pruning 50\% patches, our CP-ViT method reduces over 40\% FLOPs while maintaining accuracy loss within 1\%.

Motivation & Objective

• Explore sparsity in Vision Transformers (ViT) to identify informative patches and heads (PH-regions) that drive accuracy.
• Develop a cascade pruning framework to dynamically prune uninformative PH-regions with minimal accuracy loss.
• Introduce the cumulative score mechanism to preserve informative PH-regions across layers.
• Propose a layer-aware pruning ratio adjustment method based on attention range to balance efficiency and accuracy.
• Demonstrate broad applicability across ViT models and show performance with and without finetuning.

Proposed method

• Define informative PH-regions in ViT via patch/head informativeness derived from attention probability.
• Use a progressive sparsity prediction framework to compute cumulative scores across layers.
• Apply cascade pruning by masking uninformative PH-regions in MHSA and FFN, enabling skipped computation in subsequent layers.
• Compute pruning ratios per layer with a Layer-Aware Pruning Ratio method that leverages attention range to adapt pruning per layer.
• Perform pruning based on maximum attention probability values to approximate informativeness efficiently.
• Optionally finetune pre-trained models to enhance accuracy under pruning.

Experimental results

Research questions

• RQ1 Can ViT models exhibit sparsity through informative patches and heads that significantly affect accuracy?
• RQ2 Can a cascade pruning approach preserve accuracy while discarding uninformative PH-regions across layers?
• RQ3 How can pruning ratios be dynamically adjusted per layer to account for inter-layer differences in attention range?
• RQ4 Does CP-ViT maintain high accuracy with substantial FLOPs reduction with and without finetuning across diverse ViT architectures?

Key findings

• CP-ViT can prune about 50% of patches with over 40% FLOPs reduction while keeping accuracy loss under 1% (without finetuning).
• Pruning ratios tuned per layer via attention range improves robustness and reduces accuracy loss at larger pruning levels.
• Progressive sparsity prediction using cumulative scores preserves informative PH-regions across layers, outperforming random pruning and naive layer-unaware pruning.
• CP-ViT delivers favorable accuracy-FLOPs trade-offs compared with other ViT pruning methods, with or without finetuning.
• Finetuning CP-ViT significantly increases achievable FLOPs savings while maintaining or slightly improving accuracy for several ViT models.