[Paper Review] Visual Prompting Upgrades Neural Network Sparsification: A Data-Model Perspective

Can Jin, Tianjin Huang+5 more|arXiv (Cornell University)|Dec 3, 2023

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes VPNs, a data-model co-design framework that jointly optimizes visual prompts and learnable weight masks to enhance neural network sparsification in vision models. By co-training prompts and masks, VPNs achieve state-of-the-art accuracy across multiple architectures and datasets, with up to 95% fewer training epochs than prior methods and superior subnetwork transferability.

ABSTRACT

The rapid development of large-scale deep learning models questions the affordability of hardware platforms, which necessitates the pruning to reduce their computational and memory footprints. Sparse neural networks as the product, have demonstrated numerous favorable benefits like low complexity, undamaged generalization, etc. Most of the prominent pruning strategies are invented from a model-centric perspective, focusing on searching and preserving crucial weights by analyzing network topologies. However, the role of data and its interplay with model-centric pruning has remained relatively unexplored. In this research, we introduce a novel data-model co-design perspective: to promote superior weight sparsity by learning important model topology and adequate input data in a synergetic manner. Specifically, customized Visual Prompts are mounted to upgrade neural Network sparsification in our proposed VPNs framework. As a pioneering effort, this paper conducts systematic investigations about the impact of different visual prompts on model pruning and suggests an effective joint optimization approach. Extensive experiments with 3 network architectures and 8 datasets evidence the substantial performance improvements from VPNs over existing start-of-the-art pruning algorithms. Furthermore, we find that subnetworks discovered by VPNs from pre-trained models enjoy better transferability across diverse downstream scenarios. These insights shed light on new promising possibilities of data-model co-designs for vision model sparsification.

Motivation & Objective

• Address the underexplored role of data and its interaction with model-centric pruning in neural network sparsification.
• Investigate whether visual prompts can enhance performance in sparse vision models, particularly when applied post-pruning.
• Develop a co-design paradigm that synergistically optimizes visual prompts and weight masks to discover high-performing, sparse subnetworks.
• Demonstrate improved efficiency and transferability of subnetworks discovered via the proposed method compared to existing pruning algorithms.

Proposed method

• Introduce a novel data-model co-design framework, VPNs, that co-trains visual prompts and differentiable weight masks during the pruning process.
• Design specialized visual prompts (pad, random, and fix prompts) with tunable parameters to inject data-level inductive bias into the sparsification process.
• Implement a joint optimization scheme where both the visual prompt and the binary mask are updated end-to-end to identify optimal sparse subnetworks.
• Use a learnable prompt embedding space to adaptively attend to relevant input regions, enhancing feature representation in sparse models.
• Apply the framework across multiple architectures (e.g., ResNet-18) and datasets (e.g., CIFAR-100, ImageNet) to evaluate generalization and robustness.
• Employ a two-stage training process: first, mask finding with visual prompts; second, subnetwork tuning with prompts, enabling progressive refinement.

Experimental results

Research questions

• RQ1 Can post-pruning visual prompts effectively improve the performance of sparse vision models, especially when applied after model fine-tuning?
• RQ2 How do different visual prompt designs (e.g., pad, random, fix) impact the accuracy and convergence of sparse subnetworks?
• RQ3 What is the relative contribution of visual prompts during mask finding versus subnetwork tuning in the pruning pipeline?
• RQ4 To what extent do subnetworks discovered by the proposed method generalize across diverse downstream tasks and datasets?
• RQ5 Can the joint optimization of visual prompts and weight masks lead to faster convergence and higher sparsity with maintained performance?

Key findings

• VPNs outperform state-of-the-art pruning methods such as LTH, GraSP, and HYDRA, achieving higher test accuracy across 8 datasets and 3 architectures.
• At 90% sparsity, VPNs require 95% fewer epochs than LTH, 50% fewer than GraSP, and 50% fewer than HYDRA, demonstrating superior training efficiency.
• The pad prompt design consistently outperforms random and fix prompts, with optimal performance at a pad size of 16, while larger pad sizes degrade performance.
• Subnetworks discovered by VPNs exhibit strong transferability across datasets, indicating their practical utility in downstream applications.
• VPNs reduce training time by 26% and 8.97% compared to LTH and GraSP, respectively, and maintain time consumption comparable to GraSP despite higher sparsity.
• The ablation study confirms that visual prompts contribute more significantly during subnetwork tuning than during mask finding, validating the importance of prompt integration in later training stages.