[Paper Review] Improving Visual Prompt Tuning by Gaussian Neighborhood Minimization for Long-Tailed Visual Recognition

Mengke Li, Ye Liu+4 more|arXiv (Cornell University)|Oct 28, 2024

Advanced Image and Video Retrieval Techniques4 citations

TL;DR

This paper proposes Gaussian Neighborhood Minimization Prompt Tuning (GNM-PT), a novel optimization strategy for visual prompt tuning (VPT) that improves generalization in long-tailed visual recognition. By minimizing the mean loss over a Gaussian parameter neighborhood—offering a tighter upper bound than existing methods—it flattens the loss landscape, enhancing performance on tail classes without increasing computational cost, achieving state-of-the-art accuracy of 90.3%, 76.5%, and 50.1% on CIFAR100-LT (IR 100), iNaturalist 2018, and Places-LT respectively.

ABSTRACT

Long-tail learning has garnered widespread attention and achieved significant progress in recent times. However, even with pre-trained prior knowledge, models still exhibit weaker generalization performance on tail classes. The promising Sharpness-Aware Minimization (SAM) can effectively improve the generalization capability of models by seeking out flat minima in the loss landscape, which, however, comes at the cost of doubling the computational time. Since the update rule of SAM necessitates two consecutive (non-parallelizable) forward and backpropagation at each step. To address this issue, we propose a novel method called Random SAM prompt tuning (RSAM-PT) to improve the model generalization, requiring only one-step gradient computation at each step. Specifically, we search for the gradient descent direction within a random neighborhood of the parameters during each gradient update. To amplify the impact of tail-class samples and avoid overfitting, we employ the deferred re-weight scheme to increase the significance of tail-class samples. The classification accuracy of long-tailed data can be significantly improved by the proposed RSAM-PT, particularly for tail classes. RSAM-PT achieves the state-of-the-art performance of 90.3\%, 76.5\%, and 50.1\% on benchmark datasets CIFAR100-LT (IF 100), iNaturalist 2018, and Places-LT, respectively. The source code is temporarily available at https://github.com/Keke921/GNM-PT.

Motivation & Objective

• Address the poor generalization of visual prompt tuning (VPT) on tail classes in long-tailed datasets.
• Improve model generalization across all classes, especially underrepresented tail classes, by flattening the loss landscape.
• Develop an efficient optimization strategy that avoids additional gradient computations while maintaining performance.
• Provide a theoretically grounded alternative to Sharpness-Aware Minimization (SAM) tailored for long-tailed learning.
• Enhance VPT performance by integrating high-level prompt information with class tokens for better classification.

Proposed method

• Introduce a novel Gaussian neighborhood loss (GNM) that minimizes the expected loss over a random Gaussian perturbation of model parameters.
• Use random sampling from a normal distribution to approximate the neighborhood mean, avoiding explicit gradient computation for sharpness estimation.
• Replace SAM’s maximum-based neighborhood loss with a mean-based loss, providing a tighter upper bound on the true loss distribution.
• Apply the GNM loss during VPT training to guide optimization toward flatter minima, improving generalization.
• Merge the learned visual prompt with the class token to enhance classification representation in ViT-based models.
• The method is parameter-efficient and computationally lightweight, requiring no additional backpropagation beyond standard VPT.

Experimental results

Research questions

• RQ1 Can a mean-based neighborhood loss provide a tighter bound on the true loss distribution than maximum-based approaches like SAM?
• RQ2 Does minimizing the expected loss over a Gaussian parameter neighborhood lead to flatter minima and improved generalization in long-tailed recognition?
• RQ3 Can GNM-PT enhance VPT performance on tail classes without increasing computational cost?
• RQ4 How does GNM-PT compare to SAM and other SOTA methods in terms of accuracy and generalization across head and tail classes?
• RQ5 Is the integration of prompts with class tokens effective in boosting classification performance in long-tailed settings?

Key findings

• GNM-PT achieves state-of-the-art accuracy of 90.3% on CIFAR100-LT with an imbalance ratio of 100, significantly outperforming prior methods.
• On iNaturalist 2018, GNM-PT attains 76.5% top-1 accuracy, demonstrating strong generalization on long-tailed benchmarks with fine-grained categories.
• On Places-LT, GNM-PT achieves 50.1% accuracy, highlighting its effectiveness in fine-grained, long-tailed recognition under extreme class imbalance.
• The method improves generalization across all classes, with notable gains on tail classes, while maintaining computational efficiency.
• Abllation studies confirm that GNM-PT enhances model performance without increasing training cost, validating its efficiency and robustness.
• Theoretical analysis supports that the mean-based neighborhood loss provides a tighter upper bound than maximum-based alternatives, justifying its superior optimization behavior.