[Paper Review] Self-regulating Prompts: Foundational Model Adaptation without Forgetting

Muhammad Uzair Khattak, Syed Talal Wasim+4 more|arXiv (Cornell University)|Jul 13, 2023

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes PromptSRC, a self-regulating prompt learning framework that prevents overfitting in foundational vision-language models like CLIP by jointly optimizing for task-specific and task-agnostic representations. It uses mutual agreement with frozen CLIP features, Gaussian-weighted self-ensembling over training trajectories, and textual diversity via multiple prompt templates, achieving state-of-the-art performance with +1.42% harmonic mean gain over MaPLe and +8.26% over CLIP on base-to-novel generalization benchmarks.

ABSTRACT

Prompt learning has emerged as an efficient alternative for fine-tuning foundational models, such as CLIP, for various downstream tasks. Conventionally trained using the task-specific objective, i.e., cross-entropy loss, prompts tend to overfit downstream data distributions and find it challenging to capture task-agnostic general features from the frozen CLIP. This leads to the loss of the model's original generalization capability. To address this issue, our work introduces a self-regularization framework for prompting called PromptSRC (Prompting with Self-regulating Constraints). PromptSRC guides the prompts to optimize for both task-specific and task-agnostic general representations using a three-pronged approach by: (a) regulating prompted representations via mutual agreement maximization with the frozen model, (b) regulating with self-ensemble of prompts over the training trajectory to encode their complementary strengths, and (c) regulating with textual diversity to mitigate sample diversity imbalance with the visual branch. To the best of our knowledge, this is the first regularization framework for prompt learning that avoids overfitting by jointly attending to pre-trained model features, the training trajectory during prompting, and the textual diversity. PromptSRC explicitly steers the prompts to learn a representation space that maximizes performance on downstream tasks without compromising CLIP generalization. We perform extensive experiments on 4 benchmarks where PromptSRC overall performs favorably well compared to the existing methods. Our code and pre-trained models are publicly available at: https://github.com/muzairkhattak/PromptSRC.

Motivation & Objective

• To address prompt overfitting in foundational vision-language models like CLIP, which compromises their generalization to new tasks.
• To preserve the original zero-shot generalization capability of frozen CLIP while adapting to downstream tasks.
• To develop a self-regulating prompt learning framework that balances task-specific adaptation with task-agnostic representation learning.
• To mitigate data imbalance between visual and textual modality supervision, especially in low-resource settings.
• To improve generalization under domain shift, few-shot learning, and cross-dataset transfer without fine-tuning.

Proposed method

• Introduces a mutual agreement maximization loss between prompted and frozen CLIP features to preserve generalizable knowledge.
• Employs a Gaussian-weighted prompt aggregation (GPA) strategy that combines prompts across training epochs to stabilize learning and leverage complementary strengths.
• Enhances textual supervision by using multiple diverse prompt templates per class to improve diversity and reduce bias in text encoder features.
• Applies a feature-level consistency loss (LSCL) using L1 distance metric to enforce alignment between visual and textual prompt representations.
• Maintains inference efficiency by pre-computing CLIP text features and requiring only one additional forward pass for visual features during training.
• Uses a three-pronged regularization strategy: mutual agreement with frozen model, self-ensemble over training trajectory, and textual diversity via template augmentation.

Experimental results

Research questions

• RQ1 Can self-regulation via mutual agreement with frozen CLIP features improve generalization without sacrificing downstream performance?
• RQ2 Does self-ensembling of prompts over training trajectories enhance robustness and generalization in prompt learning?
• RQ3 Can increasing textual diversity through multiple prompt templates mitigate imbalance and improve representation quality?
• RQ4 How does PromptSRC perform under domain shift, few-shot, and cross-dataset generalization settings?
• RQ5 What is the trade-off between performance gain and computational cost in self-regulated prompt learning?

Key findings

• On the base-to-novel generalization benchmark across 11 datasets, PromptSRC achieves a harmonic mean of 79.97%, outperforming MaPLe by +1.42% and CLIP by +8.26%.
• PromptSRC achieves state-of-the-art performance in domain generalization, with an average accuracy of 60.65% on out-of-distribution datasets, surpassing all prior methods.
• The method maintains low training cost, with only 9.3% longer training time than IVLP and 0.13x higher GFLOPs, while preserving the same inference speed and GFLOPs as baseline methods.
• Ablation studies show that L1 matching loss outperforms cosine similarity and MSE in consistency loss, and Gaussian-weighted prompt aggregation yields better performance than equal-weighted averaging.
• Increasing the number of textual templates improves performance, indicating that textual diversity enhances supervision and representation quality.
• Optimal performance is achieved with 4 vision-language prompts and a prompt length of 4 tokens, balancing expressiveness and generalization.