[Paper Review] Introducing Language Guidance in Prompt-based Continual Learning

Muhammad Gul Zain Ali Khan, Muhammad Ferjad Naeem+4 more|arXiv (Cornell University)|Aug 30, 2023

Domain Adaptation and Few-Shot LearningComputer Science3 citations

TL;DR

This paper proposes Language Guidance for Prompt-based Continual Learning (LGCL), a plug-in method that enhances prompt-based continual learning by aligning prompt pool keys and vision encoder outputs with pre-trained language embeddings. Without additional learnable parameters, LGCL improves generalization and reduces catastrophic forgetting, setting a new state-of-the-art on benchmark datasets like Split CIFAR-100 and ImageNet-R.

ABSTRACT

Continual Learning aims to learn a single model on a sequence of tasks without having access to data from previous tasks. The biggest challenge in the domain still remains catastrophic forgetting: a loss in performance on seen classes of earlier tasks. Some existing methods rely on an expensive replay buffer to store a chunk of data from previous tasks. This, while promising, becomes expensive when the number of tasks becomes large or data can not be stored for privacy reasons. As an alternative, prompt-based methods have been proposed that store the task information in a learnable prompt pool. This prompt pool instructs a frozen image encoder on how to solve each task. While the model faces a disjoint set of classes in each task in this setting, we argue that these classes can be encoded to the same embedding space of a pre-trained language encoder. In this work, we propose Language Guidance for Prompt-based Continual Learning (LGCL) as a plug-in for prompt-based methods. LGCL is model agnostic and introduces language guidance at the task level in the prompt pool and at the class level on the output feature of the vision encoder. We show with extensive experimentation that LGCL consistently improves the performance of prompt-based continual learning methods to set a new state-of-the art. LGCL achieves these performance improvements without needing any additional learnable parameters.

Motivation & Objective

• To address catastrophic forgetting in continual learning without relying on data replay or architecture expansion.
• To leverage the semantic consistency of pre-trained language models as a unified representation space for disjoint task classes.
• To improve prompt-based continual learning by introducing language guidance at both task-level (prompt pool keys) and class-level (output features).
• To develop a model-agnostic method that enhances performance without adding trainable parameters.
• To validate the robustness and effectiveness of language guidance across different text encoders and key initialization strategies.

Proposed method

• Introduces task-level language guidance by aligning learnable keys in the prompt pool with the CLIP- or BERT-style text embedding of the entire task's class labels.
• Applies class-level language guidance by aligning the output feature of the vision encoder with the language embedding of the ground-truth class label.
• Uses contrastive loss to minimize the distance between the prompt key embeddings and the corresponding task-level language representation.
• Uses contrastive loss to minimize the distance between the vision encoder's output feature and the class-level language embedding.
• Employs a pre-trained text encoder (e.g., CLIP, RoBERTa, BERT) to provide semantic embeddings for both task and class-level guidance.
• Maintains model-agnostic design by integrating only loss functions, not additional parameters, into existing prompt-based continual learning frameworks.

Experimental results

Research questions

• RQ1 Can language embeddings serve as a shared semantic space to unify disjoint class sets across continual learning tasks?
• RQ2 Does introducing language guidance at the prompt pool key level improve task generalization and reduce forgetting?
• RQ3 Can aligning the vision encoder’s output features with class-level language representations enhance classification accuracy and robustness?
• RQ4 How does the choice of text encoder (e.g., CLIP vs. BERT) affect the performance of the language-guided method?
• RQ5 Does the method remain effective when keys in the prompt pool are frozen or learnable?

Key findings

• LGCL achieves a new state-of-the-art on Split CIFAR-100, with Dual Prompt + LGCL reaching 87.23% average accuracy and only 5.10% forgetting using CLIP as the text encoder.
• On the more challenging Split ImageNet-R, LGCL improves Dual Prompt from 69.46% to 69.46% average accuracy and reduces forgetting from 4.26% to 4.20% when both task- and class-level guidance are applied.
• The ablation study confirms that both task-level and class-level language guidance are necessary, as removing either component leads to performance degradation.
• Using CLIP’s text encoder yields the best performance (87.23% accuracy), though BERT and RoBERTa also show strong results, indicating robustness to text encoder choice.
• Freezing CLIP’s text encoder keys results in lower performance (86.15% accuracy) compared to learnable keys (87.23%), demonstrating the benefit of optimizing key representations.
• The method achieves performance gains without adding any learnable parameters, making it highly efficient and deployable in resource-constrained settings.