[Paper Review] AttriCLIP: A Non-Incremental Learner for Incremental Knowledge Learning

Runqi Wang, Xiaoyue Duan+6 more|arXiv (Cornell University)|May 19, 2023

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

AttriCLIP proposes a non-incremental continual learning framework that leverages a frozen CLIP backbone and learns task-specific attribute prompts from a fixed attribute word bank, eliminating the need for parameter expansion or replay memory. It achieves state-of-the-art performance on long-sequence, domain-shift continual learning benchmarks by using dynamic, attribute-based textual prompts for zero-shot classification.

ABSTRACT

Continual learning aims to enable a model to incrementally learn knowledge from sequentially arrived data. Previous works adopt the conventional classification architecture, which consists of a feature extractor and a classifier. The feature extractor is shared across sequentially arrived tasks or classes, but one specific group of weights of the classifier corresponding to one new class should be incrementally expanded. Consequently, the parameters of a continual learner gradually increase. Moreover, as the classifier contains all historical arrived classes, a certain size of the memory is usually required to store rehearsal data to mitigate classifier bias and catastrophic forgetting. In this paper, we propose a non-incremental learner, named AttriCLIP, to incrementally extract knowledge of new classes or tasks. Specifically, AttriCLIP is built upon the pre-trained visual-language model CLIP. Its image encoder and text encoder are fixed to extract features from both images and text. Text consists of a category name and a fixed number of learnable parameters which are selected from our designed attribute word bank and serve as attributes. As we compute the visual and textual similarity for classification, AttriCLIP is a non-incremental learner. The attribute prompts, which encode the common knowledge useful for classification, can effectively mitigate the catastrophic forgetting and avoid constructing a replay memory. We evaluate our AttriCLIP and compare it with CLIP-based and previous state-of-the-art continual learning methods in realistic settings with domain-shift and long-sequence learning. The results show that our method performs favorably against previous state-of-the-arts. The implementation code can be available at https://github.com/bhrqw/AttriCLIP.

Motivation & Objective

• Address catastrophic forgetting and classifier bias in continual learning without expanding model parameters.
• Eliminate the need for replay memory by leveraging pre-trained visual-language models with dynamic prompt tuning.
• Enable incremental knowledge learning in a non-incremental manner, where model parameters remain fixed after initial pre-training.
• Improve generalization and stability in long-sequence, domain-shift continual learning scenarios using attribute-aware textual supervision.
• Develop a prompt-based framework that adapts to new classes without fine-tuning the vision or text encoders.

Proposed method

• Utilizes the frozen image and text encoders of CLIP as fixed feature extractors.
• Introduces a learnable attribute word bank containing (key, prompt) pairs, where keys represent image attributes and prompts are tunable textual descriptions.
• For each input image, selects top-k prompts based on similarity between image attribute features and key embeddings.
• Concatenates selected prompts with the class name to form a dynamic text description for contrastive classification.
• Performs end-to-end training on the prompt parameters only, keeping the CLIP encoders frozen.
• Uses contrastive loss between image features and the combined text representation (class name + selected prompts) for classification.

Experimental results

Research questions

• RQ1 Can a non-incremental learner achieve stable continual learning without parameter expansion or replay memory?
• RQ2 Can attribute-based prompt tuning effectively mitigate catastrophic forgetting in long-sequence, domain-shift continual learning?
• RQ3 How does the performance of AttriCLIP compare to SOTA methods in zero-shot and few-shot continual learning settings?
• RQ4 To what extent do learned prompts capture meaningful, class-discriminative attributes across different image categories?
• RQ5 Can a fixed-size attribute word bank generalize across diverse, sequentially arriving datasets without catastrophic forgetting?

Key findings

• AttriCLIP achieves 82.3% average accuracy on CIFAR100 when trained incrementally after ImageNet100, outperforming CLIP-based and prior SOTA methods without any replay memory.
• On the ImageNet100 + CIFAR100 long-sequence benchmark, AttriCLIP achieves 78.3% accuracy, significantly outperforming DualPrompt (67.1%) and L2P (64.6%).
• In the ImageNet100 benchmark, AttriCLIP achieves 90.3% accuracy after fine-tuning from ImageNet100, demonstrating strong stability and low forgetting.
• The method shows robust generalization to new datasets, with minimal performance drop on previously unseen domains.
• Visualization via Grad-CAM confirms that learned prompts attend to semantically meaningful image attributes, such as 'hat' or 'lying on grass'.
• AttriCLIP avoids classifier bias and parameter growth, maintaining a fixed model size throughout continual learning.