[Paper Review] Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models

Zangwei Zheng, Mingyuan Ma+4 more|arXiv (Cornell University)|Mar 12, 2023

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes ZSCL, a novel continual learning method for vision-language models that prevents zero-shot transfer degradation by distilling from an initial pre-trained model on a semantic-diverse reference dataset and using weight ensemble to stabilize parameter updates. ZSCL achieves state-of-the-art performance, improving average accuracy by 9.7% over prior methods on a new Multi-domain Task Incremental Learning benchmark.

ABSTRACT

Continual learning (CL) can help pre-trained vision-language models efficiently adapt to new or under-trained data distributions without re-training. Nevertheless, during the continual training of the Contrastive Language-Image Pre-training (CLIP) model, we observe that the model's zero-shot transfer ability significantly degrades due to catastrophic forgetting. Existing CL methods can mitigate forgetting by replaying previous data. However, since the CLIP dataset is private, replay methods cannot access the pre-training dataset. In addition, replaying data of previously learned downstream tasks can enhance their performance but comes at the cost of sacrificing zero-shot performance. To address this challenge, we propose a novel method ZSCL to prevent zero-shot transfer degradation in the continual learning of vision-language models in both feature and parameter space. In the feature space, a reference dataset is introduced for distillation between the current and initial models. The reference dataset should have semantic diversity but no need to be labeled, seen in pre-training, or matched image-text pairs. In parameter space, we prevent a large parameter shift by averaging weights during the training. We propose a more challenging Multi-domain Task Incremental Learning (MTIL) benchmark to evaluate different methods, where tasks are from various domains instead of class-separated in a single dataset. Our method outperforms other methods in the traditional class-incremental learning setting and the MTIL by 9.7% average score. Our code locates at https://github.com/Thunderbeee/ZSCL.

Motivation & Objective

• Address catastrophic forgetting in continual learning of pre-trained vision-language models, particularly the degradation of zero-shot transfer ability.
• Overcome the limitation of replay-based methods due to the private nature of the CLIP pre-training dataset.
• Develop a method that preserves both feature space knowledge and parameter stability during incremental learning.
• Introduce a more challenging Multi-domain Task Incremental Learning (MTIL) benchmark to better evaluate continual learning methods.
• Achieve robust performance across different training orderings and task distributions, especially in zero-shot generalization.

Proposed method

• Introduce a reference dataset with semantic diversity but no labels or paired image-text data for distillation in feature space.
• Use the original pre-trained CLIP model as the teacher for knowledge distillation, rather than the current fine-tuned model.
• Apply knowledge distillation loss to preserve the relative similarity between reference images and text embeddings, minimizing feature space deviation.
• Implement a weight ensemble strategy across training stages to prevent large parameter shifts, effectively interpolating between initial and fine-tuned model weights.
• Combine feature-space distillation and parameter-space regularization to jointly mitigate forgetting in both representation and model weight spaces.
• Design a Multi-domain Task Incremental Learning (MTIL) benchmark with tasks from diverse domains (e.g., food, vehicles, landscapes) to test generalization and robustness.

Experimental results

Research questions

• RQ1 Can we prevent zero-shot transfer degradation in continual learning of vision-language models when the pre-training dataset is inaccessible?
• RQ2 How effective is knowledge distillation from the initial pre-trained model using a non-labeled, diverse reference dataset in preserving zero-shot performance?
• RQ3 Does parameter space regularization via weight ensemble improve stability and generalization in continual vision-language learning?
• RQ4 How does the proposed method perform under challenging, real-world-like training orders and multi-domain task sequences?
• RQ5 Can a new benchmark like MTIL better evaluate the robustness and generalization of continual learning methods for vision-language models?

Key findings

• ZSCL improves average accuracy by 9.7% over prior methods on the proposed Multi-domain Task Incremental Learning (MTIL) benchmark.
• On CIFAR100 in class-incremental learning, ZSCL achieves 82.15% average accuracy and 73.65% last-task accuracy, outperforming the second-best method (iCaRL) by +7.68% and +7.73% respectively.
• On TinyImageNet, ZSCL achieves 80.27% average accuracy and 73.57% last-task accuracy, improving by +10.65% and +8.27% over the previous state-of-the-art (iCaRL).
• ZSCL maintains consistent performance across different training orders (e.g., Order-I vs. Order-II), with only a 0.2% drop in Last score, indicating high robustness.
• The method significantly reduces catastrophic forgetting in both zero-shot transfer and downstream task performance, especially under large step counts.
• Ablation studies confirm that using the original CLIP model as the teacher and a diverse reference dataset are critical for performance gains.