[Paper Review] Kaizen: Practical Self-supervised Continual Learning with Continual Fine-tuning

Chi Ian Tang, Lorena Qendro+4 more|arXiv (Cornell University)|Mar 30, 2023

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

Kaizen proposes a practical self-supervised continual learning framework that mitigates catastrophic forgetting by jointly training a feature extractor and classifier using both unlabeled and labeled data at each task via a novel loss function with continual knowledge distillation. It achieves up to 16.5% higher accuracy and 25.4% less forgetting than prior methods on split CIFAR-100, outperforming state-of-the-art SSL-based continual learning models by balancing knowledge retention and new task learning end-to-end.

ABSTRACT

Self-supervised learning (SSL) has shown remarkable performance in computer vision tasks when trained offline. However, in a Continual Learning (CL) scenario where new data is introduced progressively, models still suffer from catastrophic forgetting. Retraining a model from scratch to adapt to newly generated data is time-consuming and inefficient. Previous approaches suggested re-purposing self-supervised objectives with knowledge distillation to mitigate forgetting across tasks, assuming that labels from all tasks are available during fine-tuning. In this paper, we generalize self-supervised continual learning in a practical setting where available labels can be leveraged in any step of the SSL process. With an increasing number of continual tasks, this offers more flexibility in the pre-training and fine-tuning phases. With Kaizen, we introduce a training architecture that is able to mitigate catastrophic forgetting for both the feature extractor and classifier with a carefully designed loss function. By using a set of comprehensive evaluation metrics reflecting different aspects of continual learning, we demonstrated that Kaizen significantly outperforms previous SSL models in competitive vision benchmarks, with up to 16.5% accuracy improvement on split CIFAR-100. Kaizen is able to balance the trade-off between knowledge retention and learning from new data with an end-to-end model, paving the way for practical deployment of continual learning systems.

Motivation & Objective

• To address catastrophic forgetting in self-supervised continual learning (CSSL) under realistic data constraints where labeled data from previous tasks is unavailable.
• To design a framework that enables end-to-end training using both unlabeled and labeled data at every task, supporting deployment at any stage of the continual learning process.
• To balance knowledge retention and learning from new data through a unified training objective that combines feature extraction and classifier fine-tuning.
• To evaluate the method across comprehensive metrics, including continual accuracy, final accuracy, forgetting, and forward transfer, under practical assumptions.
• To demonstrate that continual distillation across both feature and classifier heads significantly improves performance and stability in real-world continual learning scenarios.

Proposed method

• Introduces a novel training objective that combines contrastive self-supervision with knowledge distillation for both the feature extractor and classifier heads.
• Employs a dual-stream distillation mechanism: one for the feature encoder using momentum-averaged student-teacher networks, and another for the classifier using soft labels from previous task predictions.
• Uses a replay buffer with a configurable size to store and re-engage with past data, improving retention without requiring full retraining.
• Applies continual fine-tuning at each task using both unlabeled data for contrastive pre-training and labeled data for classifier adaptation and distillation.
• Designs a loss function that balances self-supervised contrastive learning, distillation loss for feature representation, and distillation loss for classification head predictions.
• Supports deployment at any point during training with a functional classifier, enabling practical real-world deployment.

Experimental results

Research questions

• RQ1 Can a self-supervised continual learning framework effectively mitigate catastrophic forgetting when labeled data from earlier tasks is not available?
• RQ2 How does continual fine-tuning with both unlabeled and labeled data at each task improve performance compared to standard SSL or supervised CL methods?
• RQ3 To what extent does joint distillation on feature and classifier heads reduce forgetting while maintaining high accuracy on new tasks?
• RQ4 What is the trade-off between knowledge retention, forward transfer, and replay data size in a practical continual learning setup?
• RQ5 How does the proposed method compare to state-of-the-art SSL and continual learning baselines across multiple evaluation metrics?

Key findings

• Kaizen achieves up to 16.5% absolute accuracy improvement over prior SSL-based continual learning methods on split CIFAR-100, with the best performance using MoCoV2+ as the backbone.
• The method reduces forgetting by 25.4% compared to baseline methods, as measured by the forgetting metric, across all evaluated SSL techniques.
• With the MoCoV2+ backbone, Kaizen achieves a final accuracy of 0.570 and continual accuracy of 0.409, significantly outperforming the next best method (CaSSLe) by 0.138 and 0.088 respectively.
• Even without any replay data (0% replay), Kaizen maintains strong performance, demonstrating robustness under strict data constraints.
• Increasing replay data size improves accuracy and reduces forgetting but reduces forward transfer, highlighting a key trade-off in system design.
• Kaizen shows negative forward transfer, indicating a trade-off between retention and learning new tasks, but still achieves the best overall balance among all evaluated methods.