[Paper Review] Efficient Self-supervised Continual Learning with Progressive Task-correlated Layer Freezing

Li Yang, Sen Lin+3 more|arXiv (Cornell University)|Mar 13, 2023

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes Progressive Task-correlated Layer Freezing (PTLF), a method for efficient self-supervised continual learning (SSCL) that reduces training cost and memory usage by progressively freezing layers with the highest task correlation ratios. Experiments show 12–14% faster training, 23–26% less memory, and 33–35% fewer backward FLOPs compared to SOTA methods like LUMP, with reduced forgetting and no accuracy drop.

ABSTRACT

Inspired by the success of Self-supervised learning (SSL) in learning visual representations from unlabeled data, a few recent works have studied SSL in the context of continual learning (CL), where multiple tasks are learned sequentially, giving rise to a new paradigm, namely self-supervised continual learning (SSCL). It has been shown that the SSCL outperforms supervised continual learning (SCL) as the learned representations are more informative and robust to catastrophic forgetting. However, if not designed intelligently, the training complexity of SSCL may be prohibitively high due to the inherent training cost of SSL. In this work, by investigating the task correlations in SSCL setup first, we discover an interesting phenomenon that, with the SSL-learned background model, the intermediate features are highly correlated between tasks. Based on this new finding, we propose a new SSCL method with layer-wise freezing which progressively freezes partial layers with the highest correlation ratios for each task to improve training computation efficiency and memory efficiency. Extensive experiments across multiple datasets are performed, where our proposed method shows superior performance against the SoTA SSCL methods under various SSL frameworks. For example, compared to LUMP, our method achieves 12\%/14\%/12\% GPU training time reduction, 23\%/26\%/24\% memory reduction, 35\%/34\%/33\% backward FLOPs reduction, and 1.31\%/1.98\%/1.21\% forgetting reduction without accuracy degradation on three datasets, respectively.

Motivation & Objective

• To address the high computational and memory cost of self-supervised continual learning (SSCL), which limits its real-world deployment.
• To investigate task correlations in SSCL and exploit them to reduce training overhead without sacrificing performance.
• To develop a method that selectively freezes layers based on task correlation to improve training and memory efficiency.
• To mitigate catastrophic forgetting in SSCL while significantly reducing computational cost.
• To demonstrate that SSCL representations are more robust to aggressive layer freezing than supervised learning.

Proposed method

• The method introduces a task correlation ratio based on gradient projection norms to quantify similarity between current and prior tasks.
• Layers are ranked by their correlation ratios and progressively frozen in descending order for each task to reduce computation.
• Freezing is applied layer-wise during training, with a fixed ratio (e.g., 0.4) of top-correlated layers frozen per task.
• The approach is integrated into existing SSL frameworks (e.g., SimSiam, BarlowTwins) without architectural changes.
• The freezing schedule is adaptive per task and based on inter-task feature similarity, not fixed layer order.
• The method is evaluated across multiple benchmarks: Split CIFAR-10, CIFAR-100, TinyImageNet, and ImageNet-100.

Figure 1: The overview of our proposed method which progressively freezes partial layers during the whole training process for each task.

Experimental results

Research questions

• RQ1 How do intermediate features in self-supervised continual learning correlate across sequential tasks?
• RQ2 Can task correlation ratios derived from gradient projections be used to guide efficient layer freezing in SSCL?
• RQ3 Does progressive freezing of high-correlation layers reduce training cost and memory usage while maintaining performance?
• RQ4 How does the proposed method compare to naive layer freezing (e.g., by index order) in terms of forgetting and accuracy?
• RQ5 Is SSCL more robust to aggressive layer freezing than supervised continual learning?

Key findings

• Compared to LUMP, the proposed method reduces GPU training time by 12% on Split CIFAR-10, 14% on CIFAR-100, and 12% on TinyImageNet.
• Memory usage is reduced by 23% on CIFAR-10, 26% on CIFAR-100, and 24% on TinyImageNet, with no accuracy degradation.
• Backward FLOPs are reduced by 35% on CIFAR-10, 34% on CIFAR-100, and 33% on TinyImageNet.
• Catastrophic forgetting is reduced by 1.31% on CIFAR-10, 1.98% on CIFAR-100, and 1.21% on TinyImageNet compared to LUMP.
• The method maintains or improves accuracy while significantly reducing computational and memory costs across all evaluated datasets and SSL frameworks.
• Layer freezing decisions are highly consistent across tasks, indicating that SSCL representations are general and robust to freezing.

Figure 3: The overlap of layer selection for different tasks