[Paper Review] New Insights for the Stability-Plasticity Dilemma in Online Continual Learning

Dahuin Jung, Dongjin Lee+4 more|arXiv (Cornell University)|Feb 17, 2023

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes MuFAN, a novel online continual learning framework that addresses the stability-plasticity dilemma by leveraging multi-scale features from a pre-trained encoder, a structure-wise distillation loss to preserve knowledge across tasks, and a stability-plasticity normalization (SPN) module that separately optimizes for stability and plasticity. MuFAN achieves state-of-the-art performance on SVHN, CIFAR100, miniImageNet, and CORe50, outperforming existing methods by up to 6.5% in average accuracy.

ABSTRACT

The aim of continual learning is to learn new tasks continuously (i.e., plasticity) without forgetting previously learned knowledge from old tasks (i.e., stability). In the scenario of online continual learning, wherein data comes strictly in a streaming manner, the plasticity of online continual learning is more vulnerable than offline continual learning because the training signal that can be obtained from a single data point is limited. To overcome the stability-plasticity dilemma in online continual learning, we propose an online continual learning framework named multi-scale feature adaptation network (MuFAN) that utilizes a richer context encoding extracted from different levels of a pre-trained network. Additionally, we introduce a novel structure-wise distillation loss and replace the commonly used batch normalization layer with a newly proposed stability-plasticity normalization module to train MuFAN that simultaneously maintains high plasticity and stability. MuFAN outperforms other state-of-the-art continual learning methods on the SVHN, CIFAR100, miniImageNet, and CORe50 datasets. Extensive experiments and ablation studies validate the significance and scalability of each proposed component: 1) multi-scale feature maps from a pre-trained encoder, 2) the structure-wise distillation loss, and 3) the stability-plasticity normalization module in MuFAN. Code is publicly available at https://github.com/whitesnowdrop/MuFAN.

Motivation & Objective

• To address the heightened plasticity-vulnerability in online continual learning due to limited single-data-point training signals.
• To improve both stability (forgetting prevention) and plasticity (new task learning) simultaneously in streaming data settings.
• To overcome limitations of existing methods that rely on single-scale features, point-wise distillation, or standard normalization layers.
• To design a unified framework that scales effectively across diverse continual learning benchmarks.
• To validate the individual contributions of multi-scale features, structure-wise distillation, and the SPN module through ablation studies.

Proposed method

• MuFAN uses multi-scale feature maps extracted from shallow to deep layers of a pre-trained encoder to enrich contextual representation for downstream classification.
• It introduces a structure-wise distillation loss that leverages relationships between non-consecutive tasks in a replay buffer to generate additional knowledge-preserving signals.
• The stability-plasticity normalization (SPN) module applies batch normalization for plasticity and spatial normalization (IN/LN) for stability in parallel, optimizing both objectives simultaneously.
• The framework integrates these components into an end-to-end training pipeline that maintains high performance across sequential tasks without access to old data.
• The structure-wise distillation loss is computed as a contrastive loss over feature maps from different tasks, using arccos-based similarity to measure consistency.
• Ablation studies validate the necessity and scalability of each component, including the use of multi-scale features and the SPN module.

Experimental results

Research questions

• RQ1 How can multi-scale features from a pre-trained encoder improve performance in online continual learning?
• RQ2 Can a structure-wise distillation loss that leverages non-consecutive task relationships reduce catastrophic forgetting more effectively than point-wise distillation?
• RQ3 Does a dual-branch normalization module that separately optimizes for stability and plasticity outperform standard normalization layers in online CL?
• RQ4 What is the optimal number of samples per task needed to effectively utilize the structure-wise distillation loss?
• RQ5 How does MuFAN compare to state-of-the-art methods across diverse continual learning benchmarks?

Key findings

• MuFAN achieves state-of-the-art performance on SVHN, CIFAR100, miniImageNet, and CORe50, with a 6.5% improvement in average accuracy over previous SOTA on CORe50.
• The structure-wise distillation loss significantly reduces forgetting, with performance stabilizing at N ≥ 5 samples per task, and no degradation observed at N = 10 or 20.
• The SPN module outperforms all baseline normalization layers, including SN and CN, with a 52.80% accuracy on CORe50 and 61.90% LA, exceeding CN by 1.62% and 2.62% respectively.
• Ablation studies confirm that all three components—multi-scale features, structure-wise distillation, and SPN—contribute significantly to performance, with SPN showing the highest impact on stability.
• The SPN module with IN and LN for stability and BN for plasticity achieves 72.7% LA on miniImageNet, outperforming BN+IN and BN+LN by 5.3 and 5.3 percentage points respectively.
• The framework demonstrates strong scalability, as integrating multi-scale features into existing online CL methods like ER-Ring leads to consistent performance gains.