[Paper Review] Towards Better Plasticity-Stability Trade-off in Incremental Learning: A Simple Linear Connector

Guoliang Lin, Hanglu Chu+1 more|arXiv (Cornell University)|Oct 15, 2021

Domain Adaptation and Few-Shot Learning37 references4 citations

TL;DR

This paper proposes a simple linear connector that combines a null-space projected network for previous task stability and a standard SGD-trained network for current task plasticity, achieving state-of-the-art performance in data-free incremental learning. The method improves the plasticity-stability trade-off without storing past data, attaining 79.79% accuracy on 10-split-CIFAR-100 and 64.61% on 25-split-TinyImageNet, outperforming prior methods like Adam-NSCL.

ABSTRACT

Plasticity-stability dilemma is a main problem for incremental learning, where plasticity is referring to the ability to learn new knowledge, and stability retains the knowledge of previous tasks. Many methods tackle this problem by storing previous samples, while in some applications, training data from previous tasks cannot be legally stored. In this work, we propose to employ mode connectivity in loss landscapes to achieve better plasticity-stability trade-off without any previous samples. We give an analysis of why and how to connect two independently optimized optima of networks, null-space projection for previous tasks and simple SGD for the current task, can attain a meaningful balance between preserving already learned knowledge and granting sufficient flexibility for learning a new task. This analysis of mode connectivity also provides us a new perspective and technology to control the trade-off between plasticity and stability. We evaluate the proposed method on several benchmark datasets. The results indicate our simple method can achieve notable improvement, and perform well on both the past and current tasks. On 10-split-CIFAR-100 task, our method achieves 79.79% accuracy, which is 6.02% higher. Our method also achieves 6.33% higher accuracy on TinyImageNet. Code is available at https://github.com/lingl1024/Connector.

Motivation & Objective

• To address the plasticity-stability dilemma in incremental learning without storing past data.
• To develop a method that balances knowledge retention of previous tasks and learning capacity for new tasks.
• To leverage mode connectivity in loss landscapes to connect independently optimized network minima.
• To provide a theoretically grounded, sample-free alternative to existing rehearsal and regularization-based methods.
• To achieve superior performance on benchmark datasets with minimal architectural or training complexity.

Proposed method

• Train a first network using null-space projection to preserve knowledge from previous tasks.
• Train a second network via standard SGD on the current task to maximize plasticity.
• Construct a linear connector by averaging the two networks with a learnable weight β.
• Theoretical analysis shows the linear path minimizes an upper bound on empirical loss across all tasks.
• The method operates without storing any previous data, satisfying the data-free IL constraint.
• The connector is optimized via a simple averaging scheme that balances stability and plasticity.

Experimental results

Research questions

• RQ1 Can a linear path between two independently trained network minima achieve high performance in incremental learning without past data?
• RQ2 How can mode connectivity in loss landscapes be leveraged to improve the plasticity-stability trade-off?
• RQ3 What is the optimal balance between stability (knowledge retention) and plasticity (new task learning) in data-free IL?
• RQ4 Can a simple averaging of two specialized networks (one for stability, one for plasticity) outperform complex regularization or rehearsal methods?
• RQ5 Does the proposed method maintain high accuracy across all tasks, including earlier ones, without catastrophic forgetting?

Key findings

• The proposed method achieves 79.79% accuracy on the 10-split-CIFAR-100 benchmark, a 6.02% improvement over prior methods.
• On 20-split-CIFAR-100, the method reaches 80.80% accuracy, outperforming Adam-NSCL by 4.85%.
• On 25-split-TinyImageNet, the method achieves 64.61% accuracy, a 6.33% improvement over the baseline.
• The method achieves a better plasticity-stability trade-off, with a BWT of -0.92 (vs. -1.6 for Adam-NSCL) and an Intransigence Measure of 8.10 (vs. 14.50), indicating superior plasticity.
• Ablation studies confirm that β ≈ 1/t provides the best balance between stability and plasticity across all datasets.
• The linear path between the two networks shows smooth accuracy transitions, with no sharp drops, confirming stable generalization along the connector.