[Paper Review] Overcoming Catastrophic Forgetting by Incremental Moment Matching

Sang-Woo Lee, Jin-Hwa Kim+3 more|arXiv (Cornell University)|Mar 24, 2017

Domain Adaptation and Few-Shot LearningComputer Science293 citations

TL;DR

IMM Incrementally matches posterior moments to alleviate catastrophic forgetting; mean-IMM and mode-IMM balance old/new tasks with transfer techniques like weight-transfer, L2-transfer, and drop-transfer, achieving state-of-the-art continual learning across multiple datasets.

ABSTRACT

Catastrophic forgetting is a problem of neural networks that loses the information of the first task after training the second task. Here, we propose a method, i.e. incremental moment matching (IMM), to resolve this problem. IMM incrementally matches the moment of the posterior distribution of the neural network which is trained on the first and the second task, respectively. To make the search space of posterior parameter smooth, the IMM procedure is complemented by various transfer learning techniques including weight transfer, L2-norm of the old and the new parameter, and a variant of dropout with the old parameter. We analyze our approach on a variety of datasets including the MNIST, CIFAR-10, Caltech-UCSD-Birds, and Lifelog datasets. The experimental results show that IMM achieves state-of-the-art performance by balancing the information between an old and a new network.

Motivation & Objective

• Motivate continual learning and amelioration of catastrophic forgetting in deep neural networks.
• Introduce a Bayesian-inspired framework that approximates posteriors with Gaussian mixtures for sequential tasks.
• Propose two moment-matching variants (mean-IMM and mode-IMM) to merge task-specific posteriors.
• Enhance the IMM search space with transfer techniques to produce smooth, convex-like optimization paths.
• Demonstrate empirical gains across diverse datasets (MNIST, CIFAR-10, Caltech-UCSD Birds, Lifelog).

Proposed method

• Model the posterior over network parameters as Gaussians and approximate the mixture of task posteriors with a single Gaussian q(θ|μ,Σ).
• Mean-IMM: minimize the weighted KL divergence sum KL(qk||q1:K) leading to μ* = ∑k αk μk and Σ* = ∑k αk(Σk + (μk−μ*)(μk−μ*)T).
• Mode-IMM: approximate the mixture’s mode using a Laplacian approximation, giving μ* = Σ* (∑k αk Σk−1 μk) and Σ* = (∑k αk Σk−1)−1.
• Apply transfer techniques (weight-transfer, L2-transfer, drop-transfer) to make the optimization path between task posteriors smooth and convex-like.
• Assume diagonal covariance to reduce complexity and use Fisher information to define Σk in mode-IMM.

Experimental results

Research questions

• RQ1 How can posterior moments of sequential task networks be merged to prevent forgetting?
• RQ2 Do mean-IMM and mode-IMM effectively balance performance between old and new tasks across varied datasets?
• RQ3 Can transfer techniques (weight-transfer, L2-transfer, drop-transfer) improve IMM performance by smoothing the loss landscape?
• RQ4 Does a Bayesian-moment-matching view explain and guide continual learning in deep networks?
• RQ5 What are the practical limits of IMM when tasks differ in scale and data distribution?

Key findings

• Mean-IMM and mode-IMM produce competitive to state-of-the-art continual learning performance across multiple benchmarks.
• Drop-transfer and L2-transfer substantially boost IMM performance and improve stability of the trade-off between old and new tasks.
• Mode-IMM demonstrates robustness to transfer techniques, often outperforming mean-IMM particularly when task scales differ.
• IMM can explicitly balance task importance online by adjusting αt, enabling dynamic weighting of old vs. new information.
• In ImageNet-to-CUB transfer, IMM variants achieved modest gains over prior LwF baselines, indicating applicability to heterogeneous task pairs.