[Paper Review] Unifying Regularisation Methods for Continual Learning

Frederik Benzing|arXiv (Cornell University)|Jun 11, 2020

Domain Adaptation and Few-Shot Learning47 references4 citations

TL;DR

This paper unifies three leading continual learning regularization methods—Elastic Weight Consolidation (EWC), Synaptic Intelligence (SI), and Memory Aware Synapses (MAS)—by demonstrating they all approximate the square root of the Fisher Information matrix, a theoretically grounded quantity. Despite different motivations, SI and MAS perform similarly due to a previously unknown bias in SI, and the authors propose Second-Order Synapses (SOS), an improved method that outperforms SI in large-batch training by correcting this bias.

ABSTRACT

Continual Learning addresses the challenge of learning a number of different tasks sequentially. The goal of maintaining knowledge of earlier tasks without re-accessing them starkly conflicts with standard SGD training for artificial neural networks. An influential method to tackle this problem without storing old data are so-called regularisation approaches. They measure the importance of each parameter for solving a given task and subsequently protect important parameters from large changes. In the literature, three ways to measure parameter importance have been put forward and they have inspired a large body of follow-up work. Here, we present strong theoretical and empirical evidence that these three methods, Elastic Weight Consolidation (EWC), Synaptic Intelligence (SI) and Memory Aware Synapses (MAS), are surprisingly similar and are all linked to the same theoretical quantity. Concretely, we show that, despite stemming from very different motivations, both SI and MAS approximate the square root of the Fisher Information, with the Fisher being the theoretically justified basis of EWC. Moreover, we show that for SI the relation to the Fisher -- and in fact its performance -- is due to a previously unknown bias. On top of uncovering unknown similarities and unifying regularisation approaches, we also demonstrate that our insights enable practical performance improvements for large batch training.

Motivation & Objective

• To resolve the long-standing question of how similar the three dominant regularization methods in continual learning—EWC, SI, and MAS—are in practice and theory.
• To provide a theoretical foundation for the effectiveness of SI and MAS, which previously lacked rigorous justification despite their empirical success.
• To identify and correct a previously unknown bias in SI that underlies its performance and drives its similarity to the Fisher Information.
• To develop a new, improved regularization method—Second-Order Synapses (SOS)—based on the corrected understanding of SI’s underlying mechanism.

Proposed method

• Theoretical analysis shows that SI’s importance measure is biased and that this bias causes SI to approximate the square root of the Fisher Information matrix.
• Empirical validation across multiple continual learning benchmarks (e.g., Permuted MNIST, Split CIFAR-10/100) confirms that SI and MAS both closely approximate the square root of the Fisher Information.
• A new algorithm, Second-Order Synapses (SOS), is proposed by correcting the bias in SI’s importance estimation, leading to more accurate parameter importance measurement.
• The method involves reweighting SI’s importance scores using a rescaling step that removes the bias, improving stability and performance in large-batch training.
• Experiments compare SI, SIU (unbiased version), SIB (biased version), and SOS across multiple datasets and training settings, with careful control of hyperparameters, initialization, and training duration.
• Theoretical and empirical analysis confirms that EWC’s use of the diagonal Fisher Information is consistent with the same underlying quantity approximated by SI and MAS.

Experimental results

Research questions

• RQ1 Are the three main regularization methods—EWC, SI, and MAS—more similar in practice than previously assumed?
• RQ2 What is the theoretical basis for the effectiveness of SI and MAS, given their different motivations and formulations?
• RQ3 Does the performance of SI stem from its original motivation or from an underlying bias in its importance estimation?
• RQ4 Can a corrected version of SI lead to improved performance in continual learning, especially under large-batch training?
• RQ5 Is there a unifying theoretical quantity that explains the behavior of EWC, SI, and MAS?

Key findings

• SI’s performance is primarily driven by a previously unknown bias in its importance estimation, not its original motivation of attributing loss reduction to individual parameters.
• Both SI and MAS approximate the square root of the Fisher Information matrix, explaining their empirical similarity despite different derivations.
• The theoretical unification shows that EWC, SI, and MAS are all linked to the same underlying quantity—the Fisher Information—providing a common theoretical foundation for their effectiveness.
• The proposed Second-Order Synapses (SOS) algorithm, which corrects the bias in SI, outperforms SI in large-batch training on standard benchmarks like Split CIFAR-10/100.
• Empirical results show that training duration, initialization (e.g., Glorot vs. PyTorch default), and hyperparameters significantly impact performance, with longer training improving regularisation-based methods substantially.
• On Permuted MNIST, SI’s performance can be improved to 98% accuracy with 200 epochs of training, compared to 90% with only 20 epochs, highlighting the importance of training duration.