[Paper Review] Lifelong Learning with Sketched Structural Regularization

Haoran Li, Aditya Krishnan+4 more|arXiv (Cornell University)|Apr 17, 2021

Domain Adaptation and Few-Shot Learning38 references4 citations

TL;DR

This paper proposes Sketched Structural Regularization (Sketched SR), a computationally efficient method that uses linear sketching to better approximate the importance matrix in structural regularization for continual learning. By capturing off-diagonal parameter interactions often missed by diagonal approximations, Sketched SR significantly reduces catastrophic forgetting and outperforms diagonal SR across Permuted-MNIST and CIFAR-100 benchmarks with minimal computational overhead.

ABSTRACT

Preventing catastrophic forgetting while continually learning new tasks is an essential problem in lifelong learning. Structural regularization (SR) refers to a family of algorithms that mitigate catastrophic forgetting by penalizing the network for changing its "critical parameters" from previous tasks while learning a new one. The penalty is often induced via a quadratic regularizer defined by an \emph{importance matrix}, e.g., the (empirical) Fisher information matrix in the Elastic Weight Consolidation framework. In practice and due to computational constraints, most SR methods crudely approximate the importance matrix by its diagonal. In this paper, we propose \emph{Sketched Structural Regularization} (Sketched SR) as an alternative approach to compress the importance matrices used for regularizing in SR methods. Specifically, we apply \emph{linear sketching methods} to better approximate the importance matrices in SR algorithms. We show that sketched SR: (i) is computationally efficient and straightforward to implement, (ii) provides an approximation error that is justified in theory, and (iii) is method oblivious by construction and can be adapted to any method that belongs to the structural regularization class. We show that our proposed approach consistently improves various SR algorithms' performance on both synthetic experiments and benchmark continual learning tasks, including permuted-MNIST and CIFAR-100.

Motivation & Objective

• To address catastrophic forgetting in lifelong learning by improving the approximation of importance matrices used in structural regularization.
• To overcome the limitations of diagonal approximations, which assume parameter independence and often fail to capture critical parameter interactions.
• To develop a method-agnostic, computationally efficient framework that enhances existing structural regularization algorithms without requiring architectural changes.
• To empirically validate that better importance matrix approximation leads to improved continual learning performance on standard benchmarks.

Proposed method

• The method applies linear sketching techniques to compress and approximate the full importance matrix (e.g., empirical Fisher information matrix) used in structural regularization.
• It constructs a low-dimensional sketch of the importance matrix using random projections, preserving key structural properties with provable approximation guarantees.
• The sketched matrix is used to define a quadratic regularizer during training on new tasks, penalizing changes to critical parameters from previous tasks.
• The approach is method-agnostic and can be integrated into any structural regularization algorithm, such as EWC or MAS, without modifying the core training procedure.
• The sketch size $ t $ controls the trade-off between approximation accuracy and computational cost, with small $ t $ values (e.g., 30–50) already yielding strong performance.
• Theoretical analysis shows that the sketching process maintains a bounded approximation error relative to the full importance matrix.

Experimental results

Research questions

• RQ1 Can linear sketching provide a better approximation of the importance matrix than diagonal approximation in structural regularization?
• RQ2 Does using a sketched importance matrix lead to improved continual learning performance on standard benchmarks like Permuted-MNIST and CIFAR-100?
• RQ3 How does the sketch size $ t $ affect the trade-off between approximation quality and computational efficiency?
• RQ4 Is the sketched SR framework compatible with a wide range of structural regularization methods?

Key findings

• Sketched SR significantly outperforms diagonal SR on Permuted-MNIST, achieving 89.8% average accuracy with EWC (vs. 88.3% for diagonal) and 90.4% with MAS (vs. 86.7% for diagonal).
• On CIFAR-100, sketched SR achieves 93.6% average accuracy with EWC (vs. 90.8% for diagonal) and 93.2% with MAS (vs. 89.9% for diagonal).
• Even with a small sketch size of $ t = 50 $, sketched SR consistently outperforms diagonal SR, demonstrating practical efficiency and effectiveness.
• The approximation error of sketched SR is only 8.1% on average (vs. 94.7% for diagonal) when compared to the full empirical Fisher matrix in a synthetic task.
• The method is robust across different architectures and datasets, showing consistent gains in both task 1 and task 2 performance on CIFAR-100.
• The framework is computationally efficient and scalable, avoiding the high cost of low-rank approximations like PCA while achieving superior approximation quality.