[Paper Review] Continual Learning in Low-rank Orthogonal Subspaces

Arslan Chaudhry, Naeemullah Khan+2 more|arXiv (Cornell University)|Oct 22, 2020

Domain Adaptation and Few-Shot LearningComputer Science54 references30 citations

TL;DR

The paper proposes orthogonal subspaces in a neural network for continual learning, enforcing orthonormal weight matrices via Stiefel-manifold optimization to minimize interference across tasks and achieve strong performance against memory-based baselines.

ABSTRACT

In continual learning (CL), a learner is faced with a sequence of tasks, arriving one after the other, and the goal is to remember all the tasks once the continual learning experience is finished. The prior art in CL uses episodic memory, parameter regularization or extensible network structures to reduce interference among tasks, but in the end, all the approaches learn different tasks in a joint vector space. We believe this invariably leads to interference among different tasks. We propose to learn tasks in different (low-rank) vector subspaces that are kept orthogonal to each other in order to minimize interference. Further, to keep the gradients of different tasks coming from these subspaces orthogonal to each other, we learn isometric mappings by posing network training as an optimization problem over the Stiefel manifold. To the best of our understanding, we report, for the first time, strong results over experience-replay baseline with and without memory on standard classification benchmarks in continual learning. The code is made publicly available.

Motivation & Objective

• Motivate reducing interference in continual learning by learning tasks in orthogonal subspaces rather than a shared vector space.
• Introduce a projection scheme to assign each task to a distinct low-rank subspace with orthogonality guarantees.
• Enforce isometric gradient transformations by optimizing weights on the Stiefel manifold to preserve orthogonality across layers.
• Demonstrate empirical gains over strong experience-replay baselines on MNIST, CIFAR, and ImageNet-derived tasks.

Proposed method

• Assign each task a low-rank subspace via a rank-r projection P_t in R^{m×m}, with P_t^T P_t = I and P_t^T P_k = 0 for k ≠ t.
• Convince gradients from different tasks are orthogonal at the projection layer, and preserve this via isometric transformations by updating weights on the Stiefel manifold (W_l^T W_l = I).
• Optimize over the Stiefel manifold using tangent-space projection and Cayley-transform-based retraction to keep weight matrices orthonormal during training.
• Maintain a tiny replay buffer for past tasks and compute gradients for current and past tasks to form an effective gradient g_l = g_l^t + g_l^k.
• Initialize networks with random orthonormal weights and perform offline construction of projection bases O_t to build P_t = O_t O_t^T.
• Provide algorithmic details (Alg. 1) for training orthog-subspace with adaptive learning via the Cayley transform.

Experimental results

Research questions

• RQ1 Can learning each task in an independent low-rank subspace reduce interference and forgetting in continual learning?
• RQ2 Does enforcing orthogonality of weight matrices via Stiefel-manifold optimization preserve gradient orthogonality across layers and improve performance with limited episodic memory?
• RQ3 How does orthog-subspace perform relative to strong memory-based baselines on standard continual learning benchmarks?
• RQ4 Is the projection-and-orthogonality approach more beneficial in shallow versus deep networks?

Key findings

• Orthog-subspace improves over strong experience-replay baselines on deeper networks by up to 4.7 percentage points in accuracy and up to 50% reduction in forgetting on CIFAR-100 and miniImageNet when memory is used.
• On shallower networks (Permuted/Rotated MNIST), orthog-subspace achieves 7.1–9.2 percentage point gains in average accuracy and 42–66% forgetting reduction without memory.
• Systematic ablation shows: projection alone is weaker; adding memory helps; enforcing Stiefel-manifold orthogonality yields the best accuracy and lowest forgetting.
• Gradient analysis confirms that with Stiefel constraints, gradient inner products across tasks concentrate near zero, indicating reduced interference.
• With tiny episodic memory, orthog-subspace surpasses ER-Ring on several benchmarks, and gains shrink as memory size increases.