[Paper Review] Orthogonal Gradient Descent for Continual Learning

Mehrdad Farajtabar, Navid Azizan+2 more|arXiv (Cornell University)|Oct 15, 2019

Domain Adaptation and Few-Shot Learning79 references39 citations

TL;DR

The paper introduces Orthogonal Gradient Descent (OGD), which mitigates catastrophic forgetting by projecting gradients from new tasks onto a subspace orthogonal to previous task gradient directions, enabling learning of new tasks without storing past data.

ABSTRACT

Neural networks are achieving state of the art and sometimes super-human performance on learning tasks across a variety of domains. Whenever these problems require learning in a continual or sequential manner, however, neural networks suffer from the problem of catastrophic forgetting; they forget how to solve previous tasks after being trained on a new task, despite having the essential capacity to solve both tasks if they were trained on both simultaneously. In this paper, we propose to address this issue from a parameter space perspective and study an approach to restrict the direction of the gradient updates to avoid forgetting previously-learned data. We present the Orthogonal Gradient Descent (OGD) method, which accomplishes this goal by projecting the gradients from new tasks onto a subspace in which the neural network output on previous task does not change and the projected gradient is still in a useful direction for learning the new task. Our approach utilizes the high capacity of a neural network more efficiently and does not require storing the previously learned data that might raise privacy concerns. Experiments on common benchmarks reveal the effectiveness of the proposed OGD method.

Motivation & Objective

• Address catastrophic forgetting in sequential task learning.
• Utilize gradient directions of previous tasks to constrain updates without storing raw data.
• Leverage the high capacity of neural networks to learn new tasks while preserving prior knowledge.
• Propose practical OGD variants that reduce memory needs while maintaining effectiveness.

Proposed method

• Compute the gradient directions {∇f_j(x; w)} for previous task predictions.
• Construct an orthogonal basis S from previous task gradients (per data point and class) using Gram-Schmidt.
• Modify the current task gradient g to tilde{g} = g - sum_{v in S} proj_v(g), ensuring updates are orthogonal to previous directions.
• Apply backpropagation using the orthogonalized gradient tilde{g}.
• Store a subset of gradients from each previous task (OGD-GTL variant) to bound memory usage.
• Describe two memory-reduction variants: OGD-AVE (average logits) and OGD-GTL (ground-truth logit).
• Provide Algorithm 1 for Orthogonal Gradients Descent including: gradient computation, projection, parameter update, and construction of the gradient subspace.

Experimental results

Research questions

• RQ1 Can gradient updates for a new task be constrained to minimize interference with prior tasks without storing past data?
• RQ2 How does projecting onto the orthogonal space of previous task gradients affect learning efficiency and final accuracy across multiple MNIST-based continual learning benchmarks?
• RQ3 Do memory-reduced variants (OGD-GTL, OGD-AVE) retain performance while lowering storage requirements?

Key findings

• OGD achieves competitive or superior performance compared with state-of-the-art continual learning baselines on Permuted MNIST, Rotated MNIST, and Split MNIST.
• OGD retains task-specific performance across multiple sequential tasks, approaching multi-task learning when feasible and outperforming SGD and EWC baselines in several settings.
• OGD variants that reduce memory, such as OGD-GTL, generally perform well and can outperform or match A-GEM in many experiments.
• The method relies on gradients of the model outputs (logits) rather than the loss, providing a direct means to preserve previously learned predictions.