[Paper Review] Understanding the Role of Training Regimes in Continual Learning

Seyed Iman Mirzadeh, Mehrdad Farajtabar+2 more|arXiv (Cornell University)|Jun 12, 2020

Domain Adaptation and Few-Shot LearningComputer Science38 references91 citations

TL;DR

The paper analyzes how simple training regimes (dropout, learning rate decay, small batch size) shape the loss landscape to reduce catastrophic forgetting in continual learning, proposing that wider minima lead to better stability.

ABSTRACT

Catastrophic forgetting affects the training of neural networks, limiting their ability to learn multiple tasks sequentially. From the perspective of the well established plasticity-stability dilemma, neural networks tend to be overly plastic, lacking the stability necessary to prevent the forgetting of previous knowledge, which means that as learning progresses, networks tend to forget previously seen tasks. This phenomenon coined in the continual learning literature, has attracted much attention lately, and several families of approaches have been proposed with different degrees of success. However, there has been limited prior work extensively analyzing the impact that different training regimes -- learning rate, batch size, regularization method-- can have on forgetting. In this work, we depart from the typical approach of altering the learning algorithm to improve stability. Instead, we hypothesize that the geometrical properties of the local minima found for each task play an important role in the overall degree of forgetting. In particular, we study the effect of dropout, learning rate decay, and batch size, on forming training regimes that widen the tasks' local minima and consequently, on helping it not to forget catastrophically. Our study provides practical insights to improve stability via simple yet effective techniques that outperform alternative baselines.

Motivation & Objective

• Investigate catastrophic forgetting in sequential task learning without access to old task data.
• Link forgetting to the geometry of local minima via loss landscape analysis.
• Evaluate common training regimes (dropout, LR decay, batch size) as practical tools to widen minima and improve stability.
• Compare simple training regimes against more complex continual learning methods across standard benchmarks.

Proposed method

• Model forgetting using a two-task loss and a second-order Taylor expansion around task minima to relate forgetting to the Hessian.
• Use the bound F1 ≈ (1/2) Δw^T Hessian(w1*) Δw to connect forgetting to the curvature and parameter displacement.
• Empirically estimate Hessian width via largest Hessian eigenvalue λ1^max to assess its relation to forgetting.
• Experiment with dropout, learning rate schedules, and small batch sizes to create training regimes that widen minima and reduce Δw.
• Evaluate training regimes on Rotated MNIST and Permuted MNIST to illustrate the wide/narrow minima intuition.
• Compare stable SGD against plastic (naive) SGD and state-of-the-art continual learning methods on MF benchmarks.

Experimental results

Research questions

• RQ1 How does the curvature of the loss around task minima relate to forgetting in sequential task learning?
• RQ2 Do training regime choices that widen minima (e.g., dropout, high initial LR with decay, small batch sizes) reduce forgetting?
• RQ3 Can simple optimization techniques outperform complex continual learning methods on standard benchmarks?
• RQ4 What is the empirical relationship between Hessian width and forgetting across tasks?

Key findings

• Wider minima (smaller Hessian eigenvalues) correlate with reduced forgetting across Rotated MNIST and Permuted MNIST.
• Dropout, large initial learning rate with decay, and small batch sizes induce wider minima and smaller parameter displacement between tasks.
• Stable SGD reduces forgetting more than plastic SGD and can outperform some regularization based and memory-based continual learning methods on standard benchmarks.
• The forgetting bound tightness depends on the direction of Δw with respect to the Hessian spectrum, highlighting the role of landscape geometry in forgetting.