[Paper Review] Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference

Matthew Riemer, Ignacio Cases+5 more|arXiv (Cornell University)|Oct 28, 2018

Domain Adaptation and Few-Shot LearningComputer Science345 citations

TL;DR

The paper introduces Meta-Experience Replay (MER), a method that merges experience replay with optimization-based meta-learning to maximize forward transfer and minimize interference in continual learning across supervised and reinforcement learning, without requiring task labels.

ABSTRACT

Lack of performance when it comes to continual learning over non-stationary distributions of data remains a major challenge in scaling neural network learning to more human realistic settings. In this work we propose a new conceptualization of the continual learning problem in terms of a temporally symmetric trade-off between transfer and interference that can be optimized by enforcing gradient alignment across examples. We then propose a new algorithm, Meta-Experience Replay (MER), that directly exploits this view by combining experience replay with optimization based meta-learning. This method learns parameters that make interference based on future gradients less likely and transfer based on future gradients more likely. We conduct experiments across continual lifelong supervised learning benchmarks and non-stationary reinforcement learning environments demonstrating that our approach consistently outperforms recently proposed baselines for continual learning. Our experiments show that the gap between the performance of MER and baseline algorithms grows both as the environment gets more non-stationary and as the fraction of the total experiences stored gets smaller.

Motivation & Objective

• Motivate a temporally symmetric transfer–interference view of continual learning across forward and backward time directions.
• Develop a meta-learning-based algorithm that learns to adjust gradient dynamics to favor transfer and reduce interference.
• Leverage experience replay to approximate stationary data distributions in non-stationary streams.
• Evaluate MER on diverse continual learning benchmarks and non-stationary RL environments showing robust performance gains.

Proposed method

• Define transfer and interference in terms of gradient alignment between pairs of examples.
• Propose an objective that encourages high gradient dot products between random datapoints to promote shared useful representations (Equation 4).
• Integrate experience replay with optimization-based meta-learning to create MER (Algorithm 1) that optimizes a Reptile-style objective over memory samples.
• Use reservoir sampling to maintain a memory buffer that approximates the stationary distribution of seen data (Appendix F).
• Adopt first-order meta-learning (Reptile) to avoid second-derivative computations and enable online continual learning (Equations 6–7).

Experimental results

Research questions

• RQ1 Can a temporally symmetric transfer–interference framework improve continual learning over non-stationary distributions?
• RQ2 Does Meta-Experience Replay (MER) effectively maximize forward transfer while minimizing interference across past and future data?
• RQ3 How does MER compare to existing baselines (EWC, GEM, Online, etc.) on supervised continual lifelong learning benchmarks?
• RQ4 Is MER robust to limited memory (small buffers) and increasingly non-stationary reinforcement learning environments?

Key findings

• MER consistently outperforms strong baselines (e.g., GEM, EWC, Online) on supervised continual lifelong learning benchmarks MNIST Rotations and MNIST Permutations.
• MER yields higher retained accuracy and better balance between transfer and interference, especially as non-stationarity increases.
• MER shows stronger gains with smaller memory buffers, outperforming GEM even when GEM uses substantially larger buffers.
• In increasingly non-stationary settings (Many Permutations, Omniglot), MER substantially improves retention and learning speed compared to baselines.
• In continual reinforcement learning tests with DQN on non-stationary Catcher and Flappy Bird, MER reduces forgetting and improves cross-task performance compared to standard DQN with experience replay.