[Paper Review] Online Continual Learning with Maximally Interfered Retrieval

Rahaf Aljundi, Lucas Caccia+5 more|arXiv (Cornell University)|Aug 11, 2019

Domain Adaptation and Few-Shot Learning38 references75 citations

TL;DR

The paper introduces Maximally Interfered Retrieval (MIR) for replay-based online continual learning, selecting replay samples that are most harmed by the upcoming parameter update to reduce forgetting.

ABSTRACT

Continual learning, the setting where a learning agent is faced with a never ending stream of data, continues to be a great challenge for modern machine learning systems. In particular the online or "single-pass through the data" setting has gained attention recently as a natural setting that is difficult to tackle. Methods based on replay, either generative or from a stored memory, have been shown to be effective approaches for continual learning, matching or exceeding the state of the art in a number of standard benchmarks. These approaches typically rely on randomly selecting samples from the replay memory or from a generative model, which is suboptimal. In this work, we consider a controlled sampling of memories for replay. We retrieve the samples which are most interfered, i.e. whose prediction will be most negatively impacted by the foreseen parameters update. We show a formulation for this sampling criterion in both the generative replay and the experience replay setting, producing consistent gains in performance and greatly reduced forgetting. We release an implementation of our method at https://github.com/optimass/Maximally_Interfered_Retrieval.

Motivation & Objective

• Address catastrophic forgetting in online, single-pass continual learning.
• Improve replay quality by selecting memories most affected by the next update (maximal interference).
• Demonstrate MIR benefits in both experience replay and generative replay settings.
• Explore a hybrid approach using autoencoders to compress memories for scalable MIR retrieval.

Proposed method

• Define a memory-based rehearsal with MIR: retrieve top-k memories from a buffer whose loss increases the most after applying the estimated parameter update from the incoming batch (MI-1/MI-2 criteria).
• Extend MIR to generative replay by searching in the latent space Z to maximize the difference in loss before/after the estimated update, using a regularized objective (Eq. 1 and Eq. 2).
• Use a simple buffering strategy with reservoir sampling and a diversity-promoting preselection (C samples before selecting B) to manage compute.
• For generative MIR, estimate the true label y* via predictions from a prior/current model and use KL divergence as an interference proxy; optionally apply entropy regularization to encourage confident generation (Eq. 2).
• Present a hybrid approach: compress memories with an autoencoder to enable larger buffers and perform MIR in latent space, followed by nearest-neighbor Reconstruction to real samples.

Experimental results

Research questions

• RQ1 Can learning samples that are maximally interfered by the upcoming update reduce forgetting more effectively than random replay?
• RQ2 Do MIR-based replay strategies improve online continual learning performance in both experience replay and generative replay settings?
• RQ3 Does a hybrid autoencoder-based MIR enable scalable memory usage without sacrificing performance?

Key findings

• ER-MIR improves accuracy and reduces forgetting compared to standard ER on MNIST splits and Permuted MNIST.
• In CIFAR-10 with larger memory sizes, ER-MIR shows better accuracy and forgetting than ER and some baselines like GEM and iCarl under online constraints.
• GEN-MIR improves generator loss and accuracy over GEN (random/replay baselines) on MNIST Split and Permuted MNIST, with the entropy-regularization term playing a key role.
• The hybrid AE-MIR approach, with latent-space MIR and real-reconstruction alignment, yields better forgetting trade-offs and benefits from larger compressed memories.
• Increasing the number of updates per incoming batch generally improves performance for MIR-based methods across datasets.