[Paper Review] Generative replay with feedback connections as a general strategy for continual learning

Gido M. van de Ven, Andreas S. Tolias|Lirias (KU Leuven)|Sep 27, 2018

Domain Adaptation and Few-Shot Learning37 references184 citations

TL;DR

The paper identifies three continual learning scenarios and shows that generative replay with distillation (and its cost-reducing Replay-through-Feedback variant) outperforms regularization methods across scenarios, especially when task identity must be inferred.

ABSTRACT

A major obstacle to developing artificial intelligence applications capable of true lifelong learning is that artificial neural networks quickly or catastrophically forget previously learned tasks when trained on a new one. Numerous methods for alleviating catastrophic forgetting are currently being proposed, but differences in evaluation protocols make it difficult to directly compare their performance. To enable more meaningful comparisons, here we identified three distinct scenarios for continual learning based on whether task identity is known and, if it is not, whether it needs to be inferred. Performing the split and permuted MNIST task protocols according to each of these scenarios, we found that regularization-based approaches (e.g., elastic weight consolidation) failed when task identity needed to be inferred. In contrast, generative replay combined with distillation (i.e., using class probabilities as "soft targets") achieved superior performance in all three scenarios. Addressing the issue of efficiency, we reduced the computational cost of generative replay by integrating the generative model into the main model by equipping it with generative feedback or backward connections. This Replay-through-Feedback approach substantially shortened training time with no or negligible loss in performance. We believe this to be an important first step towards making the powerful technique of generative replay scalable to real-world continual learning applications.

Motivation & Objective

• Identify and formalize three distinct continual learning scenarios based on task identity availability and inference requirements.
• Compare regularization-based and replay-based continual learning methods across these scenarios.
• Propose and evaluate a cost-efficient variant (Replay-through-Feedback) that integrates generation into the main model without a separate generator.

Proposed method

• Define Task-IL, Domain-IL, and Class-IL scenarios based on task identity availability at test time.
• Evaluate regularization methods (EWC, Online EWC, SI) and replay methods (LwF, DGR, DGR+distill) on split MNIST and permuted MNIST.
• Compare against offline joint training as an upper bound.
• Introduce Replay-through-Feedback (RtF) by embedding a generative model into the main network with feedback connections and a latent z layer.
• Use distillation targets (soft targets) with replay data; weight current and replay losses by the number of tasks seen so far.
• Demonstrate RtF and DGR+distill with equivalent architectures, and measure training time vs. performance.

Experimental results

Research questions

• RQ1 How do different continual learning strategies perform across Task-IL, Domain-IL, and Class-IL scenarios?
• RQ2 Does generative replay with distillation consistently outperform regularization across these scenarios?
• RQ3 Can a unified, cost-efficient architecture (RtF) match or surpass standard generative replay performance while reducing training time?

Key findings

• Regularization methods (EWC, Online EWC, SI) struggle when task identity must be inferred (Class-IL).
• Replay-based methods with generative replay (LwF, DGR, DGR+distill) outperform regularizers across all three scenarios, with DGR+distill generally superior to DGR.
• In both split MNIST and permuted MNIST, RtF often matches or exceeds DGR+distill while substantially reducing training time (roughly halved in many cases).
• Across tasks, Class-IL remains the most challenging; only replay-based methods maintain performance when task identity must be inferred.
• Distillation (soft targets) improves robustness of generative replay to replay sample quality, contributing to superior performance of DGR+distill over DGR.