[Paper Review] Does Continual Learning = Catastrophic Forgetting?

Anh Thai, Stefan Stojanov+2 more|arXiv (Cornell University)|Jan 18, 2021

Domain Adaptation and Few-Shot Learning67 references4 citations

TL;DR

This paper challenges the assumption that continual learning inherently causes catastrophic forgetting by demonstrating that certain tasks do not suffer from forgetting when learned incrementally. It introduces YASS, a simple yet effective algorithm that achieves state-of-the-art performance in class-incremental learning, offering insights into the benefits of continuous representation learning and converging evidence on forgetting dynamics.

ABSTRACT

Continual learning is known for suffering from catastrophic forgetting, a phenomenon where earlier learned concepts are forgotten at the expense of more recent samples. In this work, we challenge the assumption that continual learning is inevitably associated with catastrophic forgetting by presenting a set of tasks that surprisingly do not suffer from catastrophic forgetting when learned continually. The robustness of these tasks leads to the potential of having a proxy representation learning task for continual classification. We further introduce a novel yet simple algorithm, YASS that achieves state-of-the-art performance in the class-incremental categorization learning task and provide an insight into the benefit of learning the representation continuously. Finally, we present converging evidence on the forgetting dynamics of representation learning in continual models. The codebase, dataset, and pre-trained models released with this article can be found at this https URL.

Motivation & Objective

• To challenge the widely held belief that continual learning inevitably leads to catastrophic forgetting.
• To identify and analyze tasks that do not suffer from catastrophic forgetting during continual learning.
• To develop a novel algorithm, YASS, that achieves state-of-the-art performance in class-incremental learning.
• To provide insights into the benefits of continuous representation learning in continual learning scenarios.
• To offer converging evidence on the dynamics of forgetting in continual representation learning models.

Proposed method

• The authors design a set of tasks that exhibit robustness to catastrophic forgetting during continual learning, serving as a proxy for representation learning.
• They introduce YASS, a simple algorithm that enhances representation learning by continuously updating features during incremental task learning.
• YASS leverages a memory replay mechanism and feature distillation to preserve knowledge from previous tasks.
• The method is evaluated on class-incremental categorization benchmarks, demonstrating superior performance compared to prior approaches.
• The authors analyze forgetting dynamics using convergence analysis and visualization of representation changes over time.
• A public codebase, dataset, and pre-trained models are released to support reproducibility and further research.

Experimental results

Research questions

• RQ1 Can continual learning be achieved without catastrophic forgetting in certain task settings?
• RQ2 What characteristics of tasks make them resistant to forgetting during continual learning?
• RQ3 How does continuous representation learning influence model performance in incremental learning?
• RQ4 What are the dynamics of forgetting in continual representation learning models?
• RQ5 Can a simple algorithm like YASS achieve state-of-the-art performance in class-incremental learning?

Key findings

• The authors identify specific tasks that do not suffer from catastrophic forgetting when learned continually, challenging the assumption that forgetting is inevitable.
• YASS achieves state-of-the-art performance in class-incremental categorization learning, outperforming existing methods.
• The results demonstrate that continuous representation learning significantly improves model robustness and generalization in incremental settings.
• Converging evidence from multiple analyses reveals consistent patterns in the forgetting dynamics of continual learning models.
• The released codebase, dataset, and pre-trained models provide a valuable resource for future research in continual learning.