[Paper Review] A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks

Eden Belouadah, Adrian Popescu+1 more|arXiv (Cornell University)|Nov 3, 2020

Domain Adaptation and Few-Shot Learning102 references199 citations

TL;DR

The paper analyzes fixed-size/class-incremental learning methods, proposes a unified framework, and shows no single algorithm dominates across settings, with memory and incremental state size greatly influencing performance.

ABSTRACT

The ability of artificial agents to increment their capabilities when confronted with new data is an open challenge in artificial intelligence. The main challenge faced in such cases is catastrophic forgetting, i.e., the tendency of neural networks to underfit past data when new ones are ingested. A first group of approaches tackles forgetting by increasing deep model capacity to accommodate new knowledge. A second type of approaches fix the deep model size and introduce a mechanism whose objective is to ensure a good compromise between stability and plasticity of the model. While the first type of algorithms were compared thoroughly, this is not the case for methods which exploit a fixed size model. Here, we focus on the latter, place them in a common conceptual and experimental framework and propose the following contributions: (1) define six desirable properties of incremental learning algorithms and analyze them according to these properties, (2) introduce a unified formalization of the class-incremental learning problem, (3) propose a common evaluation framework which is more thorough than existing ones in terms of number of datasets, size of datasets, size of bounded memory and number of incremental states, (4) investigate the usefulness of herding for past exemplars selection, (5) provide experimental evidence that it is possible to obtain competitive performance without the use of knowledge distillation to tackle catastrophic forgetting and (6) facilitate reproducibility by integrating all tested methods in a common open-source repository. The main experimental finding is that none of the existing algorithms achieves the best results in all evaluated settings. Important differences arise notably if a bounded memory of past classes is allowed or not.

Motivation & Objective

• Define six desirable properties of incremental learning algorithms and analyze them.
• Provide a unified formalization of class incremental learning.
• Establish a thorough, multi-dataset evaluation framework with varied memory and state settings.
• Investigate the usefulness of herding-based exemplar selection.
• Demonstrate competitive performance without knowledge distillation and enable reproducibility with an open-source repository.

Proposed method

• Introduce six properties (Complexity, Memory, Accuracy, Timeliness, Plasticity, Scalability) to evaluate IL methods.
• Formalize the class incremental learning problem with an initial state and T-1 incremental states.
• Develop a common evaluation framework with multiple datasets, IL state configurations, and memory sizes.
• Experiment with herding-based exemplar selection to assess its impact across methods.
• Show that competitive results are possible without distillation by treating IL as imbalanced learning and using bias correction techniques.
• Integrate tested methods into a common open-source repository with uniform inputs/outputs.

Experimental results

Research questions

• RQ1 How do fixed-size, class-incremental learning algorithms compare across diverse visual tasks and datasets?
• RQ2 What is the impact of bounded memory size on the relative performance of different IL algorithms?
• RQ3 Is exemplar herding beneficial across IL methods under memory constraints?
• RQ4 Can competitive performance be achieved without knowledge distillation, and under what conditions?
• RQ5 How do the six proposed properties relate to practical use-case recommendations for IL in real-world settings?

Key findings

• No single class IL algorithm dominates across all evaluated settings.
• Performance differences are pronounced when memory size or incremental state count changes.
• Memory availability has a stronger impact on relative performance than other IL parameters.
• Herding-based exemplar selection consistently benefits all tested methods.
• Competitive results can be achieved without knowledge distillation by modeling IL as imbalanced learning and applying bias correction techniques.
• The authors provide an open-source repository to facilitate reproducibility across datasets and configurations.