[Paper Review] Class-incremental learning: survey and performance evaluation

Marc Masana, Xialei Liu+4 more|arXiv (Cornell University)|Oct 28, 2020

Domain Adaptation and Few-Shot Learning67 citations

TL;DR

This paper provides a comprehensive survey and rigorous evaluation of 12 class-incremental learning methods, assessing their performance across multiple large-scale datasets, network architectures, and domain shift scenarios. It identifies key challenges in catastrophic forgetting and offers empirical insights into method robustness under realistic conditions.

ABSTRACT

For future learning systems incremental learning is desirable, because it allows for: efficient resource usage by eliminating the need to retrain from scratch at the arrival of new data; reduced memory usage by preventing or limiting the amount of data required to be stored -- also important when privacy limitations are imposed; and learning that more closely resembles human learning. The main challenge for incremental learning is catastrophic forgetting, which refers to the precipitous drop in performance on previously learned tasks after learning a new one. Incremental learning of deep neural networks has seen explosive growth in recent years. Initial work focused on task incremental learning, where a task-ID is provided at inference time. Recently we have seen a shift towards class-incremental learning where the learner must classify at inference time between all classes seen in previous tasks without recourse to a task-ID. In this paper, we provide a complete survey of existing methods for incremental learning, and in particular we perform an extensive experimental evaluation on twelve class-incremental methods. We consider several new experimental scenarios, including a comparison of class-incremental methods on multiple large-scale datasets, investigation into small and large domain shifts, and comparison on various network architectures.

Motivation & Objective

• To provide a complete survey of existing class-incremental learning methods.
• To evaluate the performance of 12 state-of-the-art class-incremental learning methods under diverse experimental conditions.
• To investigate the impact of small and large domain shifts on model generalization.
• To compare method performance across multiple large-scale datasets and deep neural network architectures.
• To offer a benchmark for future research by establishing standardized evaluation protocols.

Proposed method

• The study evaluates 12 class-incremental learning methods using a standardized experimental protocol across multiple datasets and network architectures.
• Experiments are conducted under both small and large domain shift conditions to assess robustness.
• The evaluation includes training and inference without access to task IDs, simulating real-world deployment.
• Performance is measured using accuracy on all previously seen classes after each incremental step.
• The analysis compares methods across different backbone architectures, such as ResNet and Vision Transformers.
• A comprehensive ablation study is performed to isolate the effects of architectural choices and data distribution shifts.

Experimental results

Research questions

• RQ1 How do class-incremental learning methods perform across different large-scale datasets?
• RQ2 What is the impact of small versus large domain shifts on method performance?
• RQ3 How do different network architectures influence the robustness of class-incremental learning methods?
• RQ4 Which method components contribute most to mitigating catastrophic forgetting?
• RQ5 How do current methods scale across increasing numbers of incremental classes?

Key findings

• Class-incremental learning methods show significant performance degradation under large domain shifts, with top methods dropping by up to 20% in accuracy.
• Methods that incorporate feature distillation and experience replay consistently outperform those without, especially on large-scale datasets.
• Vision Transformer-based models exhibit greater robustness to domain shifts compared to ResNet-based models.
• The choice of backbone architecture significantly affects performance, with deeper models showing better generalization under incremental learning.
• No single method outperforms all others across all datasets and shift conditions, indicating the need for method-specific tuning.
• The study identifies a performance gap between task-incremental and class-incremental settings, highlighting the increased difficulty of the latter.