[Paper Review] Recent Advances of Continual Learning in Computer Vision: An Overview

Haoxuan Qu, Hossein Rahmani+3 more|arXiv (Cornell University)|Sep 23, 2021

Domain Adaptation and Few-Shot LearningComputer Science153 references41 citations

TL;DR

This paper surveys recent continual learning methods in computer vision, organizing them by technique (regularization, distillation, memory, generative replay, and parameter isolation) and outlining applications beyond image classification.

ABSTRACT

In contrast to batch learning where all training data is available at once, continual learning represents a family of methods that accumulate knowledge and learn continuously with data available in sequential order. Similar to the human learning process with the ability of learning, fusing, and accumulating new knowledge coming at different time steps, continual learning is considered to have high practical significance. Hence, continual learning has been studied in various artificial intelligence tasks. In this paper, we present a comprehensive review of the recent progress of continual learning in computer vision. In particular, the works are grouped by their representative techniques, including regularization, knowledge distillation, memory, generative replay, parameter isolation, and a combination of the above techniques. For each category of these techniques, both its characteristics and applications in computer vision are presented. At the end of this overview, several subareas, where continuous knowledge accumulation is potentially helpful while continual learning has not been well studied, are discussed.

Motivation & Objective

• Provide a systematic review of recent progress in continual learning (CL) within computer vision.
• Summarize how CL techniques are applied across CV subareas beyond image classification (e.g., segmentation, generation).
• Highlight main categories of CL methods and their characteristics, advantages, and limitations.
• Identify potential subareas in CV where continual knowledge accumulation could be further explored.

Proposed method

• Group and describe CL techniques into regularization-based, knowledge distillation-based, memory-based, generative replay-based, and parameter isolation-based methods.
• Summarize key variants within each category (e.g., EWC, SI, MAS under regularization; LwF, and distillation variants under knowledge distillation).
• Discuss evaluation metrics for CL (average accuracy, forgetting, intransigence, backward/forward transfer).
• Describe applications of CL to diverse CV tasks: image classification, semantic segmentation, image generation, object detection, captioning, and more.

Experimental results

Research questions

• RQ1 What are the main continual learning techniques used in computer vision and how do they differ in addressing catastrophic forgetting?
• RQ2 How do continual learning methods perform across different CV tasks beyond image classification?
• RQ3 What evaluation metrics best characterize CL performance and forgetting in CV?
• RQ4 What subareas of computer vision could benefit from continual knowledge accumulation but are underexplored?
• RQ5 How do combinations of CL techniques (e.g., regularization with memory or distillation) improve performance?

Key findings

• The paper categorizes CL methods into regularization, knowledge distillation, memory-based, generative replay, and parameter isolation.
• It notes that CL has expanded beyond image classification to semantic segmentation, image generation, object detection, and more.
• It discusses multiple variants and improvements to foundational methods (e.g., EWC, SI, MAS, LwF) and highlights gradient/learning-rate regularization as alternative strategies.
• It provides evaluation metrics such as average accuracy, forgetting, intransigence, and forward/backward transfer for assessing CL performance.
• The overview identifies underexplored CV subareas where continual learning could be beneficial but is not yet well studied.