[Paper Review] Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines

Yen-Chang Hsu, Yen‐Cheng Liu+2 more|arXiv (Cornell University)|Oct 30, 2018

Domain Adaptation and Few-Shot LearningComputer Science24 references206 citations

TL;DR

The paper categorizes continual learning scenarios, provides a uniform evaluation framework, and shows simple baselines can rival state-of-the-art methods across tasks; it also releases PyTorch code for fair benchmarking.

ABSTRACT

Continual learning has received a great deal of attention recently with several approaches being proposed. However, evaluations involve a diverse set of scenarios making meaningful comparison difficult. This work provides a systematic categorization of the scenarios and evaluates them within a consistent framework including strong baselines and state-of-the-art methods. The results provide an understanding of the relative difficulty of the scenarios and that simple baselines (Adagrad, L2 regularization, and naive rehearsal strategies) can surprisingly achieve similar performance to current mainstream methods. We conclude with several suggestions for creating harder evaluation scenarios and future research directions. The code is available at https://github.com/GT-RIPL/Continual-Learning-Benchmark

Motivation & Objective

• Provide a systematic categorization of continual learning experimental methodologies.
• Develop a uniform framework to generate and evaluate continual learning scenarios.
• Assess current state-of-the-art methods against strong baselines under consistent tuning budgets.
• Highlight which scenarios are genuinely hard and where simple methods suffice.
• Suggest direction for harder, more realistic continual learning benchmarks.

Proposed method

• Classify continual learning scenarios by differences in input/output distributions and task identity across T1 to T2.
• Propose a uniform, flexible framework to generate task sequences and evaluate methods under consistent settings.
• Compare a wide range of baselines (Adagrad, L2, naive rehearsal) with state-of-the-art continual learning methods using Split MNIST and Permuted MNIST.
• Control total memory overhead across methods to enable fair comparisons.
• Analyze how hyperparameter tuning impacts performance of regularization-based methods versus strong baselines.

Experimental results

Research questions

• RQ1 How do existing continual learning scenarios differ in their task and distribution shifts, and how can they be categorized coherently?
• RQ2 Do simple baselines (e.g., Adagrad, L2, naive rehearsal) match or exceed state-of-the-art continual learning methods under a uniform evaluation framework?
• RQ3 What insights emerge about the relative difficulty of incremental task, domain, and class learning scenarios?
• RQ4 How does the choice of data splitting or permutation affect the perceived difficulty and fairness of comparisons?
• RQ5 What recommendations can improve the realism and challenge of continual learning benchmarks?

Key findings

• Adagrad and L2 regularization often outperform online EWC and are on par with SI across scenarios.
• Naive rehearsal achieves comparable or superior performance to many state-of-the-art methods when memory overhead is matched.
• Incremental task learning is the easiest scenario; incremental class learning is harder than incremental domain learning, with permutation-based setups generally easier than splitting-based ones.
• Regularization-based methods require substantial hyperparameter tuning, whereas simple baselines can perform well without tuning, raising concerns about their real-world applicability.
• Permuted MNIST scenarios tend to be easier than Split MNIST scenarios for many methods, suggesting the need for harder benchmarks.