[Paper Review] Don't forget, there is more than forgetting: new metrics for Continual Learning

Natalia Díaz-Rodríguez, Vincenzo Lomonaco+2 more|arXiv (Cornell University)|Oct 31, 2018

Domain Adaptation and Few-Shot Learning11 references74 citations

TL;DR

The paper proposes a comprehensive, implementation-independent set of continual learning metrics that cover accuracy over time, backward/forward transfer, memory, and computation, and combines them into a flexible CL_score for ranking algorithms.

ABSTRACT

Continual learning consists of algorithms that learn from a stream of data/tasks continuously and adaptively thought time, enabling the incremental development of ever more complex knowledge and skills. The lack of consensus in evaluating continual learning algorithms and the almost exclusive focus on forgetting motivate us to propose a more comprehensive set of implementation independent metrics accounting for several factors we believe have practical implications worth considering in the deployment of real AI systems that learn continually: accuracy or performance over time, backward and forward knowledge transfer, memory overhead as well as computational efficiency. Drawing inspiration from the standard Multi-Attribute Value Theory (MAVT) we further propose to fuse these metrics into a single score for ranking purposes and we evaluate our proposal with five continual learning strategies on the iCIFAR-100 continual learning benchmark.

Motivation & Objective

• Motivate broader evaluation beyond forgetting for continual learning (CL) systems.
• Define a comprehensive, implementation-independent metric suite spanning accuracy over time, transfer, memory, and computation.
• Propose a multi-criteria aggregation (CL_score) and assess its utility for ranking CL strategies.
• Evaluate five CL strategies on the iCIFAR-100 benchmark to illustrate the proposed metrics.

Proposed method

• Map each criterion to a [0,1] scale using MAVT-inspired normalization to enable aggregation.
• Define Accuracy, Backward Transfer (BWT) with forgetting and remembering components, and Forward Transfer (FWT) as time-aware metrics.
• Introduce Model Size (MS), Samples Storage Size (SSS), and Computational Efficiency (CE) as efficiency metrics.
• Compute a final CL_score as a weighted sum of criteria with user-defined weights and normalize for interpretability.
• Provide a stability measure (CL_stability) based on the standard deviation of criterion scores across runs.
• Evaluate on iCIFAR-100 with five CL strategies (Naïve, Cumulative, EWC, LwF, SI) using a CNN baseline and report metrics per strategy.

Experimental results

Research questions

• RQ1 How do the proposed metrics capture dynamic performance of CL methods over time.
• RQ2 How do different CL strategies compare when evaluated with the full set of proposed metrics?
• RQ3 How does the aggregated CL_score reflect trade-offs between accuracy, transfer, memory, and computation?
• RQ4 What is the variability of these metrics across multiple runs (stability) for different strategies?

Key findings

• A comprehensive metric set is proposed that includes Accuracy, REM (remembering), BWT+, FWT, MS, SSS, and CE.
• Five CL strategies (Naïve, Cumulative, EWC, LwF, SI) are evaluated on iCIFAR-100, with results summarized in a main table showing each metric.
• The CL_score aggregates criteria with equal or user-specified weights to rank strategies; LwF often performs well across several metrics.
• The study demonstrates that metrics beyond accuracy/forgetting provide meaningful distinctions between strategies, highlighting trade-offs between memory, compute, and transfer.
• Variability across runs is illustrated (via Fig. 1), emphasizing the need for multi-metric evaluation in CL research.