[Paper Review] RMM: Reinforced Memory Management for Class-Incremental Learning

Yaoyao Liu, Bernt Schiele+1 more|arXiv (Cornell University)|Jan 14, 2023

Domain Adaptation and Few-Shot LearningComputer Science28 references36 citations

TL;DR

RMM introduces a hierarchical reinforcement learning policy to dynamically allocate memory in class-incremental learning, improving replay-based baselines across CIFAR-100, ImageNet-Subset, and ImageNet-Full.

ABSTRACT

Class-Incremental Learning (CIL) [40] trains classifiers under a strict memory budget: in each incremental phase, learning is done for new data, most of which is abandoned to free space for the next phase. The preserved data are exemplars used for replaying. However, existing methods use a static and ad hoc strategy for memory allocation, which is often sub-optimal. In this work, we propose a dynamic memory management strategy that is optimized for the incremental phases and different object classes. We call our method reinforced memory management (RMM), leveraging reinforcement learning. RMM training is not naturally compatible with CIL as the past, and future data are strictly non-accessible during the incremental phases. We solve this by training the policy function of RMM on pseudo CIL tasks, e.g., the tasks built on the data of the 0-th phase, and then applying it to target tasks. RMM propagates two levels of actions: Level-1 determines how to split the memory between old and new classes, and Level-2 allocates memory for each specific class. In essence, it is an optimizable and general method for memory management that can be used in any replaying-based CIL method. For evaluation, we plug RMM into two top-performing baselines (LUCIR+AANets and POD+AANets [30]) and conduct experiments on three benchmarks (CIFAR-100, ImageNet-Subset, and ImageNet-Full). Our results show clear improvements, e.g., boosting POD+AANets by 3.6%, 4.4%, and 1.9% in the 25-Phase settings of the above benchmarks, respectively.

Motivation & Objective

• Motivate improving memory allocation in class-incremental learning (CIL) under strict memory budgets to mitigate forgetting.
• Propose a hierarchical reinforcement learning policy to optimally split memory between old exemplars and new data and then allocate per-class memory.
• Enable transferable policy learning by training on pseudo CIL tasks while applying to target CIL tasks.
• Demonstrate that RMM consistently enhances top baselines (LUCIR+AANets, POD+AANets) on multiple benchmarks.

Proposed method

• Introduce a two-level hierarchical policy: Level-1 allocates memory between exemplars (old data) and new data; Level-2 allocates exemplar memory across old classes conditioned on training entropy.
• Define state s_i for phase i as (C_i / sum_{t<=i} C_t, |M_old| / |M|) to ensure transferability and phase-distinctness.
• Level-1 action a_i^{[1]} measures the ratio of exemplar memory for the first phase and incremental changes for subsequent phases with bounds to maintain a feasible memory split.
• Level-2 action a_i^{[2]} distributes the old exemplar memory among high- and low-entropy class groups (two groups found to be effective).
• Use REINFORCE-based policy optimization with a moving-average baseline to maximize cumulative validation accuracy across all phases (R = sum r_i).
• Train policy functions on pseudo CIL tasks generated from available data (e.g., D_0) to enable past/future data access for learning transferable policies.

Experimental results

Research questions

• RQ1 Can a hierarchical RL policy learn optimal, per-phase memory allocations for old vs. new data and per-class memory distribution in CIL?
• RQ2 Do pseudo CIL task trainings enable transferable policy learning to target CIL tasks without accessing past/future data during real incremental phases?
• RQ3 How much do class-specific memory allocations and entropy-based groupings improve performance over static memory splits across benchmarks and phase counts?
• RQ4 Is RMM compatible with and beneficial to existing replay-based CIL baselines such as LUCIR+AANets and POD+AANets?
• RQ5 To what extent do transfers of policy functions between datasets affect performance in CIL?

Key findings

• RMM consistently improves two strong baselines (LUCIR+AANets and POD+AANets) across benchmarks and phase counts.
• POD+AANets with RMM achieves the best overall performance, with boosts of 3.6% (CIFAR-100, N=25) and 4.4% (ImageNet-Subset, N=25) over the baselines.
• Increases in performance are more pronounced in harder, longer-horizon settings (N=25) than in smaller phase counts (N=5).
• Hierarchical RL (two-level) outperforms single-level RL, showing gains in average and last-phase accuracy across datasets.
• Policy transfer from another dataset (cross-task) yields comparable gains to in-task RL, reducing the need for target-task RL training.
• Ablation studies show memory allocations become more balanced between old exemplars and new data under RMM, addressing data imbalance issues.