[Paper Review] DER: Dynamically Expandable Representation for Class Incremental Learning

Shipeng Yan, Jiangwei Xie+1 more|arXiv (Cornell University)|Mar 31, 2021

Domain Adaptation and Few-Shot LearningComputer Science38 references42 citations

TL;DR

DER introduces a dynamically expandable representation for class incremental learning that freezes old features and augments with a new learnable extractor at each step, aided by auxiliary loss and channel-level pruning to balance stability and plasticity under memory limits, achieving state-of-the-art results on CIFAR-100 and ImageNet subsets.

ABSTRACT

We address the problem of class incremental learning, which is a core step towards achieving adaptive vision intelligence. In particular, we consider the task setting of incremental learning with limited memory and aim to achieve better stability-plasticity trade-off. To this end, we propose a novel two-stage learning approach that utilizes a dynamically expandable representation for more effective incremental concept modeling. Specifically, at each incremental step, we freeze the previously learned representation and augment it with additional feature dimensions from a new learnable feature extractor. This enables us to integrate new visual concepts with retaining learned knowledge. We dynamically expand the representation according to the complexity of novel concepts by introducing a channel-level mask-based pruning strategy. Moreover, we introduce an auxiliary loss to encourage the model to learn diverse and discriminate features for novel concepts. We conduct extensive experiments on the three class incremental learning benchmarks and our method consistently outperforms other methods with a large margin.

Motivation & Objective

• Address the stability-plasticity dilemma in class incremental learning with limited memory.
• Propose a two-stage framework that freezes prior representation and adds new feature extractors per incremental step.
• Enable dynamic expansion of the representation with a pruning mechanism to maintain compactness and efficiency.

Proposed method

• Maintain a two-stage learning scheme: representation learning and classifier learning.
• At step t, freeze prior feature extractors and add a new extractor Ft, concatenating features as [Φt-1(x), Ft(x)].
• Use an auxiliary loss on Ft to encourage diverse and discriminative features for new classes.
• Apply differentiable channel-level mask-based pruning to Ft to adapt model capacity to novelty.
• Train with cross-entropy on memory and new data, plus an auxiliary classifier to distinguish old vs. new concepts (ER loss).
• Perform balanced finetuning of the classifier to mitigate class imbalance.

Figure 1 : The average incremental accuracy for different model size. We compare our model with prior methods (WA [ 39 ] , BiC [ 12 ] , RPSNet [ 26 ] , iCaRL [ 27 ] , UCIR [ 12 ] , PODNet [ 6 ] ) and the model trained on all the data (Joint) on the experiment CIFAR100-B0 of 10 steps.

Experimental results

Research questions

• RQ1 Can a dynamically expandable representation improve the stability-plasticity trade-off in class incremental learning under memory constraints?
• RQ2 Does augmenting the representation with a new, frozen prior plus a learnable extractor, alongside pruning and auxiliary losses, yield superior performance over existing methods?
• RQ3 What is the impact of auxiliary loss and pruning on backward/forward transfer and overall efficiency?
• RQ4 How does DER perform across CIFAR-100 and ImageNet subsets under standard incremental protocols?],
• RQ5 key_findings only from source:
• RQ6 key_findings_not_tabular:

Key findings

• DER consistently outperforms prior state-of-the-art methods across CIFAR-100 and ImageNet benchmarks under multiple incremental protocols.
• The method achieves notable gains in last-step accuracy, e.g., on CIFAR100-B0 with 50 steps the last-step accuracy improves by 7.73 percentage points compared to strong baselines.
• In CIFAR-100 experiments, DER shows increasing advantage with larger model sizes and more steps, indicating better scalability to longer incremental sequences.
• On ImageNet-100 and ImageNet-1000 benchmarks, DER improves average top-5 accuracy by about 1.79 percentage points on ImageNet-100-B0 and increases last-step top-1 accuracy by 5.15 percentage points on ImageNet-100-B50, and 3.02 percentage points on ImageNet-1000-B0.
• Ablation studies show that expandable representation yields a large gain (e.g., 12.0+ percentage points) over non-expanding baselines; adding the auxiliary loss provides additional improvements; pruning maintains compactness with minimal performance loss.
• The approach demonstrates positive backward transfer and forward transfer of the representation across steps, indicating effective knowledge retention and transfer.

Figure 2 : Dynamically Expandable Representation Learning. At step $t$ , the model is composed of super-feature extractor $\Phi_{t}$ and classifier $\mathcal{H}_{t}$ , where $\Phi_{t}$ is built by expanding the existing super-feature extractor $\Phi_{t-1}^{P}$ with new feature extractor $\mathcal{F}