[Paper Review] Isolation and Impartial Aggregation: A Paradigm of Incremental Learning without Interference

Yabin Wang, Zhiheng Ma+4 more|arXiv (Cornell University)|Nov 29, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes ESN, a rehearsal-free incremental learning framework that isolates stage-specific classifiers to prevent interference and uses anchor-based energy self-normalization to impartially aggregate them. By calibrating confidence scores via temperature-controlled energy metrics and voting-based inference, ESN achieves state-of-the-art performance on four benchmarks with minimal forgetting.

ABSTRACT

This paper focuses on the prevalent performance imbalance in the stages of incremental learning. To avoid obvious stage learning bottlenecks, we propose a brand-new stage-isolation based incremental learning framework, which leverages a series of stage-isolated classifiers to perform the learning task of each stage without the interference of others. To be concrete, to aggregate multiple stage classifiers as a uniform one impartially, we first introduce a temperature-controlled energy metric for indicating the confidence score levels of the stage classifiers. We then propose an anchor-based energy self-normalization strategy to ensure the stage classifiers work at the same energy level. Finally, we design a voting-based inference augmentation strategy for robust inference. The proposed method is rehearsal free and can work for almost all continual learning scenarios. We evaluate the proposed method on four large benchmarks. Extensive results demonstrate the superiority of the proposed method in setting up new state-of-the-art overall performance. \emph{Code is available at} \url{https://github.com/iamwangyabin/ESN}.

Motivation & Objective

• Address performance imbalance across learning stages in incremental learning, where later stages suffer from catastrophic forgetting and low accuracy.
• Overcome the challenge of aggregating stage-specific classifiers with inconsistent confidence distributions across tasks.
• Develop a rehearsal-free paradigm that avoids storing past data while maintaining high accuracy across all stages.
• Ensure impartial aggregation of classifiers by normalizing their output confidence levels to a shared energy plane.
• Design a robust inference mechanism that enhances generalization through voting-based augmentation.

Proposed method

• Use a fixed, pre-trained Vision Transformer (ViT) backbone as a frozen feature extractor to provide consistent in-stage feature representations.
• Introduce a temperature-controlled energy metric to quantify and align the confidence score levels of stage-specific classifiers.
• Propose an anchor-based energy self-normalization loss to enforce all stage classifiers to operate on the same energy plane, enabling fair aggregation.
• Apply a class-attention block (CAB) as a lightweight, task-specific decoder to learn stage-specific classifiers without fine-tuning the backbone.
• Implement a voting-based inference strategy that combines predictions from all stage classifiers to improve robustness and generalization.
• Use a dynamic temperature selection strategy based on voting outcomes to further calibrate confidence scores and enhance performance.

Experimental results

Research questions

• RQ1 Can stage-isolated classifiers be effectively aggregated without rehearsal, while maintaining impartiality across all learning stages?
• RQ2 How can confidence scores from classifiers trained on different data distributions be normalized to allow fair and robust ensemble inference?
• RQ3 What is the impact of energy-level alignment on mitigating performance imbalance across incremental learning stages?
• RQ4 How does the proposed method compare to existing rehearsal-based and rehearsal-free approaches in terms of accuracy and forgetting?
• RQ5 Can the framework generalize across different network architectures and downstream tasks without architectural constraints?

Key findings

• ESN achieves a new state-of-the-art FAA of 86.34% on Split CIFAR-100, significantly outperforming prior methods.
• The ablation study shows that removing energy self-normalization reduces FAA by 6.13 percentage points, proving its critical role in impartial aggregation.
• Temperature calibration via voting improves performance by 0.61% over fixed-temperature baselines, demonstrating its effectiveness in confidence calibration.
• Parameter isolation via the class-attention block is essential—sharing CABs across tasks reduces FAA by 2.4% compared to isolated designs.
• The method is robust to hyperparameter choice: varying the energy anchor Δ from -15 to 0 yields only minor performance fluctuations (±0.38% in FAA).
• Among different architectures, CAB-based ESN achieves the best trade-off between performance (86.34% FAA) and parameter efficiency (3.4% relative increase), outperforming VPT and DER methods despite lower parameter expansion.