[Paper Review] Diagnosing Batch Normalization in Class Incremental Learning

Minghao Zhou, Quanziang Wang+3 more|arXiv (Cornell University)|Feb 16, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper identifies a critical issue—'BN dilemma'—where standard batch normalization (BN) in class incremental learning (Class-IL) harms representation learning and causes classifier bias, worsening catastrophic forgetting. The authors propose BN Tricks: training on pure new-class batches to improve feature learning and separately updating EMA statistics with balanced batches to reduce bias, achieving significant accuracy gains across ER, DER++, and iCaRL without extra hyperparameters.

ABSTRACT

Extensive researches have applied deep neural networks (DNNs) in class incremental learning (Class-IL). As building blocks of DNNs, batch normalization (BN) standardizes intermediate feature maps and has been widely validated to improve training stability and convergence. However, we claim that the direct use of standard BN in Class-IL models is harmful to both the representation learning and the classifier training, thus exacerbating catastrophic forgetting. In this paper we investigate the influence of BN on Class-IL models by illustrating such BN dilemma. We further propose BN Tricks to address the issue by training a better feature extractor while eliminating classification bias. Without inviting extra hyperparameters, we apply BN Tricks to three baseline rehearsal-based methods, ER, DER++ and iCaRL. Through comprehensive experiments conducted on benchmark datasets of Seq-CIFAR-10, Seq-CIFAR-100 and Seq-Tiny-ImageNet, we show that BN Tricks can bring significant performance gains to all adopted baselines, revealing its potential generality along this line of research.

Motivation & Objective

• To investigate the detrimental role of standard batch normalization (BN) in class incremental learning (Class-IL), particularly its contribution to catastrophic forgetting.
• To identify the BN dilemma: poor feature learning for new classes and classifier bias due to deviated BN statistics during inference.
• To propose BN Tricks—a simple, hyperparameter-free method—to improve feature extractor quality and eliminate classification bias in rehearsal-based Class-IL models.
• To validate the effectiveness and generalization of BN Tricks across multiple benchmark datasets and baseline methods (ER, DER++, iCaRL).

Proposed method

• Train the network on batches containing only new-class samples to improve feature learning and prevent new-class representations from being dominated by previous-class buffer samples.
• Separately update the exponential moving average (EMA) statistics of BN layers using balanced batches (equal numbers of old and new class samples) to reduce classifier bias.
• Use the updated EMA statistics to faithfully transfer learned representations to the inference phase, minimizing distribution shift.
• Apply BN Tricks to existing rehearsal-based methods (ER, DER++, iCaRL) by modifying their training and statistics update procedures without altering network architecture or introducing new hyperparameters.
• Simulate inference conditions by further training on buffer samples after BN statistics are updated, ensuring robustness and fairness in classification.

Experimental results

Research questions

• RQ1 How does standard batch normalization contribute to catastrophic forgetting in class incremental learning?
• RQ2 Why does the use of EMA statistics in BN layers lead to classifier bias in continual learning scenarios?
• RQ3 Can improving feature learning for new classes and correcting BN statistics separately mitigate the BN dilemma?
• RQ4 To what extent can BN Tricks improve performance across different rehearsal-based Class-IL baselines?

Key findings

• BN Tricks achieve a 17% absolute accuracy gain over ER on Seq-CIFAR-10, with a 46% reduction in backward transfer (BWT), demonstrating strong performance gains.
• DER++-BNT improves BWT significantly and maintains higher accuracy on previous tasks, showing better stability and fairness.
• iCaRL-BNT outperforms iCaRL on all tasks, including the final task, despite a lower BWT score, indicating that BWT can be misleading when evaluating continual learning models.
• The performance gains of BN Tricks are consistent across different buffer sizes and remain effective even with long task sequences (20 tasks) on Seq-CIFAR-100-20 and Seq-Tiny-ImageNet-20.
• Visualization results (t-SNE) confirm that ER-BNT produces clearer class boundaries and reduces misclassification of old samples into new classes compared to standard ER.