[Paper Review] Overcoming Recency Bias of Normalization Statistics in Continual Learning: Balance and Adaptation

Yilin Lyu, Liyuan Wang+5 more|arXiv (Cornell University)|Oct 13, 2023

Domain Adaptation and Few-Shot LearningComputer Science3 citations

TL;DR

This paper proposes AdaB2N, a novel method to overcome recency bias in Batch Normalization (BN) during continual learning by adaptively balancing historical and current task statistics. It combines a Bayesian-based strategy for task-wise contribution weighting with a modified momentum update, achieving state-of-the-art performance—up to 7.68% accuracy gain on Split CIFAR-10—across online and offline continual learning benchmarks.

ABSTRACT

Continual learning entails learning a sequence of tasks and balancing their knowledge appropriately. With limited access to old training samples, much of the current work in deep neural networks has focused on overcoming catastrophic forgetting of old tasks in gradient-based optimization. However, the normalization layers provide an exception, as they are updated interdependently by the gradient and statistics of currently observed training samples, which require specialized strategies to mitigate recency bias. In this work, we focus on the most popular Batch Normalization (BN) and provide an in-depth theoretical analysis of its sub-optimality in continual learning. Our analysis demonstrates the dilemma between balance and adaptation of BN statistics for incremental tasks, which potentially affects training stability and generalization. Targeting on these particular challenges, we propose Adaptive Balance of BN (AdaB$^2$N), which incorporates appropriately a Bayesian-based strategy to adapt task-wise contributions and a modified momentum to balance BN statistics, corresponding to the training and testing stages. By implementing BN in a continual learning fashion, our approach achieves significant performance gains across a wide range of benchmarks, particularly for the challenging yet realistic online scenarios (e.g., up to 7.68%, 6.86% and 4.26% on Split CIFAR-10, Split CIFAR-100 and Split Mini-ImageNet, respectively). Our code is available at https://github.com/lvyilin/AdaB2N.

Motivation & Objective

• Address the recency bias in Batch Normalization (BN) statistics during continual learning, where recent tasks dominate due to gradient and statistics updates.
• Identify the inherent trade-off between balancing historical task statistics and adapting to current task distributions in BN.
• Develop a method that maintains training stability and generalization by dynamically adjusting BN statistics without relying on old data.
• Achieve strong performance in both online (one-pass) and offline (multi-epoch) continual learning settings, particularly in class-incremental and task-incremental scenarios.
• Provide a theoretically grounded, scalable solution applicable to real-world continual learning with limited access to past data.

Proposed method

• Introduce a Bayesian-based strategy to estimate task-wise contributions to BN statistics, using a categorical distribution over task identities conditioned on activation statistics.
• Propose a modified exponential moving average (EMA) momentum that adapts based on the estimated task contribution, balancing between stability and adaptability.
• Decouple the training and testing phases: during training, use adaptive momentum to stabilize learning; during inference, maintain a balanced, history-aware BN statistics estimate.
• Incorporate a regularization term with hyperparameter λ to control the influence of new task statistics on BN, ensuring smooth adaptation.
• Use a Dirichlet prior to approximate the task distribution for computational efficiency, enabling online estimation of task contributions.
• Implement the method within standard DNNs using BN layers, requiring no architectural changes or memory replay of old samples.

Experimental results

Research questions

• RQ1 How does the standard EMA-based BN update in continual learning lead to recency bias, and what are the theoretical limitations of constant-momentum EMA?
• RQ2 What is the fundamental trade-off between balancing statistics across all tasks and adapting to the current task's distribution in BN layers?
• RQ3 Can a Bayesian-based estimation of task contributions improve the balance and adaptability of BN statistics in continual learning?
• RQ4 How does the modified momentum in AdaB2N reconcile the tension between long-term stability and short-term adaptation?
• RQ5 To what extent does AdaB2N outperform existing BN adaptation strategies in online and offline continual learning settings?

Key findings

• AdaB2N achieves a 7.68% accuracy gain on Split CIFAR-10, 6.86% on Split CIFAR-100, and 4.26% on Split Mini-ImageNet in online task-incremental learning, significantly outperforming baseline BN and prior methods.
• The method maintains strong performance in offline settings, with a 1.28% improvement over BN on Task-IL and 0.38% on Class-IL for Split Mini-ImageNet (2000 memory buffer).
• Ablation studies confirm that both the Bayesian contribution weighting and adaptive momentum are essential, with consistent performance gains across a wide range of hyperparameter combinations.
• Visualization of BN statistics shows that AdaB2N closely matches the joint training baseline (upper bound), while standard BN and CN methods diverge significantly over time.
• The approach is robust to memory buffer selection strategies, delivering similar performance gains under both reservoir sampling and ring buffer memory management.
• Even in challenging offline settings where other methods like CN degrade performance, AdaB2N consistently improves accuracy, demonstrating its generalization and stability.