[Paper Review] Uncertainty-based Continual Learning with Adaptive Regularization

Hongjoon Ahn, Sungmin Cha+2 more|arXiv (Cornell University)|May 28, 2019

Domain Adaptation and Few-Shot LearningComputer Science31 references93 citations

TL;DR

The paper presents Uncertainty-based Continual Learning (UCL) with adaptive regularization, deriving a KL-based regularizer for Bayesian weight distributions, and introducing adaptive initialization of uncertainties to improve continual learning across various tasks and architectures.

ABSTRACT

We introduce a new neural network-based continual learning algorithm, dubbed as Uncertainty-regularized Continual Learning (UCL), which builds on traditional Bayesian online learning framework with variational inference. We focus on two significant drawbacks of the recently proposed regularization-based methods: a) considerable additional memory cost for determining the per-weight regularization strengths and b) the absence of gracefully forgetting scheme, which can prevent performance degradation in learning new tasks. In this paper, we show UCL can solve these two problems by introducing a fresh interpretation on the Kullback-Leibler (KL) divergence term of the variational lower bound for Gaussian mean-field approximation. Based on the interpretation, we propose the notion of node-wise uncertainty, which drastically reduces the number of additional parameters for implementing per-weight regularization. Moreover, we devise two additional regularization terms that enforce stability by freezing important parameters for past tasks and allow plasticity by controlling the actively learning parameters for a new task. Through extensive experiments, we show UCL convincingly outperforms most of recent state-of-the-art baselines not only on popular supervised learning benchmarks, but also on challenging lifelong reinforcement learning tasks. The source code of our algorithm is available at https://github.com/csm9493/UCL.

Motivation & Objective

• Motivate continual learning by mitigating catastrophic forgetting in deep networks across sequences of tasks.
• Develop a Bayesian, uncertainty-aware regularization framework that adapts to architectural depth and task structure.
• Derive a closed-form KL divergence-based regularizer that links successive weight distributions for regularization.
• Propose an adaptive initialization strategy for weight uncertainties to stabilize training in deep architectures.

Proposed method

• Derive a closed-form KL divergence between Gaussian mean-field posteriors to obtain the regularization term.
• Introduce adaptive initialization of weight uncertainty (sigma) per layer to stabilize forward and backward signal propagation.
• Extend UCL to convolutional networks by conceptually treating uncertainty at the channel/filter level and illustrating regularization propagation across tasks.
• Provide implementation details for supervised learning and reinforcement learning settings, including network architectures and hyperparameter choices.
• Demonstrate parameter-efficient regularization that favors 'active learners' for future tasks and supports graceful forgetting when appropriate.

Experimental results

Research questions

• RQ1 Can uncertainty-based regularization mitigate catastrophic forgetting across sequential tasks in deep networks?
• RQ2 How should weight uncertainty (sigma) be initialized and adapted across layers to maximize continual learning performance?
• RQ3 How does adaptive regularization perform in CNNs and RL environments compared to existing baselines (EWC, SI, HAT, VCL, etc.) in terms of retention and forward transfer?
• RQ4 Can per-layer/channel level uncertainty be effectively extended to convolutional architectures to control regularization strength across tasks?

Key findings

• A closed-form KL divergence between Gaussian mean-field posteriors leads to a per-weight regularization term that aggregates to a per-layer regularization expression.
• An adaptive initialization strategy for weight uncertainty (sigma) improves stability and performance across deep architectures, with formulas guiding forward/backward variance control.
• Extending UCL to CNNs via channel/filter-level uncertainty demonstrates how important components receive stronger regularization while less relevant parts remain trainable for future tasks.
• Empirical results (per supplementary material) show that adaptive initialization often yields superior average accuracy and better retention in various task splits (e.g., CIFAR-10/100, MNIST variants) compared to baselines, and that ablation analyses highlight the necessity of upper-layer regularization and proper term (5)/(6) contributions for performance.
• Evidence that preserving useful features (high regularization on outgoing weights) enables multiple active learners to support future tasks.
• The behavior of sigma initialization and beta (regularization strength) controls the trade-off between forgetting and forward transfer, affecting cumulative rewards in reinforcement learning experiments.