[Paper Review] DualHSIC: HSIC-Bottleneck and Alignment for Continual Learning

Zifeng Wang, Zheng Zhan+5 more|arXiv (Cornell University)|Apr 30, 2023

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

DualHSIC is a novel continual learning method that enhances rehearsal-based models by introducing two HSIC-based components: HSIC-Bottleneck for Rehearsal (HBR) to reduce inter-task interference and HSIC Alignment (HA) to promote task-invariant knowledge sharing. It consistently improves state-of-the-art rehearsal methods by up to 7.6% and outperforms regularization-enhanced baselines by up to 6.5%, especially under small buffer settings.

ABSTRACT

Rehearsal-based approaches are a mainstay of continual learning (CL). They mitigate the catastrophic forgetting problem by maintaining a small fixed-size buffer with a subset of data from past tasks. While most rehearsal-based approaches study how to effectively exploit the knowledge from the buffered past data, little attention is paid to the inter-task relationships with the critical task-specific and task-invariant knowledge. By appropriately leveraging inter-task relationships, we propose a novel CL method named DualHSIC to boost the performance of existing rehearsal-based methods in a simple yet effective way. DualHSIC consists of two complementary components that stem from the so-called Hilbert Schmidt independence criterion (HSIC): HSIC-Bottleneck for Rehearsal (HBR) lessens the inter-task interference and HSIC Alignment (HA) promotes task-invariant knowledge sharing. Extensive experiments show that DualHSIC can be seamlessly plugged into existing rehearsal-based methods for consistent performance improvements, and also outperforms recent state-of-the-art regularization-enhanced rehearsal methods. Source code will be released.

Motivation & Objective

• To address catastrophic forgetting in continual learning by improving the representation learning of rehearsal-based methods.
• To mitigate inter-task interference during continual learning by removing task-specific noise from buffered data.
• To enhance knowledge transfer by encouraging the model to learn task-invariant features across tasks.
• To provide a general, plug-and-play solution compatible with existing rehearsal frameworks without architectural overhaul.
• To systematically explore the use of Hilbert-Schmidt Independence Criterion (HSIC) in continual learning for representation disentanglement.

Proposed method

• DualHSIC introduces HSIC-Bottleneck for Rehearsal (HBR), which minimizes the statistical dependence between buffered features and task-specific labels using HSIC to suppress interference from current task-specific knowledge.
• It employs HSIC Alignment (HA), which maximizes the statistical dependence between features from current and past tasks to encourage learning of shared, task-invariant representations.
• HBR uses a multi-layered loss across intermediate network activations to strengthen suppression of task-specific interference.
• HA uses a single-layered loss with a projection head to align representations across tasks, promoting positive knowledge transfer.
• The method is designed to be seamlessly integrated into existing rehearsal-based frameworks with minimal architectural changes.
• The loss functions are optimized end-to-end during training, with HBR and HA jointly improving representation quality for both past and current tasks.

Experimental results

Research questions

• RQ1 Can HSIC-based regularization effectively reduce inter-task interference in continual learning?
• RQ2 Can HSIC-based alignment improve the sharing of task-invariant knowledge across sequential tasks?
• RQ3 How does DualHSIC perform when integrated into existing rehearsal-based continual learning methods?
• RQ4 Does DualHSIC maintain performance gains under small buffer sizes, where data efficiency is critical?
• RQ5 What is the contribution of multi-layered vs. single-layered HSIC losses in HBR and HA components?

Key findings

• DualHSIC consistently improves state-of-the-art rehearsal-based methods by up to 7.6% across multiple benchmarks, demonstrating strong generalization.
• The method outperforms stronger regularization-enhanced rehearsal methods by up to 6.5%, particularly excelling under small buffer sizes.
• HBR significantly reduces class overlap in t-SNE visualizations, showing better preservation of past task representations over time.
• The ablation study confirms that both the projection head and symmetric loss term in HBR and HA are essential for optimal performance.
• Multi-layered HBR improves performance with deeper supervision, while single-layered HA suffices and reduces computational overhead.
• DualHSIC maintains strong performance even with limited buffer capacity, indicating high data efficiency.