[Paper Review] General Incremental Learning with Domain-aware Categorical Representations

Jiangwei Xie, Shipeng Yan+1 more|arXiv (Cornell University)|Apr 8, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes a novel offline general incremental learning framework that addresses both inter-class and intra-class stability-plasticity dilemmas and domain imbalance by using a von Mises-Fisher mixture model for domain-aware class representations and a bi-level balanced memory. The method achieves state-of-the-art performance across iDigits, iDomainNet, and iCIFAR-20 benchmarks, with improvements up to +16.36% on average accuracy in the most challenging settings.

ABSTRACT

Continual learning is an important problem for achieving human-level intelligence in real-world applications as an agent must continuously accumulate knowledge in response to streaming data/tasks. In this work, we consider a general and yet under-explored incremental learning problem in which both the class distribution and class-specific domain distribution change over time. In addition to the typical challenges in class incremental learning, this setting also faces the intra-class stability-plasticity dilemma and intra-class domain imbalance problems. To address above issues, we develop a novel domain-aware continual learning method based on the EM framework. Specifically, we introduce a flexible class representation based on the von Mises-Fisher mixture model to capture the intra-class structure, using an expansion-and-reduction strategy to dynamically increase the number of components according to the class complexity. Moreover, we design a bi-level balanced memory to cope with data imbalances within and across classes, which combines with a distillation loss to achieve better inter- and intra-class stability-plasticity trade-off. We conduct exhaustive experiments on three benchmarks: iDigits, iDomainNet and iCIFAR-20. The results show that our approach consistently outperforms previous methods by a significant margin, demonstrating its superiority.

Motivation & Objective

• To address the general incremental learning problem where both class and intra-class domain distributions shift over time, introducing new challenges in stability-plasticity and data imbalance.
• To overcome the intra-class stability-plasticity dilemma, where models must adapt to new domains of previously learned classes without forgetting prior knowledge.
• To mitigate intra-class domain imbalance, where new domains dominate due to limited memory and biased data sampling.
• To develop a flexible, dynamic class representation that captures complex intra-class structures across multiple domains.
• To unify inter- and intra-class stability-plasticity trade-offs through a joint learning framework with distillation and memory balancing.

Proposed method

• Uses a von Mises-Fisher (vMF) mixture model to represent class features, enabling domain-aware clustering of intra-class data across multiple domains.
• Employs an expansion-and-reduction strategy: dynamically increasing mixture components when new domains emerge and reducing them via hierarchical clustering to maintain compactness.
• Applies the Expectation-Maximization (EM) algorithm to jointly optimize the backbone network and vMF mixture components, treating component assignments as latent variables.
• Introduces a bi-level balanced memory sampling strategy that balances data across and within classes, reducing bias toward new domains and new classes.
• Incorporates both inter-class and intra-class knowledge distillation losses to resist catastrophic forgetting during incremental updates.
• Performs iterative model updates per session: inherit previous model, assign new components, update via EM, then reduce components via clustering.

Experimental results

Research questions

• RQ1 Can a flexible, domain-aware class representation effectively model intra-class data distribution shifts across multiple domains?
• RQ2 How can the intra-class stability-plasticity trade-off be improved when domain labels are unknown and new domains emerge over time?
• RQ3 To what extent can a bi-level memory strategy mitigate data imbalance both within and across classes in general incremental learning?
• RQ4 Does integrating domain-aware representation with EM-based learning and distillation lead to consistent performance gains across diverse benchmarks?
• RQ5 How robust is the method to hyperparameter choices such as the threshold in mixture reduction and memory size?

Key findings

• The proposed method achieves a +16.36% improvement in average incremental accuracy on the iCIFAR-20-NCD split when integrated with DER, rising from 60.04% to 76.40%.
• On the iCIFAR-20-ND split, the method improves DER’s performance from 78.24% to 84.11% by addressing the domain shift bottleneck.
• UCIR combined with the proposed method achieves 90.42% accuracy on the iDigits-NCD split, outperforming all baselines.
• Ablation studies show a 0.89% gain on the ND split and a 2.07% gain on the ND split after adding the bi-level memory sampling strategy.
• t-SNE visualization confirms that the method successfully separates instances into distinct domain-specific clusters, with high purity per cluster across sessions.
• The method is robust to hyperparameter variations, particularly the threshold in mixture reduction, as shown in sensitivity analysis.