[Paper Review] DEJA VU: Continual Model Generalization For Unseen Domains

Chenxi Liu, Lixu Wang+4 more|arXiv (Cornell University)|Jan 25, 2023

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes RaTP, a novel framework for continual domain shift learning that enhances model generalization on unseen domains during the 'Unfamiliar Period'—before significant adaptation occurs. By integrating a training-free data augmentation (RandMix), a temporal pseudo-labeling mechanism (T2PL), and a prototype contrastive alignment (PCA) loss, RaTP achieves state-of-the-art performance in target domain generalization (TDG), with strong capabilities in target domain adaptation (TDA) and forgetting alleviation (FA) across Digits, PACS, and DomainNet benchmarks.

ABSTRACT

In real-world applications, deep learning models often run in non-stationary environments where the target data distribution continually shifts over time. There have been numerous domain adaptation (DA) methods in both online and offline modes to improve cross-domain adaptation ability. However, these DA methods typically only provide good performance after a long period of adaptation, and perform poorly on new domains before and during adaptation - in what we call the "Unfamiliar Period", especially when domain shifts happen suddenly and significantly. On the other hand, domain generalization (DG) methods have been proposed to improve the model generalization ability on unadapted domains. However, existing DG works are ineffective for continually changing domains due to severe catastrophic forgetting of learned knowledge. To overcome these limitations of DA and DG in handling the Unfamiliar Period during continual domain shift, we propose RaTP, a framework that focuses on improving models' target domain generalization (TDG) capability, while also achieving effective target domain adaptation (TDA) capability right after training on certain domains and forgetting alleviation (FA) capability on past domains. RaTP includes a training-free data augmentation module to prepare data for TDG, a novel pseudo-labeling mechanism to provide reliable supervision for TDA, and a prototype contrastive alignment algorithm to align different domains for achieving TDG, TDA and FA. Extensive experiments on Digits, PACS, and DomainNet demonstrate that RaTP significantly outperforms state-of-the-art works from Continual DA, Source-Free DA, Test-Time/Online DA, Single DG, Multiple DG and Unified DA&DG in TDG, and achieves comparable TDA and FA capabilities.

Motivation & Objective

• To address the critical challenge of poor model performance during the 'Unfamiliar Period'—when models face sudden, significant domain shifts before adaptation—especially in real-world applications like surveillance and medical imaging.
• To overcome the limitations of existing domain adaptation (DA) and domain generalization (DG) methods, which either require access to source data or suffer from catastrophic forgetting under continual domain shifts.
• To develop a unified framework that simultaneously enables strong target domain generalization (TDG), effective target domain adaptation (TDA), and forgetting alleviation (FA) in continual learning scenarios.
• To design a training-free, scalable solution that does not rely on source domain data or extensive retraining, suitable for real-time and privacy-sensitive applications.

Proposed method

• Introduces RandMix, a training-free data augmentation module that generates diverse, domain-agnostic samples via random mixing of features, enhancing generalization to unseen domains.
• Proposes T2PL (Temporal Two-Stream Pseudo-Labeling), a novel pseudo-labeling mechanism that leverages predictions from both current and past model states to generate reliable supervision for online adaptation.
• Employs PCA (Prototype Contrastive Alignment), a loss function that aligns class prototypes across domains using contrastive learning, improving domain-invariant representation learning.
• Combines cross-entropy loss, logits distillation, and the PCA loss in a unified optimization objective to jointly optimize for TDG, TDA, and FA.
• Uses a dual-stream inference mechanism in T2PL to stabilize pseudo-label generation by comparing predictions from a current model and a momentum-averaged teacher model.
• Enables source-free, continual adaptation by relying only on target domain data during inference, avoiding the need for source data or retraining from scratch.

Experimental results

Research questions

• RQ1 Can a model achieve strong generalization on unseen domains before seeing significant target data, i.e., during the 'Unfamiliar Period'?
• RQ2 How can a model maintain performance on previously seen domains while adapting to new ones, mitigating catastrophic forgetting in continual domain shift scenarios?
• RQ3 Can a unified framework simultaneously achieve high performance in target domain generalization (TDG), target domain adaptation (TDA), and forgetting alleviation (FA)?
• RQ4 What components are most critical for improving performance during the 'Unfamiliar Period' in continual domain shift learning?

Key findings

• RaTP achieves a 56.8% average accuracy on the DomainNet benchmark for target domain generalization (TDG), significantly outperforming state-of-the-art methods in this setting.
• On the PACS dataset, RaTP achieves 57.2% TDG accuracy, surpassing existing SOTA methods in the single-domain generalization setting.
• In continual domain shift experiments on Digits with the order US→SD→SN→MM→MT, RaTP maintains a 49.0% average TDG performance, demonstrating robustness across multiple domain shifts.
• Ablation studies confirm that removing RandMix, T2PL, or PCA leads to significant performance drops, validating the effectiveness of each component.
• RaTP achieves comparable or better performance in target domain adaptation (TDA) and forgetting alleviation (FA) compared to SOTA methods in continual DA, source-free DA, and test-time DA settings.
• The framework achieves a 56.8% average TDG score on DomainNet, outperforming the previous SOTA by over 6 percentage points, and maintains strong FA performance with a 53.7% average forgetting accuracy on the same benchmark.