[Paper Review] ViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptation

Jiaming Liu, Senqiao Yang+6 more|arXiv (Cornell University)|Jun 7, 2023

Domain Adaptation and Few-Shot Learning13 citations

TL;DR

ViDA introduces dual-rank Visual Domain Adapters with a Homeostatic Knowledge Allotment strategy to tackle error accumulation and forgetting in Continual Test-Time Adaptation, achieving state-of-the-art results on classification and segmentation benchmarks.

ABSTRACT

Since real-world machine systems are running in non-stationary environments, Continual Test-Time Adaptation (CTTA) task is proposed to adapt the pre-trained model to continually changing target domains. Recently, existing methods mainly focus on model-based adaptation, which aims to leverage a self-training manner to extract the target domain knowledge. However, pseudo labels can be noisy and the updated model parameters are unreliable under dynamic data distributions, leading to error accumulation and catastrophic forgetting in the continual adaptation process. To tackle these challenges and maintain the model plasticity, we design a Visual Domain Adapter (ViDA) for CTTA, explicitly handling both domain-specific and domain-shared knowledge. Specifically, we first comprehensively explore the different domain representations of the adapters with trainable high-rank or low-rank embedding spaces. Then we inject ViDAs into the pre-trained model, which leverages high-rank and low-rank features to adapt the current domain distribution and maintain the continual domain-shared knowledge, respectively. To exploit the low-rank and high-rank ViDAs more effectively, we further propose a Homeostatic Knowledge Allotment (HKA) strategy, which adaptively combines different knowledge from each ViDA. Extensive experiments conducted on four widely used benchmarks demonstrate that our proposed method achieves state-of-the-art performance in both classification and segmentation CTTA tasks. Note that, our method can be regarded as a novel transfer paradigm for large-scale models, delivering promising results in adaptation to continually changing distributions. Project page: https://sites.google.com/view/iclr2024-vida/home.

Motivation & Objective

• Motivate continual test-time adaptation (CTTA) and address error accumulation and catastrophic forgetting under non-stationary target domains.
• Propose Visual Domain Adapters (ViDAs) with high-rank and low-rank representations to capture domain-specific and domain-shared knowledge.
• Introduce a Homeostatic Knowledge Allotment (HKA) strategy to dynamically fuse information from ViDAs based on uncertainty.
• Enable parameter-efficient adaptation by re-parameterizing ViDAs into the pre-trained model without increasing parameters.
• Demonstrate state-of-the-art performance on multiple CTTA benchmarks for both classification and segmentation.

Proposed method

• Inject high-rank and low-rank ViDAs into pre-trained networks to capture domain-specific and domain-shared knowledge.
• Use a teacher-student framework with a consistency loss to update ViDAs via pseudo labels from a teacher model.
• Compute an uncertainty score via MC Dropout to quantify distribution shift for each sample.
• Dynamically fuse ViDA outputs with original features through Homeostatic Knowledge Allotment (HKA), adjusting fusion weights by uncertainty.
• Project ViDAs into the backbone with re-parameterization to avoid extra parameters during inference.
• Optimize with a consistency loss and EMA-updated teacher model to guide continual adaptation.

Experimental results

Research questions

• RQ1 How can CTTA be improved to mitigate error accumulation and catastrophic forgetting in continually changing target domains?
• RQ2 Can dual-representation ViDAs (high-rank and low-rank) capture domain-specific and domain-shared knowledge effectively?
• RQ3 Does a homeostatic fusion strategy improve knowledge integration across domain shifts in CTTA?
• RQ4 Can parameter-efficient adapters enable continual adaptation of foundation and large-scale models without sacrificing plasticity?
• RQ5 Do ViDAs improve performance across both classification and segmentation CTTA benchmarks and generalize to unseen domains?

Key findings

• Low-rank ViDAs tend to learn domain-shared knowledge and reduce inter-domain divergence, aiding robustness across target domains.
• High-rank ViDAs focus on domain-specific knowledge and help mitigate error accumulation within domains.
• The proposed HKA strategy dynamically fuses ViDAs based on an uncertainty measure, improving adaptation quality.
• ViDAs can be embedded via re-parameterization with no extra parameters during inference, preserving model plasticity.
• Experiments show state-of-the-art performance on four CTTA benchmarks for classification and segmentation, and improved generalization to unseen domains.