[Paper Review] Towards Unsupervised Domain Adaptation via Domain-Transformer

Ren Chuan-Xian, Zhai Yi-Ming+2 more|arXiv (Cornell University)|Feb 24, 2022

Domain Adaptation and Few-Shot Learning58 references4 citations

TL;DR

This paper proposes Domain-Transformer (DoT), a novel unsupervised domain adaptation method that integrates CNN backbones with a plug-and-play domain-level attention mechanism to learn local semantic consistency across domains. By leveraging optimal transport theory and manifold regularization, DoT achieves state-of-the-art performance without pseudo-labels or explicit domain discrepancy optimization, significantly reducing domain shift and generalization error.

ABSTRACT

As a vital problem in pattern analysis and machine intelligence, Unsupervised Domain Adaptation (UDA) attempts to transfer an effective feature learner from a labeled source domain to an unlabeled target domain. Inspired by the success of the Transformer, several advances in UDA are achieved by adopting pure transformers as network architectures, but such a simple application can only capture patch-level information and lacks interpretability. To address these issues, we propose the Domain-Transformer (DoT) with domain-level attention mechanism to capture the long-range correspondence between the cross-domain samples. On the theoretical side, we provide a mathematical understanding of DoT: 1) We connect the domain-level attention with optimal transport theory, which provides interpretability from Wasserstein geometry; 2) From the perspective of learning theory, Wasserstein distance-based generalization bounds are derived, which explains the effectiveness of DoT for knowledge transfer. On the methodological side, DoT integrates the domain-level attention and manifold structure regularization, which characterize the sample-level information and locality consistency for cross-domain cluster structures. Besides, the domain-level attention mechanism can be used as a plug-and-play module, so DoT can be implemented under different neural network architectures. Instead of explicitly modeling the distribution discrepancy at domain-level or class-level, DoT learns transferable features under the guidance of long-range correspondence, so it is free of pseudo-labels and explicit domain discrepancy optimization. Extensive experiment results on several benchmark datasets validate the effectiveness of DoT.

Motivation & Objective

• To address the challenge of domain shift in unsupervised domain adaptation (UDA) where source and target domains exhibit distribution discrepancies.
• To develop a method that enhances local semantic consistency across domains without relying on pseudo-labels or explicit domain discrepancy minimization.
• To integrate the strengths of CNNs and Transformers by introducing a domain-level attention mechanism that models sample correspondence between domains.
• To theoretically connect the domain-level attention mechanism with optimal transport and statistical learning theory, providing a new interpretation of attention in feature transfer.

Proposed method

• Proposes a plug-and-play domain-level attention mechanism based on cross-attention in Transformers to learn sample-wise correspondence between source and target domains.
• Integrates the domain-level attention with pre-trained CNN backbones, preserving the inductive bias of CNNs while enabling long-range dependency modeling.
• Employs manifold regularization to enforce local semantic consistency in the feature space, promoting intra-class compactness and inter-class separability.
• Theoretically links the domain-level attention to optimal transport via a Wasserstein metric, deriving a generalization error bound that connects attention to domain alignment.
• Avoids explicit domain discrepancy optimization (e.g., MMD, adversarial training) and does not require pseudo-labels for target samples.
• Uses a 2-Wasserstein distance (W2-distance) as a proxy for domain discrepancy, which is minimized during training to reduce the generalization upper bound.

Experimental results

Research questions

• RQ1 Can a domain-level attention mechanism based on Transformers improve feature transfer in unsupervised domain adaptation without replacing CNN backbones?
• RQ2 How does domain-level attention contribute to local semantic consistency and domain alignment across source and target domains?
• RQ3 Can the domain-level attention mechanism be theoretically linked to optimal transport and statistical learning theory to provide a principled understanding of its generalization behavior?
• RQ4 Does DoT outperform existing SOTA methods in UDA benchmarks without relying on pseudo-labels or explicit domain discrepancy loss?
• RQ5 How does the proposed method compare in terms of domain discrepancy metrics like A-distance and A_C-distance?

Key findings

• DoT achieves state-of-the-art performance on ImageCLEF-DA and VisDA-2017 benchmarks, outperforming existing methods including DeepJDOT and ATM.
• The target domain’s within-class scatter decreases significantly during training, indicating improved intra-class compactness due to domain-level attention.
• Between-class scatter in the target domain stabilizes at a high level, demonstrating enhanced inter-class separability.
• The SrcMean-to-TarMean distance decreases over time, confirming improved class-wise domain alignment.
• The 2-Wasserstein distance (W2-distance) between domains decreases progressively, showing effective minimization of the generalization upper bound.
• DoT achieves the smallest A-distance and A_C-distance on both ImageCLEF-DA (I→C) and VisDA-2017 (S→R), confirming superior domain and class-wise alignment.