[Paper Review] Transformer-Based Source-Free Domain Adaptation

Guanglei Yang, Hao Tang+5 more|arXiv (Cornell University)|May 28, 2021

Domain Adaptation and Few-Shot Learning70 references21 citations

TL;DR

This paper proposes TransDA, a Transformer-based framework for source-free domain adaptation that enhances model generalization by encouraging attention on object regions. By injecting a Transformer module into a convolutional backbone and using self-supervised knowledge distillation with pseudo-labels, TransDA improves attention focus and achieves state-of-the-art performance across closed-set, partial-set, and open-set adaptation benchmarks.

ABSTRACT

In this paper, we study the task of source-free domain adaptation (SFDA), where the source data are not available during target adaptation. Previous works on SFDA mainly focus on aligning the cross-domain distributions. However, they ignore the generalization ability of the pretrained source model, which largely influences the initial target outputs that are vital to the target adaptation stage. To address this, we make the interesting observation that the model accuracy is highly correlated with whether or not attention is focused on the objects in an image. To this end, we propose a generic and effective framework based on Transformer, named TransDA, for learning a generalized model for SFDA. Specifically, we apply the Transformer as the attention module and inject it into a convolutional network. By doing so, the model is encouraged to turn attention towards the object regions, which can effectively improve the model's generalization ability on the target domains. Moreover, a novel self-supervised knowledge distillation approach is proposed to adapt the Transformer with target pseudo-labels, thus further encouraging the network to focus on the object regions. Experiments on three domain adaptation tasks, including closed-set, partial-set, and open-set adaption, demonstrate that TransDA can greatly improve the adaptation accuracy and produce state-of-the-art results. The source code and trained models are available at https://github.com/ygjwd12345/TransDA.

Motivation & Objective

• Address the limitation of existing source-free domain adaptation (SFDA) methods that neglect the generalization ability of the pretrained source model.
• Investigate the correlation between attention focus on objects and model accuracy in SFDA.
• Improve model generalization on target domains by enhancing attention mechanisms without requiring source data.
• Develop a self-supervised knowledge distillation strategy to refine attention on target pseudo-labels.
• Demonstrate effectiveness across diverse SFDA settings: closed-set, partial-set, and open-set adaptation.

Proposed method

• Inject a Vision Transformer module after the final convolutional layer of a ResNet-50 backbone to model long-range dependencies and encourage attention on object regions.
• Use Grad-CAM visualization to empirically validate that attention on objects correlates with higher prediction accuracy on target domains.
• Apply self-supervised knowledge distillation using target pseudo-labels to further refine attention, where the student model learns from its own predictions.
• Introduce a teacher-student training strategy with exponential moving average (EMA) to stabilize knowledge distillation, though ablation shows the main gains come from distillation, not the teacher model.
• Train the model end-to-end with cross-entropy loss on pseudo-labels and knowledge distillation loss to align features and improve attention focus.
• Evaluate the method on three SFDA benchmarks: Office-31, Office-Home, and VisDA, under closed-set, partial-set, and open-set settings.

Experimental results

Research questions

• RQ1 How does attention focus on object regions affect model accuracy in source-free domain adaptation?
• RQ2 Can injecting a Transformer module into a CNN backbone improve generalization on unseen target domains?
• RQ3 Does self-supervised knowledge distillation with pseudo-labels further enhance attention focus and adaptation performance?
• RQ4 How does the proposed method compare to state-of-the-art SFDA approaches across different domain shift settings?
• RQ5 Is the improvement due to better attention focus, or is it driven by the distillation mechanism alone?

Key findings

• TransDA achieves state-of-the-art performance on all three SFDA benchmarks: 79.3% accuracy on Office-Home (closed-set), up from 71.8% with prior SOTA.
• On the VisDA dataset, TransDA improves accuracy from 81.9% to 87.6% in the open-set setting, demonstrating strong generalization.
• The ablation study confirms that adding the Transformer module boosts accuracy from 72.1% to 78.8% on Office-Home, showing consistent gains.
• Self-knowledge distillation further improves performance, increasing accuracy from 78.8% to 79.3% on Office-Home, confirming its effectiveness.
• Grad-CAM visualizations show that TransDA significantly increases attention on object regions compared to baseline models.
• t-SNE visualization confirms that features become more compact within classes and the domain gap between source and target is reduced.