[Paper Review] Prior Knowledge Guided Unsupervised Domain Adaptation

Tao Sun, Cheng Lu+1 more|arXiv (Cornell University)|Jul 18, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes Knowledge-guided Unsupervised Domain Adaptation (KUDA), a novel framework that improves model generalization in domain shift scenarios by incorporating prior knowledge about target domain class distributions. It introduces a rectification module that refines pseudo-labels using unary bounds and binary relationships as constraints via a Zero-One Programming formulation with smooth regularization, significantly boosting performance in self-training UDA methods like SHOT and DINE on multiple benchmarks.

ABSTRACT

The waive of labels in the target domain makes Unsupervised Domain Adaptation (UDA) an attractive technique in many real-world applications, though it also brings great challenges as model adaptation becomes harder without labeled target data. In this paper, we address this issue by seeking compensation from target domain prior knowledge, which is often (partially) available in practice, e.g., from human expertise. This leads to a novel yet practical setting where in addition to the training data, some prior knowledge about the target class distribution are available. We term the setting as Knowledge-guided Unsupervised Domain Adaptation (KUDA). In particular, we consider two specific types of prior knowledge about the class distribution in the target domain: Unary Bound that describes the lower and upper bounds of individual class probabilities, and Binary Relationship that describes the relations between two class probabilities. We propose a general rectification module that uses such prior knowledge to refine model generated pseudo labels. The module is formulated as a Zero-One Programming problem derived from the prior knowledge and a smooth regularizer. It can be easily plugged into self-training based UDA methods, and we combine it with two state-of-the-art methods, SHOT and DINE. Empirical results on four benchmarks confirm that the rectification module clearly improves the quality of pseudo labels, which in turn benefits the self-training stage. With the guidance from prior knowledge, the performances of both methods are substantially boosted. We expect our work to inspire further investigations in integrating prior knowledge in UDA. Code is available at https://github.com/tsun/KUDA.

Motivation & Objective

• To address the challenge of unsupervised domain adaptation (UDA) under large domain shifts where labeled target data are unavailable.
• To explore the practical feasibility of integrating prior knowledge about target class distributions—commonly available from domain experts or historical data—into UDA.
• To develop a general, plug-in rectification module that improves pseudo-label quality by enforcing prior knowledge constraints.
• To validate that prior knowledge can substantially enhance self-training-based UDA methods, especially when distribution shifts are large.
• To inspire future research on systematic integration of prior knowledge in domain adaptation.

Proposed method

• Proposes a novel setting, Knowledge-guided Unsupervised Domain Adaptation (KUDA), where prior knowledge about target class distribution is available alongside unlabeled target data.
• Defines two types of prior knowledge: Unary Bound (lower and upper bounds on individual class probabilities) and Binary Relationship (relative ordering between two class probabilities).
• Designs a rectification module that formulates pseudo-label refinement as a Zero-One Programming (ZOP) problem constrained by the prior knowledge.
• Introduces a smooth regularizer to maintain consistency of refined pseudo-labels across neighboring samples.
• Integrates the rectification module into self-training-based UDA methods such as SHOT and DINE, creating kSHOT and kDINE.
• Solves the ZOP problem approximately using convex relaxation and iterative optimization to enable end-to-end training.

Experimental results

Research questions

• RQ1 Can prior knowledge about target class distributions improve the quality of pseudo-labels in unsupervised domain adaptation?
• RQ2 How effective is the proposed rectification module in refining model-generated pseudo-labels using unary bounds and binary relationships?
• RQ3 Does integrating prior knowledge lead to consistent performance gains across diverse domain shift scenarios, especially with large distribution shifts?
• RQ4 How does the proposed method compare to state-of-the-art UDA methods like SHOT and DINE when prior knowledge is available?
• RQ5 Can the rectification module be effectively and flexibly integrated into existing self-training-based UDA frameworks?

Key findings

• The proposed rectification module significantly improves the quality of pseudo-labels by enforcing prior knowledge constraints, as evidenced by reduced Kullback-Leibler divergence between teacher model outputs and ground-truth target distributions.
• kSHOT and kDINE, the enhanced versions of SHOT and DINE with the rectification module, achieve substantially higher accuracy than their vanilla counterparts on four UDA benchmarks.
• On Office-Home, kDINE with unary bounds (UB) achieves an average accuracy of 71.4%, outperforming DINE (69.7%) and DINE∗∗ (67.9%), demonstrating the effectiveness of prior knowledge integration.
• On VisDA-2017, kDINE with binary relationships (BR) achieves 85.3% average accuracy, surpassing DINE’s 84.2%, confirming performance gains under label-shifted settings.
• Ablation studies show that using one-hot labels instead of smoothed labels degrades performance, but kDINE∗ still outperforms DINE∗∗, proving the value of prior knowledge even under suboptimal settings.
• The method is robust to initialization, with low standard deviation across runs, indicating stable performance improvements.