[Paper Review] Information-Theoretic Analysis of Unsupervised Domain Adaptation

Ziqiao Wang, Yongyi Mao|arXiv (Cornell University)|Oct 3, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper presents an information-theoretic analysis of unsupervised domain adaptation (UDA), introducing novel generalization bounds that link KL divergence between domain distributions to generalization error. It proposes two algorithm-dependent regularization techniques—gradient penalty and class-wise labeling—that improve KL-guided domain alignment, demonstrating consistent performance gains across multiple benchmarks including RotatedMNIST, SVHN→MNIST, and VisDA17.

ABSTRACT

This paper uses information-theoretic tools to analyze the generalization error in unsupervised domain adaptation (UDA). We present novel upper bounds for two notions of generalization errors. The first notion measures the gap between the population risk in the target domain and that in the source domain, and the second measures the gap between the population risk in the target domain and the empirical risk in the source domain. While our bounds for the first kind of error are in line with the traditional analysis and give similar insights, our bounds on the second kind of error are algorithm-dependent, which also provide insights into algorithm designs. Specifically, we present two simple techniques for improving generalization in UDA and validate them experimentally.

Motivation & Objective

• To provide a theoretical foundation for unsupervised domain adaptation (UDA) using information-theoretic tools.
• To analyze two types of generalization error: population-to-population (PP) and expected empirical-to-population (EP) error.
• To develop algorithm-dependent generalization bounds that inform principled algorithm design in UDA.
• To propose and validate two new regularization techniques—gradient penalty and class-wise labeling—that enhance KL-guided domain alignment.

Proposed method

• Uses the Donsker-Varadhan representation of KL divergence as a theoretical foundation for deriving generalization bounds.
• Derives uniform upper bounds on PP generalization error using KL divergence between source and target domain distributions.
• Introduces algorithm-dependent bounds on EP generalization error to guide the design of new regularization strategies.
• Proposes two new techniques: gradient penalty (KL-GP) and class-wise labeling (KL-CL) to improve generalization in KL-guided UDA.
• Employs a two-stream training process: one stream optimizes the main classifier, and a second auxiliary stream learns pseudo-labels for improved representation alignment.
• Validates methods on RotatedMNIST, SVHN→MNIST, and VisDA17 using t-SNE visualization and ablation studies on hyperparameters.

Experimental results

Research questions

• RQ1 How does KL divergence between source and target domain distributions relate to generalization error in UDA?
• RQ2 Can algorithm-dependent generalization bounds lead to improved UDA algorithm design?
• RQ3 Does incorporating gradient penalty or class-wise labeling improve the performance of KL-guided domain alignment?
• RQ4 How does the number of unlabelled target samples affect model generalization in UDA?
• RQ5 Can the dynamic of Jeffrey’s divergence in the representation space predict test error evolution?

Key findings

• The PP generalization error is uniformly bounded by the KL divergence between source and target domains, recovering and extending prior bounds from Nguyen et al. (2022).
• Minimizing KL divergence also minimizes other discrepancy measures such as Total Variation and Wasserstein distance, explaining the effectiveness of KL-based alignment.
• KL-GP (KL with gradient penalty) improves performance on SVHN→MNIST by 1.3% (93.6±1.2 vs. 92.5±0.9) and on RotatedMNIST by 12.5% (88.0±8.1 vs. 75.5±2.4) over baseline KL.
• KL-CL (KL with class-wise labeling) achieves 71.3±0.4 on VisDA17, outperforming the baseline KL (70.6±0.5) and showing consistent gains across domains.
• The dynamic of Jeffrey’s divergence in the representation space correlates strongly with test error, with optimal performance achieved using less than half the available unlabelled target data.
• Ablation studies confirm that setting the gradient penalty hyperparameter λ1 to zero reduces KL-GP to standard KL, and that λ1 > 0 significantly improves performance.