[Paper Review] On Localized Discrepancy for Domain Adaptation

Yuchen Zhang, Mingsheng Long+2 more|arXiv (Cornell University)|Aug 14, 2020

Domain Adaptation and Few-Shot Learning46 references4 citations

TL;DR

This paper introduces localized discrepancies for domain adaptation by restricting hypothesis spaces after localization, leading to tighter generalization bounds. It proves these discrepancies reduce overestimation, reveal asymmetric transfer difficulty, and enable faster sample complexity and super transfer with boosted variants.

ABSTRACT

We propose the discrepancy-based generalization theories for unsupervised domain adaptation. Previous theories introduced distribution discrepancies defined as the supremum over complete hypothesis space. The hypothesis space may contain hypotheses that lead to unnecessary overestimation of the risk bound. This paper studies the localized discrepancies defined on the hypothesis space after localization. First, we show that these discrepancies have desirable properties. They could be significantly smaller than the pervious discrepancies. Their values will be different if we exchange the two domains, thus can reveal asymmetric transfer difficulties. Next, we derive improved generalization bounds with these discrepancies. We show that the discrepancies could influence the rate of the sample complexity. Finally, we further extend the localized discrepancies for achieving super transfer and derive generalization bounds that could be even more sample-efficient on source domain.

Motivation & Objective

• Address the overestimation problem in classical discrepancy-based domain adaptation theories, which use the supremum over full hypothesis spaces.
• Develop a new localization technique for discrepancies that improves generalization bounds and reduces conservatism.
• Investigate whether localized discrepancies can reveal asymmetric transfer difficulties between source and target domains.
• Derive improved generalization bounds where sample complexity depends on localized discrepancy values.
• Explore conditions under which super transfer—faster convergence on the source domain than target—can be achieved theoretically.

Proposed method

• Define localized discrepancies by restricting hypothesis spaces using empirical risk and confidence intervals, creating localized hypothesis sets $\tilde{\mathcal{H}}_{r^+}$ and $\tilde{\mathcal{H}}_{r^-}$.
• Introduce the $\gamma$-boosted localized $\mathcal{H}\Delta\mathcal{H}$-discrepancy to enhance sensitivity and enable super transfer.
• Derive a generalization bound (Theorem 13) that combines empirical risk, localized discrepancy, and statistical error terms with $O(\cdot)^\gamma$ rates.
• Use VC dimension and concentration inequalities to bound the generalization error with high probability.
• Establish that the source domain sample complexity can be faster than the target’s when the $\gamma$-boosted discrepancy is small.
• Leverage the structure of the hypothesis space after localization to ensure tighter bounds than classical supremum-based discrepancies.

Experimental results

Research questions

• RQ1 Can localized discrepancies significantly reduce the overestimation inherent in classical discrepancy measures?
• RQ2 Do localized discrepancies reveal asymmetric transfer difficulties when the source and target domains are swapped?
• RQ3 Can localized discrepancies lead to improved generalization bounds with faster sample complexity rates?
• RQ4 Under what conditions can the $\gamma$-boosted localized discrepancy achieve super transfer in unsupervised domain adaptation?
• RQ5 Is there a theoretical justification for using localized discrepancy over classical discrepancy in domain adaptation?

Key findings

• Localized discrepancies are strictly smaller than classical discrepancies, reducing overestimation of the risk bound.
• The values of localized discrepancies differ when the source and target domains are exchanged, enabling detection of asymmetric transfer difficulty.
• Generalization bounds derived using localized discrepancies show that sample complexity depends on the discrepancy value, with faster rates when discrepancies are small.
• The $\gamma$-boosted localized $\mathcal{H}\Delta\mathcal{H}$-discrepancy enables super transfer, where the source domain's generalization rate becomes faster than the target’s.
• The improved generalization bound (Theorem 13) includes $O(\cdot)^\gamma$ terms, showing that the rate of convergence can be accelerated with smaller discrepancies.
• The theoretical framework provides the first provable sample complexity improvement under classical assumptions (Ben-David et al., 2007) using discrepancy-based bounds.