[Paper Review] On Learning Invariant Representation for Domain Adaptation

Han Zhao, Rémi Tachet des Combes+2 more|arXiv (Cornell University)|Jan 27, 2019

Domain Adaptation and Few-Shot LearningComputer Science43 references156 citations

TL;DR

The paper shows that learning invariant representations with small source error is not sufficient for target performance; it provides a generalization upper bound that accounts for conditional shift and proves an information-theoretic lower bound, plus experiments confirming the theory.

ABSTRACT

Due to the ability of deep neural nets to learn rich representations, recent advances in unsupervised domain adaptation have focused on learning domain-invariant features that achieve a small error on the source domain. The hope is that the learnt representation, together with the hypothesis learnt from the source domain, can generalize to the target domain. In this paper, we first construct a simple counterexample showing that, contrary to common belief, the above conditions are not sufficient to guarantee successful domain adaptation. In particular, the counterexample exhibits \emph{conditional shift}: the class-conditional distributions of input features change between source and target domains. To give a sufficient condition for domain adaptation, we propose a natural and interpretable generalization upper bound that explicitly takes into account the aforementioned shift. Moreover, we shed new light on the problem by proving an information-theoretic lower bound on the joint error of \emph{any} domain adaptation method that attempts to learn invariant representations. Our result characterizes a fundamental tradeoff between learning invariant representations and achieving small joint error on both domains when the marginal label distributions differ from source to target. Finally, we conduct experiments on real-world datasets that corroborate our theoretical findings. We believe these insights are helpful in guiding the future design of domain adaptation and representation learning algorithms.

Motivation & Objective

• Assess whether invariant representations plus small source error guarantee target accuracy.
• Identify conditions under which domain adaptation with invariant representations succeeds.
• Provide an interpretable generalization upper bound that accounts for conditional shift between domains.
• Establish an information-theoretic lower bound showing tradeoffs when label distributions differ across domains.
• Validate theoretical insights with experiments on real-world datasets.

Proposed method

• Construct a simple counterexample showing that aligned invariant representations can yield large joint error.
• Propose a generalization upper bound that explicitly incorporates conditional shift between source and target.
• Prove an information-theoretic lower bound on the joint error for any invariant-representation-based method.
• Relate the new bound to the classic Ben-David et al. framework and discuss when it is tighter.
• Use empirical Rademacher complexity to derive data-dependent estimates of the bound components.
• Conduct experiments on real-world datasets to corroborate theoretical findings.

Experimental results

Research questions

• RQ1 Is finding invariant representations while achieving small source error sufficient to guarantee small target error?
• RQ2 Under what conditions does invariant representation learning lead to successful domain adaptation?
• RQ3 How does conditional shift between source and target affect domain adaptation bounds?
• RQ4 What is the fundamental tradeoff between learning invariant representations and achieving small joint error when marginal label distributions differ?
• RQ5 Do empirical results on real data support the proposed bounds and insights?

Key findings

• A counterexample demonstrates that invariant representations with small source error do not guarantee small target error.
• A generalization upper bound is derived that accounts for conditional shift and can be tighter than prior bounds in some cases.
• An information-theoretic lower bound shows a fundamental tradeoff between invariant representations and joint target-source error when label distributions differ.
• The bounds imply that adapting should also align label distributions when learning invariant representations.
• Empirical experiments on real-world datasets corroborate the theoretical conclusions.