[Paper Review] Support and Invertibility in Domain-Invariant Representations

Fredrik Johansson, David Sontag+1 more|arXiv (Cornell University)|Mar 8, 2019

Domain Adaptation and Few-Shot Learning32 references42 citations

TL;DR

The paper critiques domain-invariant representation learning for unsupervised domain adaptation, highlighting that non-invertible representations and strict invariance can hinder target performance, and proposes a bound based on support overlap to better characterize adaptation risk.

ABSTRACT

Learning domain-invariant representations has become a popular approach to unsupervised domain adaptation and is often justified by invoking a particular suite of theoretical results. We argue that there are two significant flaws in such arguments. First, the results in question hold only for a fixed representation and do not account for information lost in non-invertible transformations. Second, domain invariance is often a far too strict requirement and does not always lead to consistent estimation, even under strong and favorable assumptions. In this work, we give generalization bounds for unsupervised domain adaptation that hold for any representation function by acknowledging the cost of non-invertibility. In addition, we show that penalizing distance between densities is often wasteful and propose a bound based on measuring the extent to which the support of the source domain covers the target domain. We perform experiments on well-known benchmarks that illustrate the short-comings of current standard practice.

Motivation & Objective

• Question the sufficiency of domain-invariant representations under covariate shift and partial overlap.
• Provide generalization bounds that account for non-invertible representations.
• Propose a bound based on source-target support overlap rather than solely density distances.
• Investigate when representation learning helps or hurts target performance and relate theory to benchmarks.

Proposed method

• Review and formalize domain adaptation bounds under covariate shift with a fixed representation and under non-invertible transformations.
• Introduce the support sufficiency divergence to measure overlap between source and target supports.
• Derive a new target risk bound (Theorem 2) that separates observable and unobservable terms related to information loss in representations.
• Extend the bound with an IPM-based variant (Theorem 3) that leverages a function class to tighten the bound under extrapolation assumptions.
• Analyze the cost of enforcing domain invariance through density matching, including a comparison to MMD-based distances.
• Conduct experiments on MNIST/MNIST-M to illustrate when standard domain-invariant methods may underperform.

Experimental results

Research questions

• RQ1 Under what conditions do domain-invariant representations yield reliable target risk minimization?
• RQ2 How does non-invertibility of representations affect the unobservable adaptation error?
• RQ3 Can a bound based on support overlap provide tighter, more informative guarantees than density-distance-based bounds?
• RQ4 When does enforcing domain invariance harm predictive performance due to information loss?
• RQ5 How do theoretical insights align with empirical results on standard domain adaptation benchmarks?

Key findings

• Domain-invariant representations can be suboptimal or fail to guarantee good target risk when supports overlap is limited or non-invertible information is lost.
• A bound based on the extent of source support covering the target (support sufficiency) can better characterize adaptation risk than bounds relying solely on density distances.
• Non-invertible representations induce an unobservable information loss term that can degrade target performance even if source performance looks good.
• Penalizing density distances (e.g., MMD) is not always beneficial and can misrepresent the true adaptation difficulty when supports do not fully overlap.
• A tighter IPM-based bound (with assumptions on the loss class) improves over general bounds in settings with extrapolation assumptions.
• Empirical experiments on MNIST/MNIST-M illustrate scenarios where domain-adversarial training may underperform compared to source-only or target-tuned baselines, especially under label marginal shift.