[Paper Review] Domain Invariant Representation Learning with Domain Density Transformations

Antoinette T Nguyen, Trần Khánh Toàn+2 more|arXiv (Cornell University)|Feb 9, 2021

Domain Adaptation and Few-Shot Learning25 citations

TL;DR

The paper proposes learning domain-invariant representations by enforcing invariance under domain density transformations learned via GANs, improving domain generalization across Rotated MNIST, VLCS, and PACS datasets.

ABSTRACT

Domain generalization refers to the problem where we aim to train a model on data from a set of source domains so that the model can generalize to unseen target domains. Naively training a model on the aggregate set of data (pooled from all source domains) has been shown to perform suboptimally, since the information learned by that model might be domain-specific and generalize imperfectly to target domains. To tackle this problem, a predominant approach is to find and learn some domain-invariant information in order to use it for the prediction task. In this paper, we propose a theoretically grounded method to learn a domain-invariant representation by enforcing the representation network to be invariant under all transformation functions among domains. We also show how to use generative adversarial networks to learn such domain transformations to implement our method in practice. We demonstrate the effectiveness of our method on several widely used datasets for the domain generalization problem, on all of which we achieve competitive results with state-of-the-art models.

Motivation & Objective

• Motivate and formalize the problem of domain generalization and the need for joint marginal and conditional distribution alignment across domains.
• Characterize necessary and sufficient conditions for the existence of a domain-invariant representation.
• Propose a theoretically grounded method that enforces representation invariance under domain density transformations.
• Provide a practical implementation using GAN-based domain transformation learning (DIRT-GAN).
• Empirically validate the approach on standard domain generalization benchmarks (Rotated MNIST, VLCS, PACS).

Proposed method

• Define a domain-invariant representation z satisfying p(z|d) and p(y|z,d) invariance across domains.
• Introduce domain density transformation functions f_{d,d'} between domains and prove that p(z|x)=p(z|f_{d,d'}(x)) implies joint alignment across domains.
• Optimize a loss combining prediction loss l(y,g_theta(x)) with a domain-invariance term dis(g_theta(x), g_theta(f_{d,d'}(x))).
• Learn the domain transformation functions f_{d,d'} using StarGAN to enable practical estimation of domain densities.
• Implement the approach as DIRT-GAN, integrating StarGAN with a representation network g_theta to encourage invariance during training.
• Evaluate on Rotated MNIST, VLCS, and PACS with leave-one-domain-out experiments and standard backbones (CNNs, AlexNet, ResNet18).

Experimental results

Research questions

• RQ1 Under what conditions does a domain-invariant representation exist across multiple source domains?
• RQ2 Can a representation invariant to learned domain density transformations achieve joint marginal and conditional alignment across domains?
• RQ3 Does incorporating GAN-learned domain transformations into the representation learning objective improve domain generalization performance?
• RQ4 How does the proposed method compare to existing domain-invariant baselines on standard DG benchmarks?
• RQ5 Is the approach scalable to multiple domains and practical at test time?

Key findings

• DIRT-GAN outperforms domain-invariance baselines on Rotated MNIST, achieving state-of-the-art results among reported methods.
• On VLCS and PACS, DIRT-GAN significantly improves over CIDG, CIDDG, and DGER baselines with the same backbones.
• Visualization shows better alignment of marginal and conditional distributions in the learned representation compared to naive training.
• The approach maintains competitive test-time efficiency since the StarGAN component is used only during training.
• Across datasets, DIRT-GAN demonstrates robust domain generalization without requiring uniform label distributions across domains.