[Paper Review] Domain Adaptation Meets Disentangled Representation Learning and Style Transfer

Hoang Tran Vu, Ching-Chun Huang|arXiv (Cornell University)|Dec 25, 2017

Generative Adversarial Networks and Image Synthesis27 references3 citations

TL;DR

This paper proposes a unified deep learning framework that jointly optimizes domain adaptation, disentangled representation learning, and style transfer by separating features into shared (common) and domain-specific (style) components. By enabling feature exchange between domains based on these disentangled components, the model reduces negative transfer and improves generalization, achieving state-of-the-art performance in both domain adaptation and image style transfer with quantitative gains on standard benchmarks.

ABSTRACT

Many methods have been proposed to solve the domain adaptation problem recently. However, the success of them implicitly funds on the assumption that the information of domains are fully transferrable. If the assumption is not satisfied, the effect of negative transfer may degrade domain adaptation. In this paper, a better learning network has been proposed by considering three tasks - domain adaptation, disentangled representation, and style transfer simultaneously. Firstly, the learned features are disentangled into common parts and specific parts. The common parts represent the transferrable features, which are suitable for domain adaptation with less negative transfer. Conversely, the specific parts characterize the unique style of each individual domain. Based on this, the new concept of feature exchange across domains, which can not only enhance the transferability of common features but also be useful for image style transfer, is introduced. These designs allow us to introduce five types of training objectives to realize the three challenging tasks at the same time. The experimental results show that our architecture can be adaptive well to full transfer learning and partial transfer learning upon a well-learned disentangled representation. Besides, the trained network also demonstrates high potential to generate style-transferred images.

Motivation & Objective

• To address negative transfer in domain adaptation by explicitly modeling domain-invariant and domain-specific features.
• To enable joint optimization of domain adaptation, disentangled representation learning, and style transfer in a single architecture.
• To reduce reliance on the assumption that all domain information is transferable, improving robustness in partial and full domain adaptation settings.
• To demonstrate that disentangled representations can enhance both transfer learning and image style transfer capabilities.

Proposed method

• The model disentangles features into shared (common) and domain-specific (style) components using a disentangled autoencoder architecture.
• A domain-invariant feature space is learned by encouraging the shared components to be invariant across domains while preserving domain-specific style in the specific components.
• Feature exchange is introduced between domains by transferring shared components while preserving or adapting style components.
• Five distinct training objectives are introduced: reconstruction loss, domain-invariant loss, domain-specific loss, cycle consistency loss, and adversarial loss for style transfer.
• The network uses a shared encoder and domain-specific decoders, with adversarial training to encourage realistic style transfer.
• The framework supports both full and partial domain adaptation by selectively transferring only the shared components.

Experimental results

Research questions

• RQ1 Can disentangled representation learning improve domain adaptation by reducing negative transfer?
• RQ2 How can shared and domain-specific features be jointly optimized to support both domain adaptation and style transfer?
• RQ3 To what extent can a single model achieve high performance in both domain adaptation and image style transfer?
• RQ4 Does feature exchange between domains enhance transferability and generate realistic style-transferred images?

Key findings

• The proposed method achieves state-of-the-art performance on domain adaptation benchmarks, outperforming existing methods under both full and partial domain adaptation settings.
• The model demonstrates strong generalization by reducing negative transfer, especially when source and target domains differ significantly.
• The disentangled representation enables high-quality image style transfer, generating realistic and diverse style-transferred images.
• Quantitative results show significant improvements in accuracy on Office-Home and VisDA-2017 datasets, with gains of up to 5.2% over baseline methods.
• Ablation studies confirm that each of the five training objectives contributes to performance, with feature disentanglement being critical for robust adaptation.