[Paper Review] Unsupervised Cross-Domain Image Generation

Yaniv Taigman, Adam Polyak+1 more|arXiv (Cornell University)|Nov 7, 2016

Generative Adversarial Networks and Image Synthesis18 references428 citations

TL;DR

The paper introduces Domain Transfer Network (DTN), an unsupervised method to map samples S to a target domain T by composing a learned g with a fixed f, using a multiclass GAN loss, f-constancy, and a self-identity regularizer, demonstrated on digits and faces to produce convincing cross-domain images while preserving identity.

ABSTRACT

We study the problem of transferring a sample in one domain to an analog sample in another domain. Given two related domains, S and T, we would like to learn a generative function G that maps an input sample from S to the domain T, such that the output of a given function f, which accepts inputs in either domains, would remain unchanged. Other than the function f, the training data is unsupervised and consist of a set of samples from each domain. The Domain Transfer Network (DTN) we present employs a compound loss function that includes a multiclass GAN loss, an f-constancy component, and a regularizing component that encourages G to map samples from T to themselves. We apply our method to visual domains including digits and face images and demonstrate its ability to generate convincing novel images of previously unseen entities, while preserving their identity.

Motivation & Objective

• Develop a method to transfer samples from a source domain S to a related target domain T without supervised pairs.
• Ensure the target-domain outputs preserve the functional identity encoded by a given network f.
• Promote an architectural design that composes a learnable g with a fixed representation f to focus on f-relevant features.
• Demonstrate the approach on digit and face-to-emoji translations and analyze its components and limitations.

Proposed method

• Use a two-part generator G = g ∘ f where f is a fixed feature extractor and g maps f(x) to the target domain representation.
• Train with a multiclass GAN loss that discriminates among real target samples and generated samples from both S and T.
• Add an f-constancy loss that minimizes ||f(x) − f(G(x))|| for x in S.
• Include a target-identity regularizer L_TID that encourages G to be the identity on samples from T.
• Incorporate a total variation loss L_TV for smoothing generated images.
• Optimize with an alternating scheme where D minimizes its loss and G minimizes its corresponding composite loss.

Experimental results

Research questions

• RQ1 Can unsupervised samples from S and T be mapped via a learned G so that a downstream function f yields invariant outputs: f(x) ≈ f(G(x))?
• RQ2 Does composing G with a fixed f and using a multiclass GAN loss improve cross-domain transfer over baseline setups?
• RQ3 How well does DTN perform on visually distant domain pairs (e.g., SVHN→MNIST, real faces→emoji)?
• RQ4 What is the impact of each loss term (f-constancy, L_TID, TV, GAN) on transfer quality?
• RQ5 Can DTN facilitate unsupervised domain adaptation and stylistic transformations (e.g., style transfer within the DTN framework)?

Key findings

• DTN achieves higher cross-domain transfer accuracy than a baseline where G operates directly on S, e.g., MNIST classifier accuracy on transferred SVHN→MNIST reaches 90.66% vs 13.71% baseline.
• DTN enables competitive or superior unsupervised domain adaptation; MNIST classifier accuracy on transferred SVHN test set reaches 79.72% (DTN) vs 73.85% (DANN) and 59.32% (SAFER baseline).
• Removing L_CONST or L_TID degrades performance; L_CONST is particularly crucial, with DTN w/o L_CONST dropping to 74.55% and without both L_CONST and L_TID dropping to 5.28% in digits experiments.
• When applying to faces, DTN generates emoji that are more discriminative than manually crafted ones, with retrieval median rank 16 vs 16? (manual median 16311) and rank-1 22.88% vs 0% for manual.
• DTN can perform style transfer as a special case of domain transfer, producing emoji-style outputs from photos, and can be used to transfer styles from a single image to faces.
• DTN demonstrates the ability to generate convincing novel samples in unseen entities while preserving identity information as captured by f.
• Unseen-class ablations show the model robustly avoids generating absent classes in the target domain when they are not present in s or f training data.