[Paper Review] U-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation

Junho Kim, Minjae Kim+2 more|arXiv (Cornell University)|Jul 25, 2019

Generative Adversarial Networks and Image SynthesisComputer Science42 references300 citations

TL;DR

Introduces an unsupervised image-to-image translation model with a new attention module and AdaLIN normalization, enabling both shape- and texture-changing translations without altering architecture or hyper-parameters.

ABSTRACT

We propose a novel method for unsupervised image-to-image translation, which incorporates a new attention module and a new learnable normalization function in an end-to-end manner. The attention module guides our model to focus on more important regions distinguishing between source and target domains based on the attention map obtained by the auxiliary classifier. Unlike previous attention-based method which cannot handle the geometric changes between domains, our model can translate both images requiring holistic changes and images requiring large shape changes. Moreover, our new AdaLIN (Adaptive Layer-Instance Normalization) function helps our attention-guided model to flexibly control the amount of change in shape and texture by learned parameters depending on datasets. Experimental results show the superiority of the proposed method compared to the existing state-of-the-art models with a fixed network architecture and hyper-parameters. Our code and datasets are available at https://github.com/taki0112/UGATIT or https://github.com/znxlwm/UGATIT-pytorch.

Motivation & Objective

• Motivate robust unsupervised image-to-image translation that handles both holistic shape changes and texture/style changes without dataset-specific tuning.
• Introduce an attention mechanism that guides generation and discrimination based on domain-discriminative regions.
• Develop AdaLIN, a learnable normalization that balances instance and layer normalization to adapt to dataset characteristics.
• Demonstrate improved translation quality across diverse datasets and provide ablations to justify design choices.

Proposed method

• Propose a two-generator two-discriminator framework with shared attention modules guided by auxiliary classifiers.
• In the generator, compute an attention map a_s(x) from encoder features via CAM-inspired weights, then translate via G_t(a_s(x)).
• In residual blocks, use AdaLIN to dynamically mix IN and LN with gamma/beta learned from the attention map.
• Discriminator uses attention a_Dt(x) to focus on real-vs-fake differences in the target domain.
• Optimize with a combination of LSGAN adversarial loss, cycle-consistency loss, identity loss, and CAM losses to guide attention and translation.
• Ground training with unpaired data from X_s and X_t and enforce cross-domain alignment via CAM-based supervision.

Experimental results

Research questions

• RQ1 Can an attention-guided generator and discriminator improve unsupervised image-to-image translation across domains with varying geometry?
• RQ2 Does AdaLIN provide flexible control over shape vs. texture changes without changing network architecture or hyper-parameters?
• RQ3 Do attention maps from auxiliary classifiers help localize meaningful translation regions and stabilize training?
• RQ4 How does U-GAT-IT perform relative to state-of-the-art unsupervised translation methods across diverse datasets?
• RQ5 Is a fixed architecture able to handle both large geometric changes (e.g., selfie2anime) and texture/style transfers (e.g., photo2vangogh) without dataset-specific tuning?

Key findings

• The proposed attention modules guide both generator and discriminator to focus on semantically important regions, improving translation quality across datasets.
• AdaLIN enables flexible control over content preservation and style change by adapting IN/LN balance per layer, improving results for both shape-heavy and texture-heavy translations.
• U-GAT-IT with CAM and AdaLIN outperforms several baselines (CycleGAN, UNIT, MUNIT, DRIT, AGGAN) on multiple datasets in qualitative and quantitative assessments.
• Kernel Inception Distance (KID) ablations show the lowest scores when both attention and AdaLIN are used, and remain competitive when used separately.
• User studies indicate higher perceptual preferences for U-GAT-IT translations, especially on datasets requiring significant shape changes (e.g., selfie2anime).
• The model achieves strong performance with a fixed architecture and hyper-parameters across five diverse datasets, illustrating robustness and transferability.