[Paper Review] Unsupervised Attention-guided Image to Image Translation

Youssef Alami Mejjati, Christian Richardt+3 more|arXiv (Cornell University)|Jun 6, 2018

Multimodal Machine Learning ApplicationsComputer Science201 citations

TL;DR

This paper introduces unsupervised attention mechanisms integrated into CycleGAN-like image-to-image translation to focus translation on foreground objects while preserving background, improving realism without supervision.

ABSTRACT

Current unsupervised image-to-image translation techniques struggle to focus their attention on individual objects without altering the background or the way multiple objects interact within a scene. Motivated by the important role of attention in human perception, we tackle this limitation by introducing unsupervised attention mechanisms that are jointly adversarialy trained with the generators and discriminators. We demonstrate qualitatively and quantitatively that our approach is able to attend to relevant regions in the image without requiring supervision, and that by doing so it achieves more realistic mappings compared to recent approaches.

Motivation & Objective

• Motivate improving unsupervised image-to-image translation by focusing on relevant scene regions.
• Introduce trainable attention networks that guide generators to translate only foreground objects.
• jointly train attention networks with generators and discriminators to alter only discriminative regions.
• Maintain background content to avoid unrealistic background alterations.
• Demonstrate both qualitative and quantitative improvements over existing methods.

Proposed method

• Add attention networks A_S and A_T to the CycleGAN framework.
• Compute attention maps and apply them as masks to combine foreground-translated regions with preserved background (s' = s_a ⊙ F(s) + (1 - s_a) ⊙ s).
• Train using adversarial losses plus cycle-consistency losses, with a learning objective that enforces attended-region translation.
• Adopt a discriminator working on masked regions to prevent background painting and stabilize training (with a staged training schedule).
• Use a continuous attention mask during training and thresholded masks for the discriminator to avoid mode collapse.
• Provide an algorithmic training procedure that alternates updates for generators, attentions, and discriminators.

Experimental results

Research questions

• RQ1 Can unsupervised attention mechanisms learn to focus translation on meaningful regions without supervision?
• RQ2 Does attention-guided translation improve image realism and preserve backgrounds compared to existing unsupervised I2I methods?
• RQ3 What is the impact of training schedule and discriminator masking on learning stable attention and high-quality translations?
• RQ4 How does the method perform across datasets with multi-scale objects and diverse backgrounds?

Key findings

• Ours achieves the lowest Kernel Inception Distance (KID) across mappings, indicating closer visual similarity to real images.
• Qualitative results show attention maps focusing on domain-descriptive objects and avoiding background changes.
• Ablations show the importance of cycle-consistency and dual attention (A_S and A_T) for sharp, localized attention.
• Using attention-guided discriminators and staged training reduces mode collapse and background artifacts.
• Compared to baselines (CycleGAN, DiscoGAN, DualGAN, UNIT, RA), the proposed method yields more realistic translations, especially with multi-scale objects.