[Paper Review] InstaGAN: Instance-aware Image-to-Image Translation

Sangwoo Mo, Minsu Cho+1 more|arXiv (Cornell University)|Dec 28, 2018

Generative Adversarial Networks and Image Synthesis44 references99 citations

TL;DR

InstaGAN uses instance segmentation masks to perform unsupervised, multi-instance image-to-image translation with a context-preserving loss and sequential mini-batch training.

ABSTRACT

Unsupervised image-to-image translation has gained considerable attention due to the recent impressive progress based on generative adversarial networks (GANs). However, previous methods often fail in challenging cases, in particular, when an image has multiple target instances and a translation task involves significant changes in shape, e.g., translating pants to skirts in fashion images. To tackle the issues, we propose a novel method, coined instance-aware GAN (InstaGAN), that incorporates the instance information (e.g., object segmentation masks) and improves multi-instance transfiguration. The proposed method translates both an image and the corresponding set of instance attributes while maintaining the permutation invariance property of the instances. To this end, we introduce a context preserving loss that encourages the network to learn the identity function outside of target instances. We also propose a sequential mini-batch inference/training technique that handles multiple instances with a limited GPU memory and enhances the network to generalize better for multiple instances. Our comparative evaluation demonstrates the effectiveness of the proposed method on different image datasets, in particular, in the aforementioned challenging cases. Code and results are available in https://github.com/sangwoomo/instagan

Motivation & Objective

• Motivate image-to-image translation to handle multiple target instances and significant shape changes.
• Incorporate instance information (segmentation masks) to improve transfiguration fidelity.
• Ensure permutation invariance of instance order while translating corresponding attributes.
• Preserve background context while transforming target instances.
• Enable training and inference with many instances under limited GPU memory.

Proposed method

• Propose an architecture that translates both an image and the corresponding set of instance attributes, while being permutation-invariant to instance order.
• Encode image and instance attributes with joint feature representations where the set of attributes is aggregated via permutation-invariant operations (e.g., summation).
• Use a cycle-consistency based GAN framework to learn bidirectional mappings between domains (XY and YX).
• Introduce a context preserving loss that emphasizes transforming target instances and preserving background content.
• Develop a sequential mini-batch translation technique to handle many instances with limited GPU memory, providing data augmentation via intermediate samples.

Experimental results

Research questions

• RQ1 How can instance-level information (e.g., segmentation masks) be integrated into unsupervised image-to-image translation to handle multiple instances and shape changes?
• RQ2 Does a context-preserving loss improve background fidelity and instance-transformation quality in multi-instance scenarios?
• RQ3 Can sequential mini-batch translation enable training and inference with many instances under memory constraints while preserving permutation invariance?

Key findings

• Outperforms CycleGAN on multi-instance transfiguration tasks across CCP, MHP, and COCO datasets.
• Delivers better shape-aware translations and preserves layout and viewing direction of instances.
• Context preserving loss improves background preservation and translation quality, beyond just background retention.
• Sequential mini-batch translation enables training with many instances and acts as data augmentation for improved generalization.
• The method allows controlling which instances to translate by conditioning on input masks.