[Paper Review] MC-GAN: Multi-conditional Generative Adversarial Network for Image Synthesis

Hyojin Park, Youngjoon Yoo+1 more|arXiv (Cornell University)|May 3, 2018

Generative Adversarial Networks and Image Synthesis10 references47 citations

TL;DR

MC-GAN generates a foreground object described by text on a specified location while preserving the background from a base image, using a novel synthesis block to disentangle foreground and background.

ABSTRACT

In this paper, we introduce a new method for generating an object image from text attributes on a desired location, when the base image is given. One step further to the existing studies on text-to-image generation mainly focusing on the object's appearance, the proposed method aims to generate an object image preserving the given background information, which is the first attempt in this field. To tackle the problem, we propose a multi-conditional GAN (MC-GAN) which controls both the object and background information jointly. As a core component of MC-GAN, we propose a synthesis block which disentangles the object and background information in the training stage. This block enables MC-GAN to generate a realistic object image with the desired background by controlling the amount of the background information from the given base image using the foreground information from the text attributes. From the experiments with Caltech-200 bird and Oxford-102 flower datasets, we show that our model is able to generate photo-realistic images with a resolution of 128 x 128. The source code of MC-GAN is released.

Motivation & Objective

• Define a novel multi-modal conditional synthesis problem using base image, text, and location.
• Develop MC-GAN with synthesis blocks that disentangle foreground and background during training.
• Demonstrate photo-realistic object generation on a given background across bird and flower datasets.

Proposed method

• Encode text into a embedding and combine with a noise vector to form a seed feature map.
• Use synthesis blocks that gate background information with foreground features to generate image and segmentation mask.
• Train a multi-input discriminator that evaluates image, mask, and text with real/fake and matching/mismatching conditions.
• Apply conditioning augmentation and a background reconstruction loss to stabilize training and preserve background.
• Optionally extend with a StackGAN-style two-stage generator for higher resolution 128×128 results.

Experimental results

Research questions

• RQ1 Can a GAN generate a text-described object at a specified location while preserving the background from a base image?
• RQ2 Does a synthesis-block architecture effectively disentangle foreground and background in multi-modal image synthesis?
• RQ3 How does MC-GAN compare to prior multi-modal baselines in maintaining background integrity while adding foreground objects?
• RQ4 What is the impact of conditioning augmentation and background reconstruction losses on training stability and output quality?

Key findings

• MC-GAN can generate 128×128 images with a foreground object described by text on a given background.
• The synthesis block gates background information using a switch mechanism to prevent object-background overlap and preserve realism.
• MC-GAN maintains background details even when the base image lacks similar objects to the target object.
• Compared to the baseline multi-modal method, MC-GAN better preserves background information and yields plausible object textures and poses.
• The model supports interpolation in text and noise spaces to produce diverse, gradual changes in generated images.