[Paper Review] RetrieveGAN: Image Synthesis via Differentiable Patch Retrieval

Hung-Yu Tseng, Hsin-Ying Lee+3 more|arXiv (Cornell University)|Jul 16, 2020

Generative Adversarial Networks and Image Synthesis43 references4 citations

TL;DR

RetrieveGAN proposes a differentiable patch retrieval module for conditional image synthesis from scene descriptions, enabling end-to-end training and iterative selection of mutually compatible image patches. By integrating Gumbel-softmax differentiability and a co-occurrence loss, the method generates more realistic and diverse images with semantically coherent, compatible patches, outperforming non-differentiable baselines in both quantitative metrics and user studies.

ABSTRACT

Image generation from scene description is a cornerstone technique for the controlled generation, which is beneficial to applications such as content creation and image editing. In this work, we aim to synthesize images from scene description with retrieved patches as reference. We propose a differentiable retrieval module. With the differentiable retrieval module, we can (1) make the entire pipeline end-to-end trainable, enabling the learning of better feature embedding for retrieval; (2) encourage the selection of mutually compatible patches with additional objective functions. We conduct extensive quantitative and qualitative experiments to demonstrate that the proposed method can generate realistic and diverse images, where the retrieved patches are reasonable and mutually compatible.

Motivation & Objective

• To address the limitations of non-differentiable retrieval in semi-parametric image generation, where pre-defined embeddings are decoupled from the generation process.
• To improve mutual compatibility among retrieved image patches by modeling their co-occurrence relationships during retrieval.
• To enable end-to-end training of the retrieval and generation pipeline by making the retrieval process differentiable.
• To enhance image synthesis quality by jointly optimizing patch embeddings and generation based on scene graph inputs.
• To validate that differentiable, iterative retrieval leads to more realistic and diverse image generation with semantically coherent patches.

Proposed method

• Introduces a differentiable retrieval module using the Gumbel-softmax trick to allow backpropagation through the discrete retrieval operation.
• Employs an iterative retrieval strategy that selects patches one at a time, conditioned on already selected patches to ensure compatibility.
• Designs a co-occurrence loss function that encourages selected patches to be mutually compatible by modeling their co-occurrence patterns in real images.
• Uses a ground-truth selection loss to guide the retrieval module toward selecting patches that are semantically relevant to the scene graph.
• Combines the retrieved patches with a conditional GAN-based image synthesis network to generate the final image.
• Trains the entire pipeline end-to-end, allowing the retrieval module to learn embedding functions that optimize for image generation quality.

Experimental results

Research questions

• RQ1 Can differentiable retrieval improve the quality and compatibility of image patches selected for conditional image synthesis?
• RQ2 How does iterative, context-aware patch retrieval compare to independent, non-differentiable retrieval in terms of image realism and semantic consistency?
• RQ3 To what extent does joint optimization of retrieval and generation improve image fidelity and diversity?
• RQ4 Can the proposed co-occurrence loss effectively model and enforce mutual compatibility among multiple objects in a scene?
• RQ5 Does the differentiable retrieval module lead to better generalization and performance on complex scene descriptions compared to non-differentiable baselines?

Key findings

• RetrieveGAN achieves state-of-the-art FID scores on both COCO-Stuff and Visual Genome datasets, indicating superior image realism compared to baselines like AttnGAN and PasteGAN.
• The method achieves higher Inception Score (IS) on COCO-Stuff, demonstrating improved diversity in generated images.
• User studies show that patches retrieved by RetrieveGAN are significantly more mutually compatible than those from PasteGAN, with a statistically significant preference for co-occurring, semantically coherent objects.
• Ablation studies confirm that both the ground-truth selection loss and co-occurrence loss contribute to improved image quality and patch compatibility.
• Qualitative results show that RetrieveGAN generates images with clearer, more realistic object appearances and better spatial consistency, especially in complex scenes with multiple interacting objects.
• The differentiable retrieval module enables end-to-end training, allowing the embedding space to be optimized jointly with the image generator, leading to better alignment between retrieved patches and the target image.