[Paper Review] A Survey and Taxonomy of Adversarial Neural Networks for Text-to-Image Synthesis

Jorge Agnese, Jonathan Herrera+2 more|arXiv (Cornell University)|Oct 21, 2019

Generative Adversarial Networks and Image Synthesis70 references9 citations

TL;DR

This paper presents a comprehensive survey and taxonomy of generative adversarial networks (GANs) for text-to-image synthesis, categorizing state-of-the-art models into four groups: semantic enhancement, resolution enhancement, diversity enhancement, and motion enhancement GANs. It reviews key architectures like StackGAN, AttnGAN, and HDGAN, demonstrating their ability to generate high-fidelity, photo-realistic images from natural language descriptions with improved semantic alignment and visual quality.

ABSTRACT

Text-to-image synthesis refers to computational methods which translate human written textual descriptions, in the form of keywords or sentences, into images with similar semantic meaning to the text. In earlier research, image synthesis relied mainly on word to image correlation analysis combined with supervised methods to find best alignment of the visual content matching to the text. Recent progress in deep learning (DL) has brought a new set of unsupervised deep learning methods, particularly deep generative models which are able to generate realistic visual images using suitably trained neural network models. In this paper, we review the most recent development in the text-to-image synthesis research domain. Our survey first introduces image synthesis and its challenges, and then reviews key concepts such as generative adversarial networks (GANs) and deep convolutional encoder-decoder neural networks (DCNN). After that, we propose a taxonomy to summarize GAN based text-to-image synthesis into four major categories: Semantic Enhancement GANs, Resolution Enhancement GANs, Diversity Enhancement GANS, and Motion Enhancement GANs. We elaborate the main objective of each group, and further review typical GAN architectures in each group. The taxonomy and the review outline the techniques and the evolution of different approaches, and eventually provide a clear roadmap to summarize the list of contemporaneous solutions that utilize GANs and DCNNs to generate enthralling results in categories such as human faces, birds, flowers, room interiors, object reconstruction from edge maps (games) etc. The survey will conclude with a comparison of the proposed solutions, challenges that remain unresolved, and future developments in the text-to-image synthesis domain.

Motivation & Objective

• To provide a systematic review of recent advancements in text-to-image synthesis using adversarial neural networks.
• To identify and categorize key challenges in generating photo-realistic images from natural language descriptions.
• To propose a novel taxonomy organizing GAN-based text-to-image synthesis into four distinct categories: semantic, resolution, diversity, and motion enhancement.
• To compare state-of-the-art models in terms of architecture, design, and performance on benchmark datasets.
• To outline open challenges and future research directions in the field of text-to-image generation using deep generative models.

Proposed method

• Proposes a taxonomy of GAN-based text-to-image synthesis into four categories: semantic enhancement, resolution enhancement, diversity enhancement, and motion enhancement GANs.
• Reviews key deep learning components including generative adversarial networks (GANs), deep convolutional encoder-decoder networks (DCNNs), and attention mechanisms.
• Analyzes representative models such as StackGAN, StackGAN++, AttnGAN, HDGAN, and TAC-GAN, focusing on their architectural innovations and training strategies.
• Utilizes feature embedding functions (e.g., φ()) to map text descriptions into latent vectors for conditional image generation.
• Employs evaluation metrics including Inception Score (IS) and Fréchet Inception Distance (FID) to compare model performance on datasets like CUB, LSUN, and COCO.
• Compares models using qualitative examples and quantitative scores across diverse image categories including birds, flowers, human faces, and indoor scenes.

Experimental results

Research questions

• RQ1 How do different GAN architectures improve semantic alignment between text descriptions and generated images?
• RQ2 What architectural innovations enable higher resolution and sharper image generation in text-to-image synthesis?
• RQ3 How do diversity-enhancing techniques prevent mode collapse and improve sample variety?
• RQ4 What role do attention mechanisms and multi-scale feature learning play in improving generation quality?
• RQ5 What are the key limitations and unresolved challenges in generating photo-realistic images from natural language descriptions?

Key findings

• StackGAN++ achieved an Inception Score of 32.64 on the CUB dataset, significantly outperforming earlier models like StackGAN (IS: 23.98).
• AttnGAN achieved an Inception Score of 35.49 on the CUB dataset, demonstrating the effectiveness of attention mechanisms in aligning text and image features.
• HDGAN achieved an Inception Score of 31.06 on the CUB dataset and 12.04 on the LSUN-Bedrooms dataset, indicating strong performance in complex scene generation.
• The proposed taxonomy effectively organizes diverse GAN-based approaches by their primary design objectives, clarifying evolutionary relationships and technical motivations.
• Models such as AttnGAN and TAC-GAN show improved attention to specific text phrases, resulting in more accurate localization of visual attributes in generated images.
• Despite progress, challenges remain in generating high-resolution images beyond 256×256 pixels and in ensuring consistent color and structural consistency across generations.