[Paper Review] Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications

Ming-Yu Liu, Xun Huang+3 more|arXiv (Cornell University)|Aug 6, 2020

Advanced Image Processing Techniques250 references33 citations

TL;DR

This paper provides a comprehensive overview of GANs for image and video synthesis, detailing training stabilization techniques, architectures, and a range of applications from image translation to neural rendering.

ABSTRACT

The generative adversarial network (GAN) framework has emerged as a powerful tool for various image and video synthesis tasks, allowing the synthesis of visual content in an unconditional or input-conditional manner. It has enabled the generation of high-resolution photorealistic images and videos, a task that was challenging or impossible with prior methods. It has also led to the creation of many new applications in content creation. In this paper, we provide an overview of GANs with a special focus on algorithms and applications for visual synthesis. We cover several important techniques to stabilize GAN training, which has a reputation for being notoriously difficult. We also discuss its applications to image translation, image processing, video synthesis, and neural rendering.

Motivation & Objective

• Motivate the use of GANs for flexible, data-driven image and video synthesis across unconditional and conditional setups.
• Survey stabilization techniques and training dynamics to address common GAN failures.
• Summarize architectural evolutions in generators and discriminators to improve quality and controllability.
• Review GAN-based applications including image translation, image processing, video synthesis, and neural rendering.

Proposed method

• Describe and compare GAN loss functions and objective formulations used for discriminator and generator updates.
• Explain training schemes (simultaneous vs alternating) and optimization choices (e.g., TTUR, ADAM, RMSProp).
• Survey regularization techniques (gradient penalty, spectral normalization, feature matching, perceptual loss, model averaging).
• Trace the evolution of generator and discriminator architectures, including conditional and projection-based discriminators and conditional activations.
• Discuss image translation frameworks (supervised and unsupervised) and multimodal/diverse translation approaches (e.g., CycleGAN, UNIT, MUNIT, StarGAN, SPADE).

Experimental results

Research questions

• RQ1 What are the principal GAN losses and training dynamics that yield stable image and video synthesis?
• RQ2 How have generator and discriminator architectures evolved to improve quality, controllability, and efficiency?
• RQ3 What are the state-of-the-art approaches for image-to-image translation in supervised and unsupervised settings, including multimodal outputs?
• RQ4 How can GANs be applied across image processing and neural rendering tasks, and what are their limitations?
• RQ5 How do regularization techniques and training schemes impact the stability and performance of GANs in practice?

Key findings

• GANs provide a flexible, data-driven objective for generating realistic images and videos in unconditional and conditional settings.
• A range of stabilization techniques (gradient penalty, spectral normalization, feature matching, perceptual loss, model averaging) improve training reliability and output quality.
• Conditional architectures (conditional normalization, conditional convolutions, projection discriminators) enhance controllability and performance on labeled or paired data.
• Image-to-image translation has progressed from supervised (pix2pix) to unsupervised (CycleGAN, UNIT, StarGAN) with multimodal and example-guided variants (MUNIT, FUNIT, SPADE).
• There is a clear architectural evolution from MLPs to deep convolutional/residual generators and discriminators, with ongoing improvements driven by advances in normalization and conditioning mechanisms.