[Paper Review] Transformer-based Generative Adversarial Networks in Computer Vision: A Comprehensive Survey

Shiv Ram Dubey, Satish Kumar Singh|arXiv (Cornell University)|Feb 17, 2023

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper presents a comprehensive survey of Transformer-based Generative Adversarial Networks (GANs) in computer vision, analyzing their architecture, training strategies, and performance across image and video synthesis tasks. It highlights the fusion of local inductive biases from CNNs and global modeling via self-attention in Transformers, demonstrating superior performance in image generation, translation, and video synthesis using metrics like FID, PSNR, and SSIM on benchmark datasets.

ABSTRACT

Generative Adversarial Networks (GANs) have been very successful for synthesizing the images in a given dataset. The artificially generated images by GANs are very realistic. The GANs have shown potential usability in several computer vision applications, including image generation, image-to-image translation, video synthesis, and others. Conventionally, the generator network is the backbone of GANs, which generates the samples and the discriminator network is used to facilitate the training of the generator network. The discriminator network is usually a Convolutional Neural Network (CNN). Whereas, the generator network is usually either an Up-CNN for image generation or an Encoder-Decoder network for image-to-image translation. The convolution-based networks exploit the local relationship in a layer, which requires the deep networks to extract the abstract features. Hence, CNNs suffer to exploit the global relationship in the feature space. However, recently developed Transformer networks are able to exploit the global relationship at every layer. The Transformer networks have shown tremendous performance improvement for several problems in computer vision. Motivated from the success of Transformer networks and GANs, recent works have tried to exploit the Transformers in GAN framework for the image/video synthesis. This paper presents a comprehensive survey on the developments and advancements in GANs utilizing the Transformer networks for computer vision applications. The performance comparison for several applications on benchmark datasets is also performed and analyzed. The conducted survey will be very useful to deep learning and computer vision community to understand the research trends \& gaps related with Transformer-based GANs and to develop the advanced GAN architectures by exploiting the global and local relationships for different applications.

Motivation & Objective

• To provide the first comprehensive survey on Transformer-based GANs in computer vision, as no such survey exists in the literature.
• To categorize and analyze Transformer-based GAN models across key applications, including image generation, image-to-image translation, video synthesis, and restoration tasks.
• To compare state-of-the-art models using standard metrics such as FID, PSNR, and SSIM on benchmark datasets.
• To identify research trends, open challenges, and future directions in combining Transformers with GANs for improved visual synthesis.
• To guide researchers in designing advanced GAN architectures that effectively exploit both local and global feature representations.

Proposed method

• The survey systematically classifies Transformer-based GANs based on generator and discriminator architectures, including Vision Transformers, Swin Transformers, and hybrid CNN-Transformer designs.
• It analyzes objective functions, including adversarial loss, perceptual loss, cycle-consistency loss, and masked autoencoding, to evaluate training stability and synthesis quality.
• The paper evaluates models on diverse benchmarks such as CIFAR-10, CelebA, Cityscapes, and LSUN for image and video tasks.
• It compares performance using metrics like Fréchet Inception Distance (FID), Peak Signal-to-Noise Ratio (PSNR), and Structural Similarity Index (SSIM).
• The analysis includes architectural innovations such as cross-attention, multi-head self-attention, and hybrid encoders with CNNs and Transformers.
• The survey also discusses training techniques like progressive growing, masking-based pre-training, and self-supervised learning for improved generalization.

Experimental results

Research questions

• RQ1 How do Transformer-based GANs improve upon traditional CNN-based GANs in modeling long-range dependencies in image and video synthesis?
• RQ2 What are the key architectural components—such as self-attention, cross-attention, or hybrid CNN-Transformer blocks—that enable state-of-the-art performance in image and video generation?
• RQ3 How do different loss functions, including adversarial, perceptual, and cycle-consistency losses, affect the quality and stability of generated images and videos?
• RQ4 What are the current limitations and open challenges in training and deploying Transformer-based GANs, particularly in terms of efficiency, generalization, and hyperparameter sensitivity?
• RQ5 What future research directions can further enhance the fusion of local and global representations in GANs using Transformers?

Key findings

• Transformer-based GANs outperform conventional CNN-based GANs in modeling global dependencies, leading to higher-quality image and video synthesis.
• Models using Vision Transformer (ViT) and Swin Transformer in the generator achieve state-of-the-art FID scores on image generation benchmarks such as CIFAR-10 and CelebA.
• The combination of CNN-based encoders with Transformer-based decoders in U-shaped architectures achieves strong performance in image-to-image translation and image restoration tasks.
• Hybrid architectures that integrate local inductive biases from CNNs with global modeling from Transformers show improved stability and perceptual quality in image generation and super-resolution.
• The use of cross-attention and masked token prediction in the generator improves feature alignment and reconstruction fidelity in tasks like image inpainting and video synthesis.
• Despite progress, training instability and high computational cost remain challenges, especially in video generation and high-resolution synthesis, indicating room for improved loss functions and training strategies.