[Paper Review] TransGAN: Two Transformers Can Make One Strong GAN

Yifan Jiang, Shiyu Chang+1 more|arXiv (Cornell University)|Feb 14, 2021

Generative Adversarial Networks and Image SynthesisComputer Science34 references151 citations

TL;DR

TransGAN proposes a convolution-free GAN architecture built entirely from transformers, using a memory-efficient, progressive resolution generator and a patch-level transformer discriminator. It achieves state-of-the-art performance with an IS of 10.10 and FID of 25.32 on STL-10, outperforming convolutional GANs on high-resolution image generation tasks.

ABSTRACT

The recent explosive interest on transformers has suggested their potential to become powerful universal models for computer vision tasks, such as classification, detection, and segmentation. However, how further transformers can go - are they ready to take some more notoriously difficult vision tasks, e.g., generative adversarial networks (GANs)? Driven by that curiosity, we conduct the first pilot study in building a GAN extbf{completely free of convolutions}, using only pure transformer-based architectures. Our vanilla GAN architecture, dubbed extbf{TransGAN}, consists of a memory-friendly transformer-based generator that progressively increases feature resolution while decreasing embedding dimension, and a patch-level discriminator that is also transformer-based. We then demonstrate TransGAN to notably benefit from data augmentations (more than standard GANs), a multi-task co-training strategy for the generator, and a locally initialized self-attention that emphasizes the neighborhood smoothness of natural images. Equipped with those findings, TransGAN can effectively scale up with bigger models and high-resolution image datasets. Specifically, our best architecture achieves highly competitive performance compared to current state-of-the-art GANs based on convolutional backbones. Specifically, TransGAN sets extbf{new state-of-the-art} IS score of 10.10 and FID score of 25.32 on STL-10. It also reaches competitive 8.64 IS score and 11.89 FID score on Cifar-10, and 12.23 FID score on CelebA $64 imes64$, respectively. We also conclude with a discussion of the current limitations and future potential of TransGAN. The code is available at \url{this https URL}.

Motivation & Objective

• To investigate whether transformers alone can replace convolutional networks in generative adversarial networks.
• To explore the feasibility of building a GAN architecture that is completely free of convolutions, relying solely on self-attention mechanisms.
• To improve image generation quality by leveraging data augmentations, multi-task co-training, and locally initialized self-attention in a transformer-based framework.
• To scale the model effectively on high-resolution datasets while maintaining memory efficiency and performance.

Proposed method

• Designing a memory-friendly, progressive resolution generator based on transformers that gradually increases feature map size while reducing embedding dimension.
• Implementing a patch-level discriminator using a transformer architecture to assess local image realism at the patch level.
• Applying data augmentations more effectively than standard GANs to improve training stability and generalization.
• Introducing a multi-task co-training strategy for the generator to enhance feature learning and diversity.
• Incorporating locally initialized self-attention to emphasize spatial smoothness and preserve local image structure.
• Scaling the model architecture to handle larger datasets and higher resolutions without performance degradation.

Experimental results

Research questions

• RQ1 Can a GAN be constructed entirely from transformers without any convolutional layers?
• RQ2 How do data augmentations affect training dynamics and performance in a pure transformer-based GAN?
• RQ3 Can multi-task co-training and locally initialized self-attention improve the quality and diversity of generated images in a transformer-based generator?
• RQ4 What is the performance ceiling of a convolution-free GAN when scaled to high-resolution datasets?
• RQ5 How does the proposed TransGAN compare to state-of-the-art GANs with convolutional backbones in terms of FID and IS scores?

Key findings

• TransGAN achieves a new state-of-the-art Inception Score (IS) of 10.10 and Fréchet Inception Distance (FID) of 25.32 on the STL-10 dataset.
• On Cifar-10, TransGAN attains an IS score of 8.64 and FID of 11.89, demonstrating competitive performance against convolutional GANs.
• For CelebA at 64×64 resolution, TransGAN achieves a FID score of 12.23, indicating strong generation quality on face image synthesis.
• The model benefits significantly from data augmentations, showing greater improvement than standard GANs when such techniques are applied.
• The combination of multi-task co-training and locally initialized self-attention leads to improved feature learning and image realism.
• TransGAN scales effectively with larger models and high-resolution datasets, maintaining strong performance without relying on convolutional inductive biases.