[论文解读] TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up

Yifan Jiang, Shiyu Chang+1 more|arXiv (Cornell University)|Feb 14, 2021

Generative Adversarial Networks and Image Synthesis参考文献 92被引用 263

一句话总结

TransGAN 构建了一个完全不使用卷积的 GAN，使用纯 Transformer，具备内存友好的生成器、多尺度判别器和网格自注意力，达到有竞争力的结果并实现高分辨率可扩展生成。

ABSTRACT

The recent explosive interest on transformers has suggested their potential to become powerful "universal" models for computer vision tasks, such as classification, detection, and segmentation. While those attempts mainly study the discriminative models, we explore transformers on some more notoriously difficult vision tasks, e.g., generative adversarial networks (GANs). Our goal is to conduct the first pilot study in building a GAN completely free of convolutions, using only pure transformer-based architectures. Our vanilla GAN architecture, dubbed TransGAN, consists of a memory-friendly transformer-based generator that progressively increases feature resolution, and correspondingly a multi-scale discriminator to capture simultaneously semantic contexts and low-level textures. On top of them, we introduce the new module of grid self-attention for alleviating the memory bottleneck further, in order to scale up TransGAN to high-resolution generation. We also develop a unique training recipe including a series of techniques that can mitigate the training instability issues of TransGAN, such as data augmentation, modified normalization, and relative position encoding. Our best architecture achieves highly competitive performance compared to current state-of-the-art GANs using convolutional backbones. Specifically, TransGAN sets new state-of-the-art inception score of 10.43 and FID of 18.28 on STL-10, outperforming StyleGAN-V2. When it comes to higher-resolution (e.g. 256 x 256) generation tasks, such as on CelebA-HQ and LSUN-Church, TransGAN continues to produce diverse visual examples with high fidelity and impressive texture details. In addition, we dive deep into the transformer-based generation models to understand how their behaviors differ from convolutional ones, by visualizing training dynamics. The code is available at https://github.com/VITA-Group/TransGAN.

研究动机与目标

• 激发使用纯 Transformer 架构的卷积-free GANs 来进行图像生成的探索。
• 设计一个在变换器基础的 GAN 中适合的内存高效生成器和多尺度判别器。
• 开发技术（网格自注意力、数据增强、修改后的归一化、相对位置信编码）以稳定训练并提高保真度。
• 在小型和大型数据集上评估 TransGAN，以评估其相对于基于 CNN 的 GAN 的性能和可扩展性。

提出的方法

• 使用一个内存友好的、分阶段的基于 Transformer 的生成器，逐步提高特征图分辨率。
• 实现一个多尺度判别器，处理不同尺寸的补丁以捕获全局上下文和局部纹理。
• 引入网格自注意力，以在高分辨率下降低内存负担，同时保留全局一致性。
• 应用包含强数据增强、修改后的归一化（逐-token 缩放）和相对位置编码的训练方案，以稳定训练。
• 扩展到高分辨率生成（例如 256×256），获得高质量视觉结果并进行消融研究。

实验结果

研究问题

• RQ1 是否可以在不使用任何卷积层的情况下，利用纯 Transformer 组件有效构建 GAN？
• RQ2 哪些架构与训练策略能够实现 Transformer 基 GAN 的稳定且高保真图像生成？
• RQ3 内存高效的注意力机制（如网格自注意力）在高分辨率下如何影响质量和可扩展性？
• RQ4 数据增强和相对位置编码如何影响 TransGAN 的训练稳定性和性能？

主要发现

• TransGAN 在 CIFAR-10、STL-10 和 CelebA 上，结合强数据增强，在与最先进 CNN 基 GAN 的定量结果上具有竞争力。
• 在 CIFAR-10 上，TransGAN 获得的 Inception Score 为 9.02，FID 为 9.26。
• 在 STL-10 上，达到 Inception Score 10.43，FID 18.28。
• 在 CelebA (128×128) 上，达到 FID 5.28，接近报道的最佳结果。
• TransGAN 能扩展到高分辨率生成（例如 256×256），并在 256×256 的 CelebA-HQ 与 LSUN Church 演示中显示出多样且高保真的输出。
• 消融研究表明网格自注意力和所提出的训练方案（数据增强、修改后的归一化、相对位置编码）显著提升了性能。