[Paper Review] Self-Attention Generative Adversarial Networks

Han Zhang, Ian Goodfellow+2 more|arXiv (Cornell University)|May 21, 2018

Generative Adversarial Networks and Image Synthesis2,202 citations

TL;DR

Introduces self-attention into GANs to model long-range dependencies in images, stabilizes training with spectral normalization and TTUR, and achieves state-of-the-art ImageNet class-conditional generation.

ABSTRACT

In this paper, we propose the Self-Attention Generative Adversarial Network (SAGAN) which allows attention-driven, long-range dependency modeling for image generation tasks. Traditional convolutional GANs generate high-resolution details as a function of only spatially local points in lower-resolution feature maps. In SAGAN, details can be generated using cues from all feature locations. Moreover, the discriminator can check that highly detailed features in distant portions of the image are consistent with each other. Furthermore, recent work has shown that generator conditioning affects GAN performance. Leveraging this insight, we apply spectral normalization to the GAN generator and find that this improves training dynamics. The proposed SAGAN achieves the state-of-the-art results, boosting the best published Inception score from 36.8 to 52.52 and reducing Frechet Inception distance from 27.62 to 18.65 on the challenging ImageNet dataset. Visualization of the attention layers shows that the generator leverages neighborhoods that correspond to object shapes rather than local regions of fixed shape.

Motivation & Objective

• Motivated by the limitation of convolutional GANs in capturing long-range dependencies and global structure in complex images.
• Propose a self-attention mechanism to enable global interactions across feature maps in both generator and discriminator.
• Stabilize GAN training with spectral normalization on both generator and discriminator and employ two-timescale update rules (TTUR).
• Evaluate SAGAN on ImageNet to demonstrate improved generation quality and distributional similarity over prior GANs.

Proposed method

• Incorporate a self-attention module that computes a weighted sum of features across all spatial locations to model long-range dependencies.
• Apply 1x1 convolutions to generate f, g, h spaces for attention and compute attention weights via s_ij = f(x_i)^T g(x_j), with output y_i = gamma o_i + x_i where o_i aggregates attended features.
• Place attention modules in generator and discriminator to enforce global coherence in generated images and global consistency in real-vs-fake assessments.
• Use spectral normalization on both generator and discriminator to stabilize Lipschitz constants and training dynamics.
• Adopt TTUR with imbalanced learning rates for generator and discriminator to improve convergence when regularized discriminators are used.
• Train with hinge loss adversarial objectives and conditional batch normalization in the generator and projection in the discriminator as conditioning mechanisms.

Experimental results

Research questions

• RQ1 Does integrating self-attention into GAN architectures improve modeling of long-range dependencies and global image structure?
• RQ2 How does applying spectral normalization to the generator, in addition to the discriminator, affect GAN training stability and sample quality?
• RQ3 Is TTUR beneficial for GAN training when using regularized discriminators?

Key findings

• Self-attention improves image synthesis quality, with SAGAN achieving higher Inception Scores and lower FID than baselines.
• Attentional connections at middle-to-high level feature maps (e.g., 32x32 and 64x64) yield better performance than attention at very low-resolution maps.
• Self-attention blocks outperform equivalent residual blocks in modeling long-range dependencies, especially for complex geometric structures.
• SAGAN with self-attention and stabilization techniques achieved an Inception Score of 52.52 and FID of 18.65 on ImageNet, surpassing prior work.
• Visualization shows attention focuses on semantically coherent object parts rather than just spatial proximity, enabling structures like legs on dogs to be modeled.