[Paper Review] Generating Long Sequences with Sparse Transformers

Rewon Child, Scott Gray+2 more|arXiv (Cornell University)|Apr 23, 2019

Topic Modeling25 references489 citations

TL;DR

The paper introduces sparse factorized self-attention (Sparse Transformers) to scale transformers to long sequences, achieving state-of-the-art density modeling on text, images, and audio, and enabling very long contexts with hundreds of layers.

ABSTRACT

Transformers are powerful sequence models, but require time and memory that grows quadratically with the sequence length. In this paper we introduce sparse factorizations of the attention matrix which reduce this to $O(n \sqrt{n})$. We also introduce a) a variation on architecture and initialization to train deeper networks, b) the recomputation of attention matrices to save memory, and c) fast attention kernels for training. We call networks with these changes Sparse Transformers, and show they can model sequences tens of thousands of timesteps long using hundreds of layers. We use the same architecture to model images, audio, and text from raw bytes, setting a new state of the art for density modeling of Enwik8, CIFAR-10, and ImageNet-64. We generate unconditional samples that demonstrate global coherence and great diversity, and show it is possible in principle to use self-attention to model sequences of length one million or more.

Motivation & Objective

• Motivate scalable autoregressive modeling for long sequences across text, images, and audio.
• Reduce memory and compute for attention from quadratic to near-linear via sparse factorization.
• Allow training of very deep Transformer-like models by architectural and optimization changes.
• Demonstrate state-of-the-art density modeling across multiple data modalities.

Proposed method

• Introduce factorized self-attention where each position attends to a sparse subset of previous positions.
• Explore two-dimensional factorized attention patterns: strided and fixed patterns with controllable locality and coverage.
• Incorporate Sparse Transformer blocks with pre-activation residuals and layer normalization to enable deep networks.
• Use recomputation of attention and feed-forward blocks during backpropagation to save memory.
• Implement efficient GPU kernels for mixed sparse attention, local windows, and block-wise computations.
• Train with mixed-precision, Adam with warmup, cosine learning rate decay, and gradient clipping.

Experimental results

Research questions

• RQ1 Can sparse, factorized attention match full attention on long sequences across text, images, and audio?
• RQ2 What sparse patterns (strided vs fixed) yield best performance for different data modalities?
• RQ3 How deep can Sparse Transformers be trained, and what memory/training techniques enable it?
• RQ4 What is the impact of these patterns on density modeling benchmarks and sample quality?

Key findings

• Sparse Transformers achieve comparable or better density modeling performance than dense attention across CIFAR-10, Enwik8, ImageNet-64, and music data.
• Strided and fixed sparse patterns provide substantial speedups over dense attention and, in some cases, better compression (lower bits per byte).
• Models with hundreds of layers can be trained by architectural changes and gradient recomputation, enabling long-context modeling.
• On CIFAR-10, strided sparse attention reaches 2.80–2.82 bits per dim, beating prior state-of-the-art (2.85).
• On Enwik8, Sparse Transformer with fixed attention reaches 0.99 bits per dim, matching or improving Transformer-XL with more parameters.
• On ImageNet-64, the strided sparse transformer achieves 3.44 bits per dim, better than several prior generative models.