[Paper Review] General-purpose, long-context autoregressive modeling with Perceiver AR

Curtis Hawthorne, Andrew Jaegle+13 more|arXiv (Cornell University)|Feb 15, 2022

Generative Adversarial Networks and Image SynthesisComputer Science34 citations

TL;DR

Perceiver AR introduces an autoregressive, modality-agnostic architecture that uses cross-attention to compress very long inputs into a small latent set and performs deep self-attention in latent space, enabling genuine long-context density estimation beyond standard Transformers. It achieves state-of-the-art or competitive results on long-context benchmarks across images, books, and music.

ABSTRACT

Real-world data is high-dimensional: a book, image, or musical performance can easily contain hundreds of thousands of elements even after compression. However, the most commonly used autoregressive models, Transformers, are prohibitively expensive to scale to the number of inputs and layers needed to capture this long-range structure. We develop Perceiver AR, an autoregressive, modality-agnostic architecture which uses cross-attention to map long-range inputs to a small number of latents while also maintaining end-to-end causal masking. Perceiver AR can directly attend to over a hundred thousand tokens, enabling practical long-context density estimation without the need for hand-crafted sparsity patterns or memory mechanisms. When trained on images or music, Perceiver AR generates outputs with clear long-term coherence and structure. Our architecture also obtains state-of-the-art likelihood on long-sequence benchmarks, including 64 x 64 ImageNet images and PG-19 books.

Motivation & Objective

• Motivate and enable general-purpose density modeling with very long input contexts.
• Develop an autoregressive architecture that decouples input length from compute via a latent bottleneck.
• Show that long-range dependencies can be captured across diverse domains (images, text, music).
• Demonstrate scalability advantages over traditional decoder-only Transformers and Transformer-XL.

Proposed method

• Use cross-attention to map a large input X (M tokens) to a small latent array Z (N latents).
• Apply a stack of causally masked self-attention over latents to produce outputs for each target token.
• Assign one latent per target to preserve autoregressive ordering and apply causal masking in cross-attention and self-attention.
• Maintain overall autoregressive causality while decoupling compute from input length, achieving complexity ~ O(MN) + O(LN^2).
• Train and evaluate on long-context domains (images, language, music) to demonstrate long-range coherence and density estimation.

Experimental results

Research questions

• RQ1 Can an autoregressive model attend to very long inputs (e.g., >10^4 tokens) without prohibitive compute?
• RQ2 Does decoupling input length from self-attention depth via a latent bottleneck retain autoregressive dependencies and improve scalability?
• RQ3 Is the approach effective across diverse modalities (images, text, music) for long-context density estimation?
• RQ4 How does Perceiver AR compare to decoder-only Transformers and Transformer-XL in terms of performance and efficiency at scale?

Key findings

• Perceiver AR can attend to over a hundred thousand tokens for long-context density estimation.
• On 64×64 ImageNet density estimation, increasing latents at evaluation improves bits/dim up to a point and enables test-time compute trade-offs.
• On PG-19 language modeling, Perceiver AR outperforms prior autoregressive models with the same tokenization and scales with context length up to thousands of tokens.
• On Books (MassiveTest) and Wikitext-103, Perceiver AR matches or surpasses Transformer-XL baselines and shows competitive perplexities.
• In MAESTRO symbolic and audio tasks, Perceiver AR achieves lower negative log-likelihoods than prior music-generation models, demonstrating long-range coherence.
• The model supports flexible test-time compute by varying the number of latents without retraining, enabling efficient deployment across budgets.