[Paper Review] Efficient Transformers: A Survey

Yi Tay, Mostafa Dehghani+2 more|arXiv (Cornell University)|Sep 14, 2020

Anomaly Detection Techniques and Applications220 citations

TL;DR

A comprehensive survey of efficiency-focused Transformer variants (X-formers) that reduce self-attention costs, organize models by core techniques, and discuss cross-domain applicability and trade-offs.

ABSTRACT

Transformer model architectures have garnered immense interest lately due to their effectiveness across a range of domains like language, vision and reinforcement learning. In the field of natural language processing for example, Transformers have become an indispensable staple in the modern deep learning stack. Recently, a dizzying number of "X-former" models have been proposed - Reformer, Linformer, Performer, Longformer, to name a few - which improve upon the original Transformer architecture, many of which make improvements around computational and memory efficiency. With the aim of helping the avid researcher navigate this flurry, this paper characterizes a large and thoughtful selection of recent efficiency-flavored "X-former" models, providing an organized and comprehensive overview of existing work and models across multiple domains.

Motivation & Objective

• Provide a systematic taxonomy of efficient Transformer models and their technical innovations.
• Summarize major approaches to reducing attention and overall Transformer compute (memory, FLOPs).
• Highlight representative models, their use cases, and trade-offs across language and vision tasks.

Proposed method

• Classify efficient Transformers by core technique (Fixed Patterns, Combination of Patterns, Learnable Patterns, Neural Memory, Low-Rank, Kernels, Recurrence, Downsampling, Sparse/Conditional Computation).
• Detail a representative walk-through of key models (Memory Compressed Transformer, Image Transformer, Set Transformer, Sparse Transformers, Reformer, Linformer, Performers, etc.).
• Discuss practical considerations: when local vs global attention is needed, encoder/decoder usage, and accuracy vs efficiency trade-offs.

Experimental results

Research questions

• RQ1 What architectures and techniques effectively reduce the quadratic cost of self-attention in Transformers across NLP and vision?
• RQ2 How do different efficiency paradigms (sparsity, memory, low-rank, kernels, recurrence, downsampling) compare in terms of complexity, applicability, and performance?
• RQ3 What are representative models and their trade-offs for long-sequence processing and on-device deployment?
• RQ4 How do efficient Transformers generalize across domains and modalities?

Key findings

• Efficient Transformers employ a spectrum of strategies (local/block attention, memory tokens, low-rank projections, kernel-based approximations, recurrence, downsampling, and mixture-of-experts) to reduce memory and/or compute.
• A taxonomy aggregating methods into Fixed Patterns, Combination of Patterns, Learnable Patterns, Neural Memory, Low-Rank, Kernels, Recurrence, Downsampling, and Sparse/Conditional Computation helps organize rapidly evolving work.
• Various models achieve sub-quadratic or linear attention complexities (e.g., Linformer, Performer, Longformer, Reformer, Sparse Transformer family, GShard/Switch Transformer in sparse regimes).
• There is a trade-off between maintaining global context and reducing computation, with local attention often sacrificing some global information.
• Some approaches blend multiple techniques (e.g., local plus memory, or downsampling with attention) to balance coverage and efficiency.
• The paper includes a detailed table (Table 1) summarizing model complexity, decoding capability, and class of efficiency technique across many models.