[Paper Review] Long Range Arena: A Benchmark for Efficient Transformers

Yi Tay, Mostafa Dehghani+8 more|arXiv (Cornell University)|Nov 8, 2020

Advanced Neural Network Applications47 references195 citations

TL;DR

The paper introduces Long Range Arena (LRA), a unified benchmark to evaluate efficient Transformers across long-context tasks with 1K–16K tokens, comparing ten models on diverse data types and tasks. It analyzes performance, speed, and memory to highlight trade-offs and no single-best solution.

ABSTRACT

Transformers do not scale very well to long sequence lengths largely because of quadratic self-attention complexity. In the recent months, a wide spectrum of efficient, fast Transformers have been proposed to tackle this problem, more often than not claiming superior or comparable model quality to vanilla Transformer models. To this date, there is no well-established consensus on how to evaluate this class of models. Moreover, inconsistent benchmarking on a wide spectrum of tasks and datasets makes it difficult to assess relative model quality amongst many models. This paper proposes a systematic and unified benchmark, LRA, specifically focused on evaluating model quality under long-context scenarios. Our benchmark is a suite of tasks consisting of sequences ranging from $1K$ to $16K$ tokens, encompassing a wide range of data types and modalities such as text, natural, synthetic images, and mathematical expressions requiring similarity, structural, and visual-spatial reasoning. We systematically evaluate ten well-established long-range Transformer models (Reformers, Linformers, Linear Transformers, Sinkhorn Transformers, Performers, Synthesizers, Sparse Transformers, and Longformers) on our newly proposed benchmark suite. LRA paves the way towards better understanding this class of efficient Transformer models, facilitates more research in this direction, and presents new challenging tasks to tackle. Our benchmark code will be released at https://github.com/google-research/long-range-arena.

Motivation & Objective

• Establish a unified, general benchmark for long-range Transformer models across multiple data modalities.
• Evaluate a wide set of efficient Transformer architectures under long-context challenges.
• Provide exhaustive efficiency (speed and memory) analyses to guide model selection and future research.

Proposed method

• Design a suite of long-context tasks (ListOps, Byte-level Text Classification, Byte-level Document Retrieval, Image Classification from sequences, Pathfinder and Pathfinder-X).
• Evaluate ten efficient Transformer models (Reformer, Linformer, Linear Transformers, Sparse Transformers, Longformer, Sinkhorn Transformers, Synthesizers, BigBird, Performers, and vanilla Transformer) on the tasks.
• Quantify required attention span and report task-wise and overall performance.
• Provide open-source benchmark code in JAX/Flax for easy replication and extension.

Experimental results

Research questions

• RQ1 How do different efficient Transformer architectures perform on long-range tasks across text, image, and synthetic data?
• RQ2 What are the speed and memory trade-offs among these architectures at long sequence lengths?
• RQ3 Is there a single model that consistently excels across all long-range tasks, or do trade-offs dominate?
• RQ4 How does increasing sequence length (e.g., Pathfinder-X) affect learning ability across models?

Key findings

• All LRA tasks are challenging for current models, with substantial gaps to optimal performance on several tasks.
• BigBird achieves the best overall LRA score by balance across tasks, though not the top on every individual task.
• Kernel-based variants like Performer and Linear Transformers offer strong speed/memory trade-offs, sometimes at the cost of task-specific accuracy.
• Most models struggle on extreme length (Path-X), with none solving it, highlighting limits of current architectures for ultra-long sequences.
• There is no one-size-fits-all solution; trade-offs between accuracy, speed, and memory vary by task and model.
• Memory footprint varies dramatically; Linformer can achieve near 1 GB per device at 4K, while vanilla Transformers can require ~9.48 GB per device on 4K, underscoring efficiency gaps.