[Paper Review] Big Bird: Transformers for Longer Sequences

Manzil Zaheer, Guru Guruganesh+9 more|arXiv (Cornell University)|Jul 28, 2020

Topic ModelingComputer Science111 references273 citations

TL;DR

Big Bird introduces a sparse attention mechanism that makes Transformers scale linearly with sequence length, enabling longer-context modeling with theoretical guarantees and strong empirical gains on NLP and genomics tasks.

ABSTRACT

Transformers-based models, such as BERT, have been one of the most successful deep learning models for NLP. Unfortunately, one of their core limitations is the quadratic dependency (mainly in terms of memory) on the sequence length due to their full attention mechanism. To remedy this, we propose, BigBird, a sparse attention mechanism that reduces this quadratic dependency to linear. We show that BigBird is a universal approximator of sequence functions and is Turing complete, thereby preserving these properties of the quadratic, full attention model. Along the way, our theoretical analysis reveals some of the benefits of having $O(1)$ global tokens (such as CLS), that attend to the entire sequence as part of the sparse attention mechanism. The proposed sparse attention can handle sequences of length up to 8x of what was previously possible using similar hardware. As a consequence of the capability to handle longer context, BigBird drastically improves performance on various NLP tasks such as question answering and summarization. We also propose novel applications to genomics data.

Motivation & Objective

• Motivate and address the quadratic memory bottleneck of standard self-attention for long sequences.
• Propose a sparse attention mechanism combining global tokens, local window attention, and random connections.
• Prove universal approximation and Turing completeness properties for sparse-attention transformers.
• Demonstrate empirical gains on long-context NLP tasks (QA, summarization) and genomics applications.
• Explore implications and limitations of sparse attention in theory and practice.

Proposed method

• Define a generalized sparse attention over a directed graph D with attention from each token i to a neighborhood N(i).
• Use three components: g global tokens attending to all tokens, local window attention of width w, and r random attention connections.
• Show Big Bird variants (BigBird-itc and BigBird-etc) with added global tokens to preserve expressivity.
• Prove that sparse-attention encoders are universal approximators of sequence-to-sequence functions and that sparse encoder-decoder transformers are Turing complete.
• Provide complexity analysis showing linear in n attention, in contrast to quadratic for full attention.
• Present empirical results on pretraining with MLM, QA benchmarks, long-document summarization, and genomics tasks with longer context.

Experimental results

Research questions

• RQ1 Can sparse attention achieve the empirical benefits of full quadratic attention while reducing computational and memory costs?
• RQ2 Do sparse-attention transformers retain the expressivity of full Transformers (universal approximation and Turing completeness)?
• RQ3 How much context can Big Bird effectively model, and what are the trade-offs versus local or random attention alone?
• RQ4 Do long-context models improve performance on QA, summarization, and genomics tasks compared to fixed-length-context models?

Key findings

• Big Bird reduces attention complexity to O(n) and can handle sequences up to 8x longer than previous methods on similar hardware.
• Sparse attention with global tokens preserves universal function approximation properties and is Turing complete.
• Big Bird achieves state-of-the-art or strong results on QA and summarization benchmarks using longer contexts.
• Long-context pretraining with Big Bird improves downstream tasks including promoter-region and chromatin profile prediction in genomics.
• Empirical results show gains over BERT-like baselines and comparable methods (e.g., Longformer) on multiple datasets.
• There are theoretical lower bounds showing limitations for extremely sparse attention requiring more layers for certain tasks.