[Paper Review] Combiner: Full Attention Transformer with Sparse Computation Cost

Hongyu Ren, Hanjun Dai+5 more|arXiv (Cornell University)|Jul 12, 2021

Topic ModelingComputer Science44 references28 citations

TL;DR

Combiner provides full attention with sub-quadratic cost by modeling attention as a structured conditional distribution, achieving strong results on long-sequence text and image tasks and matching or beating baselines in several benchmarks.

ABSTRACT

Transformers provide a class of expressive architectures that are extremely effective for sequence modeling. However, the key limitation of transformers is their quadratic memory and time complexity $\mathcal{O}(L^2)$ with respect to the sequence length in attention layers, which restricts application in extremely long sequences. Most existing approaches leverage sparsity or low-rank assumptions in the attention matrix to reduce cost, but sacrifice expressiveness. Instead, we propose Combiner, which provides full attention capability in each attention head while maintaining low computation and memory complexity. The key idea is to treat the self-attention mechanism as a conditional expectation over embeddings at each location, and approximate the conditional distribution with a structured factorization. Each location can attend to all other locations, either via direct attention, or through indirect attention to abstractions, which are again conditional expectations of embeddings from corresponding local regions. We show that most sparse attention patterns used in existing sparse transformers are able to inspire the design of such factorization for full attention, resulting in the same sub-quadratic cost ($\mathcal{O}(L\log(L))$ or $\mathcal{O}(L\sqrt{L})$). Combiner is a drop-in replacement for attention layers in existing transformers and can be easily implemented in common frameworks. An experimental evaluation on both autoregressive and bidirectional sequence tasks demonstrates the effectiveness of this approach, yielding state-of-the-art results on several image and text modeling tasks.

Motivation & Objective

• Motivate scalable sequence modeling with full attention for very long sequences.
• Introduce Combiner as a structured conditional expectation approach that preserves full attention within sub-quadratic cost.
• Show how to instantiate Combiner from existing sparse attention patterns to maintain expressiveness.
• Demonstrate state-of-the-art results on density estimation for images and long-range sequence tasks while being practical to implement.

Proposed method

• Reinterpret attention as a conditional expectation of value embeddings given a query position.
• Factorize the conditional distribution into a direct attention term and multiple local (abstraction-based) attentions.
• Use sub-quadratic local factorization schemes that are inspired by existing sparse patterns to achieve full attention with cost O(L log L) or O(L sqrt L).
• Parameterize probabilities with scaled dot-products and simple abstractions (max pooling or DeepSets) for local regions.
• Provide several Combiner instantiations (Fixed, Logsparse, Axial, Learnable) that convert existing sparse patterns into full-attention variants.
• Optional mixture components to increase expressiveness of the local factorization.

Experimental results

Research questions

• RQ1 Can full attention be achieved with sub-quadratic computation and memory using a structured factorization of the attention distribution?
• RQ2 How can existing sparse attention patterns be repurposed to yield full attention without sacrificing efficiency?
• RQ3 What are the empirical performance gains of Combiner on autoregressive and bidirectional tasks across text and image domains?

Key findings

• Combiner achieves full attention within each head with sub-quadratic cost, matching the asymptotic complexity of the sparse variants (e.g., O(L log L) or O(L sqrt L)).
• On CIFAR-10, Combiner-Axial attains 2.77 bits/dim, outperforming several baselines.
• On ImageNet-64, Combiner-Axial attains 3.42 bits/dim, competitive with or better than prior sparse/transformer baselines.
• In language modeling on Wiki-40B, Combiner-Fixed-2k and Combiner-Axial-2k variants achieve lower perplexity than several baselines, with further gains when using 8k sequences.
• In Long-Range Arena benchmarks, Combiner variants (Fixed, Axial) match or exceed vanilla Transformer performance across multiple tasks.
• Combiner demonstrates strong density estimation and autoregressive/bidirectional modeling performance while remaining implementable in standard frameworks.