[Paper Review] Synthesizer: Rethinking Self-Attention in Transformer Models

Yi Tay, Dara Bahri+4 more|arXiv (Cornell University)|May 2, 2020

Natural Language Processing TechniquesComputer Science18 references198 citations

TL;DR

Synthesizer replaces dot-product self-attention with synthetic attention to learn alignment without token-token interactions. Random/dense variants are competitive across tasks and can outperform some baselines, while combining synthetic and dot-product attention yields further gains.

ABSTRACT

The dot product self-attention is known to be central and indispensable to state-of-the-art Transformer models. But is it really required? This paper investigates the true importance and contribution of the dot product-based self-attention mechanism on the performance of Transformer models. Via extensive experiments, we find that (1) random alignment matrices surprisingly perform quite competitively and (2) learning attention weights from token-token (query-key) interactions is useful but not that important after all. To this end, we propose extsc{Synthesizer}, a model that learns synthetic attention weights without token-token interactions. In our experiments, we first show that simple Synthesizers achieve highly competitive performance when compared against vanilla Transformer models across a range of tasks, including machine translation, language modeling, text generation and GLUE/SuperGLUE benchmarks. When composed with dot product attention, we find that Synthesizers consistently outperform Transformers. Moreover, we conduct additional comparisons of Synthesizers against Dynamic Convolutions, showing that simple Random Synthesizer is not only $60\%$ faster but also improves perplexity by a relative $3.5\%$. Finally, we show that simple factorized Synthesizers can outperform Linformers on encoding only tasks.

Motivation & Objective

• Question the necessity of dot-product self-attention in Transformers.
• Investigate how synthetic (non-token-dependent) attention performs across language tasks.
• Propose and evaluate various Synthesizer variants (dense, random, factorized, mixtures).
• Assess how synthetic attention composes with traditional dot-product attention and with fast Transformer alternatives.

Proposed method

• Define Synthetic Attention that replaces QK weight computation with a synthesizing function F that outputs an alignment matrix.
• Implement Dense Synthesizer where each token projects to a sequence-length vector via a two-layer feed-forward network.
• Define Random Synthesizer with fixed or trainable random alignment matrix R and softmax normalization.
• Introduce factorized variants to reduce parameter count (Dense and Random) and mixtures combining multiple synthesizers.
• Evaluate variants on MT (WMT En-De/En-Fr), language modeling (LM1B), text generation, and GLUE/SuperGLUE benchmarks, and compare to Dinamic Convolutions and Linformers.

Experimental results

Research questions

• RQ1 Is dot-product self-attention essential for Transformer performance across major NLP tasks?
• RQ2 Can a synthetic attention mechanism, independent of token-token interactions, match or exceed vanilla Transformers?
• RQ3 Do mixtures of synthetic and dot-product attention yield robust gains across tasks?
• RQ4 How do random or factorized synthetic attention variants compare to fast Transformer alternatives like Linformers and Dynamic Convolutions?

Key findings

• Random Synthesizer achieves 27.27 BLEU on WMT 2014 English-German and 41.12 BLEU on English-French translation.
• Random Synthesizer can reach competitive language modeling perplexities within 1–2 points of the vanilla Transformer on LM1B.
• Simple Random Synthesizers outperform Dynamic Convolutions in perplexity by about 3.5% relative and are ~60% faster in MLM settings.
• Factorized Random Synthesizers can outperform Linformers on encoding-only tasks.
• Mixture variants that combine synthetic and dot-product attention yield consistent performance gains over Transformers across several tasks.
• On dialogue generation, dot-product attention can hurt performance, while Synthesizers (D) and (R) provide improvements.