[Paper Review] Explicit Sparse Transformer: Concentrated Attention Through Explicit Selection

Guangxiang Zhao, Junyang Lin+4 more|arXiv (Cornell University)|Dec 25, 2019

Multimodal Machine Learning ApplicationsComputer Science52 references77 citations

TL;DR

This paper introduces Explicit Sparse Transformer, which concentrates attention by selecting the top-k most contributive positions, improving performance and efficiency across NLP and vision tasks.

ABSTRACT

Self-attention based Transformer has demonstrated the state-of-the-art performances in a number of natural language processing tasks. Self-attention is able to model long-term dependencies, but it may suffer from the extraction of irrelevant information in the context. To tackle the problem, we propose a novel model called extbf{Explicit Sparse Transformer}. Explicit Sparse Transformer is able to improve the concentration of attention on the global context through an explicit selection of the most relevant segments. Extensive experimental results on a series of natural language processing and computer vision tasks, including neural machine translation, image captioning, and language modeling, all demonstrate the advantages of Explicit Sparse Transformer in model performance. We also show that our proposed sparse attention method achieves comparable or better results than the previous sparse attention method, but significantly reduces training and testing time. For example, the inference speed is twice that of sparsemax in Transformer model. Code will be available at \url{https://github.com/lancopku/Explicit-Sparse-Transformer}

Motivation & Objective

• Motivate the need for more focused attention in Transformer models to reduce distraction from irrelevant context.
• Propose Explicit Sparse Transformer with top-k selective attention to sharpen global context modeling.
• Demonstrate improvements over vanilla Transformer on neural machine translation, image captioning, and language modeling.
• Show that sparse attention can be faster than previous sparse attention methods while maintaining or improving accuracy.

Proposed method

• Compute standard QK^T attention scores and apply a top-k mask per query row to retain only the k largest scores.
• Mask non-top-k scores with -infinity before softmax to obtain a concentrated attention distribution.
• Normalize with softmax on the masked scores to produce attention weights.
• Compute context as C = AV using the sparse attention weights A and values V.
• Extend the sparse mechanism to context attention where Q is derived from decoding states.
• Provide a simple, implementation-friendly approach compatible with self-attention and context attention.

Experimental results

Research questions

• RQ1 Does explicit top-k selective attention improve model focus and performance compared to vanilla Transformer?
• RQ2 How should the hyperparameter k be chosen across tasks and datasets?
• RQ3 What are the training and inference efficiency benefits of top-k sparse attention relative to other sparse attention methods?
• RQ4 Can sparse attention help with model alignment and reduce distraction from irrelevant context?
• RQ5 What qualitative differences in attention distributions emerge when using explicit sparse attention?

Key findings

• Explicit Sparse Transformer yields higher BLEU on En-De (29.4 vs. 28.4 for Transformer).
• On En-Vi, it achieves 31.1 BLEU vs. 30.2 for Transformer.
• On De-En, it achieves 35.6 BLEU (top lines reported).
• In image captioning (COCO), it improves CIDEr and BLEU-4 slightly over a Transformer baseline.
• In language modeling (enwiki8), it outperforms Transformer-XL with comparable parameter count.
• Top-k sparse attention reduces training/inference time relative to prior sparse attention methods, achieving about 2x faster training/inference in some settings.