[Paper Review] Pay Less Attention with Lightweight and Dynamic Convolutions

Felix Wu, Angela Fan+3 more|arXiv (Cornell University)|Jan 29, 2019

Natural Language Processing Techniques322 citations

TL;DR

The paper introduces lightweight and dynamic convolutions as efficient alternatives to self-attention for sequence modeling, achieving competitive or superior results on translation, language modeling, and summarization, with faster runtime.

ABSTRACT

Self-attention is a useful mechanism to build generative models for language and images. It determines the importance of context elements by comparing each element to the current time step. In this paper, we show that a very lightweight convolution can perform competitively to the best reported self-attention results. Next, we introduce dynamic convolutions which are simpler and more efficient than self-attention. We predict separate convolution kernels based solely on the current time-step in order to determine the importance of context elements. The number of operations required by this approach scales linearly in the input length, whereas self-attention is quadratic. Experiments on large-scale machine translation, language modeling and abstractive summarization show that dynamic convolutions improve over strong self-attention models. On the WMT'14 English-German test set dynamic convolutions achieve a new state of the art of 29.7 BLEU.

Motivation & Objective

• Motivate reducing reliance on quadratic self-attention in sequence models.
• Propose lightweight convolutions with depthwise separable structure and softmax-normalized weights.
• Introduce dynamic convolutions that generate time-step specific kernels.
• Evaluate on machine translation, language modeling, and abstractive summarization to compare with self-attention baselines.

Proposed method

• Develop LightConv: a depthwise separable, softmax-normalized, and weight-shared convolution that operates with a fixed context window.
• Introduce DynamicConv: a time-step dependent kernel generated from the current input, enabling time-varying context weighting.
• Use GLU-based blocks and residual connections in an encoder-decoder architecture analogous to Transformer Big, replacing self-attention with LightConv or DynamicConv.
• Train with standard NLP objectives and hyperparameters across translation, language modeling, and summarization datasets.
• Evaluate on WMT En-De, WMT En-Fr, IWSLT De-En, WMT Zh-En, Billion word language modeling, and CNN-DailyMail summarization.

Experimental results

Research questions

• RQ1 Can lightweight, fixed-weight convolutions match or exceed self-attention performance on large-scale translation benchmarks?
• RQ2 Do dynamic, time-step dependent kernels provide additional gains over fixed lightweight convolutions?
• RQ3 Are lightweight and dynamic convolutions more efficient in runtime compared to self-attention without sacrificing accuracy?
• RQ4 Do these approaches generalize well across language modeling and abstractive summarization tasks?
• RQ5 How do these methods scale with longer sequences and larger vocabularies?

Key findings

• LightConv achieves competitive BLEU on WMT En-De and En-Fr, trailing state of the art by only 0.1 BLEU on En-Fr.
• DynamicConv surpasses the best known result on WMT En-De by 0.4 BLEU and matches the state of the art on En-Fr.
• On IWSLT De-En and WMT Zh-En, lightweight and dynamic convolutions outperform or match self-attention baselines.
• DynamicConv delivers about 20% faster runtime than strong self-attention baselines while preserving or improving accuracy.
• In language modeling on the Billion Word corpus, DynamicConv performs as well as or better than self-attention baselines.
• In CNN-DailyMail summarization, LightConv and DynamicConv outperform self-attention baselines, with DynamicConv achieving the best Rouge scores.