[Paper Review] DropAttention: A Regularization Method for Fully-Connected Self-Attention Networks

Zehui Lin, Pengfei Liu+4 more|arXiv (Cornell University)|Jul 25, 2019

Domain Adaptation and Few-Shot Learning25 references34 citations

TL;DR

DropAttention introduces dropout specifically for fully-connected self-attention in Transformers, dropping attention weights to reduce co-adaptation and improve generalization across tasks.

ABSTRACT

Variants dropout methods have been designed for the fully-connected layer, convolutional layer and recurrent layer in neural networks, and shown to be effective to avoid overfitting. As an appealing alternative to recurrent and convolutional layers, the fully-connected self-attention layer surprisingly lacks a specific dropout method. This paper explores the possibility of regularizing the attention weights in Transformers to prevent different contextualized feature vectors from co-adaption. Experiments on a wide range of tasks show that DropAttention can improve performance and reduce overfitting.

Motivation & Objective

• Motivate the need for a dropout variant tailored to self-attention in Transformers.
• Propose DropAttention (DropAttention(c) and DropAttention(e)) to regularize attention weights.
• Investigate benefits of dropping contiguous regions and normalized rescaling in attention dropout.
• Evaluate DropAttention across text classification, sequence labeling, textual entailment, and machine translation.

Proposed method

• Reformulate the self-attention output as H̃ = f(ΛV) where Λ = softmax(QK^T / sqrt(d_k)) and V are computed from H.
• Introduce two DropAttention variants: DropAttention(c) drops attention columns (vector-level), while DropAttention(e) drops individual elements in Λ.
• Augment with contiguous-region dropping inspired by DropBlock, using window size w and drop rate p.
• Apply normalized rescaling to keep attention weight sums equal to 1 after dropout, improving training stability.
• Provide pseudocode for DropAttention(e) and analogous procedure for DropAttention(c).
• Evaluate on multiple NLP tasks to assess regularization effects and complementarity with standard dropout.

Experimental results

Research questions

• RQ1 Does DropAttention improve generalization and reduce overfitting in fully-connected self-attention networks?
• RQ2 How do DropAttention(c) and DropAttention(e) compare in terms of performance and robustness across tasks?
• RQ3 What is the impact of dropping contiguous regions (window size w) on attention distributions and model behavior?
• RQ4 Is normalized rescaling preferable to traditional dropout rescaling in attention dropout?
• RQ5 How does DropAttention interact with standard dropout when used together?

Key findings

• DropAttention improves performance across text classification, sequence labeling, textual entailment, and machine translation tasks.
• Normalized rescaling generally outperforms traditional rescaling (1-p) in DropAttention.
• DropAttention(c) often yields higher performance than DropAttention(e) in classification tasks.
• Dropping contiguous regions (larger w) and higher drop rates tends to increase attention distribution entropy and head diversity, enhancing robustness.
• DropAttention complements standard dropout and can yield additional gains when combined (Dropout + DropAttention).
• In large-scale MT (WMT'16 En-De), DropAttention with p=0.2, w=2 achieves notable BLEU gains over baseline.