[Paper Review] Multi-Scale Self-Attention for Text Classification

Qipeng Guo, Xipeng Qiu+3 more|arXiv (Cornell University)|Dec 2, 2019

Topic Modeling41 references4 citations

TL;DR

This paper proposes Multi-Scale Self-Attention (MSMSA), a novel multi-head attention mechanism that assigns variable receptive fields (scales) to individual attention heads to inject inductive bias for improved generalization. By using smaller-scale heads in shallow layers and balancing scales in deeper layers, the Multi-Scale Transformer achieves consistent and significant accuracy gains over standard Transformers on 21 datasets, especially on small-to-moderate size benchmarks.

ABSTRACT

In this paper, we introduce the prior knowledge, multi-scale structure, into self-attention modules. We propose a Multi-Scale Transformer which uses multi-scale multi-head self-attention to capture features from different scales. Based on the linguistic perspective and the analysis of pre-trained Transformer (BERT) on a huge corpus, we further design a strategy to control the scale distribution for each layer. Results of three different kinds of tasks (21 datasets) show our Multi-Scale Transformer outperforms the standard Transformer consistently and significantly on small and moderate size datasets.

Motivation & Objective

• Address the poor generalization of standard self-attention mechanisms on small and moderate-sized text classification datasets due to lack of inductive bias.
• Introduce multi-scale inductive bias into the self-attention mechanism to improve feature learning and model robustness.
• Design a scale distribution strategy across layers inspired by linguistic hierarchy, favoring local (small-scale) heads in shallow layers and balanced scales in deep layers.
• Demonstrate that multi-scale attention improves performance without replacing the standard Transformer architecture or requiring additional components.

Proposed method

• Propose Multi-Scale Multi-Head Self-Attention (MSMSA), where each attention head operates over a variable-sized receptive field (scale), restricting the context window for attention computation.
• Define the scale of a head as the number of tokens it attends to, with smaller scales emphasizing local patterns and larger scales capturing global context.
• Introduce a learnable scale distribution strategy controlled by a hyperparameter α, which modulates the scale distribution across layers: higher α increases local bias in shallow layers.
• Construct the Multi-Scale Transformer by stacking MSMSA layers, replacing standard multi-head attention and feed-forward networks with the multi-scale attention mechanism.
• Use a scale selection strategy with five candidate scales: 1, 3, N/16, N/8, N/4, where N is the sequence length.
• Train the model using standard optimization (Adam, dropout, weight decay) on text classification tasks with fixed hyperparameters across experiments.

Experimental results

Research questions

• RQ1 Can injecting multi-scale inductive bias into self-attention improve generalization on small and moderate-sized text classification datasets?
• RQ2 What is the optimal scale distribution strategy across layers for multi-scale self-attention in text classification?
• RQ3 Does multi-scale attention outperform single-scale attention mechanisms when the number of parameters and layers are held constant?
• RQ4 How does the choice of scale distribution (e.g., local bias in shallow layers) affect model performance compared to global or unbalanced distributions?
• RQ5 Can the proposed method achieve state-of-the-art performance without architectural overhauls or additional components?

Key findings

• The Multi-Scale Transformer outperforms the standard Transformer on all 21 datasets tested, with consistent and significant improvements on small and moderate-sized datasets.
• Using a positive α value (favoring local bias in shallow layers) yields the best performance, achieving 85.9% accuracy on the SNLI dataset, compared to 84.3% with the worst α.
• The model with a balanced scale distribution (α = 0.5) achieved 85.9% on SNLI, outperforming all single-scale models, which showed a maximum of 84.3% accuracy.
• Single-scale models with fixed scales (e.g., N/4) underperformed significantly, with accuracy dropping to 80.7%, indicating that fixed-scale attention lacks flexibility.
• The model with α = 1.0 (strong local bias in shallow layers) achieved 85.5% accuracy on SNLI, showing that local inductive bias is crucial for performance.
• Even with moderate-sized training data (e.g., 550k samples in SNLI), the Multi-Scale Transformer outperforms standard Transformers, indicating that prior knowledge via scale bias reduces data dependency.