[Paper Review] Pay Better Attention to Attention: Head Selection in Multilingual and Multi-Domain Sequence Modeling

Hongyu Gong, Yun Tang+2 more|arXiv (Cornell University)|Jun 21, 2021

Topic Modeling35 references5 citations

TL;DR

This paper proposes adaptive attention head selection strategies—subset and group strategies—to improve multilingual and multi-domain sequence modeling by enabling selective parameter sharing in multi-head attention. By learning which heads to share across languages or domains, the method reduces interference while preserving specialization, achieving consistent gains across speech recognition, text-to-text, and speech-to-text translation tasks, with up to +2.0 BLEU in multilingual speech-to-text translation.

ABSTRACT

Multi-head attention has each of the attention heads collect salient information from different parts of an input sequence, making it a powerful mechanism for sequence modeling. Multilingual and multi-domain learning are common scenarios for sequence modeling, where the key challenge is to maximize positive transfer and mitigate negative transfer across languages and domains. In this paper, we find that non-selective attention sharing is sub-optimal for achieving good generalization across all languages and domains. We further propose attention sharing strategies to facilitate parameter sharing and specialization in multilingual and multi-domain sequence modeling. Our approach automatically learns shared and specialized attention heads for different languages and domains to mitigate their interference. Evaluated in various tasks including speech recognition, text-to-text and speech-to-text translation, the proposed attention sharing strategies consistently bring gains to sequence models built upon multi-head attention. For speech-to-text translation, our approach yields an average of $+2.0$ BLEU over $13$ language directions in multilingual setting and $+2.0$ BLEU over $3$ domains in multi-domain setting.

Motivation & Objective

• Address interference in multilingual and multi-domain sequence modeling caused by non-selective parameter sharing in multi-head attention.
• Improve generalization and knowledge transfer across diverse languages and domains by enabling dynamic, data-driven selection of shared attention heads.
• Mitigate negative transfer while preserving language and domain specificity through lightweight, efficient attention head sharing strategies.
• Evaluate the proposed method across diverse sequence modeling tasks, including speech recognition, text-to-text, and speech-to-text translation.
• Demonstrate consistent performance gains in low-resource and high-resource settings without sacrificing computational efficiency.

Proposed method

• Introduce two attention head selection strategies: subset and group strategies, to control which attention heads are shared across languages or domains.
• Apply the strategies at the layer level in both encoder and decoder of Transformer-based models, allowing per-layer selection of shared or specialized heads.
• Use a learnable gating mechanism to determine which heads are active for each language or domain, enabling dynamic adaptation during training.
• Design the selection process to be lightweight, avoiding significant computational overhead, and maintain compatibility with standard multi-head attention.
• Train models with shared and specialized heads using standard backpropagation, with attention head selection learned end-to-end.
• Visualize attention head load distributions to analyze sharing patterns and balance across languages and model layers.

Experimental results

Research questions

• RQ1 How does selective attention head sharing affect performance in multilingual and multi-domain sequence modeling?
• RQ2 Can attention head selection strategies reduce interference between dissimilar languages or domains while preserving knowledge transfer?
• RQ3 What is the impact of applying head selection in the encoder versus decoder on model performance?
• RQ4 How do sharing patterns (e.g., number of shared heads) vary across high-resource and low-resource languages?
• RQ5 How balanced is the load distribution across attention heads under different selection strategies?

Key findings

• The proposed attention head selection strategies consistently improve performance across multiple sequence modeling tasks, including multilingual speech recognition and speech-to-text translation.
• In multilingual speech-to-text translation, the method achieves an average of +2.0 BLEU across 13 language directions, demonstrating strong positive transfer.
• In multi-domain speech-to-text translation, the method yields +2.0 BLEU improvement over 3 domains, confirming effectiveness in domain generalization.
• Applying attention selection in both encoder and decoder yields better performance than applying it in only one, with encoder-only selection outperforming decoder-only selection in multilingual ASR.
• High-resource European languages (e.g., Italian, French) share fewer attention heads due to sufficient data, reducing interference, while low-resource languages benefit from sharing with high-resource counterparts.
• The group strategy produces more balanced attention head load distribution across layers compared to the subset strategy, especially in middle layers of encoder and decoder.