[Paper Review] Learning Multiscale Transformer Models for Sequence Generation

Bei Li, Tong Zheng+4 more|arXiv (Cornell University)|Jun 19, 2022

Topic Modeling4 citations

TL;DR

This paper proposes Umst, a Universal Multi-Scale Transformer that explicitly models sub-word, word, and phrase-level scales using word boundaries and phrase-level prior knowledge to improve sequence generation. By integrating inter-scale relationships—inter-individual, intra-group, and inter-group—it achieves consistent gains over standard and multiscale Transformers without efficiency loss, with +0.88 BLEU and +1 ROUGE point gains on average.

ABSTRACT

Multiscale feature hierarchies have been witnessed the success in the computer vision area. This further motivates researchers to design multiscale Transformer for natural language processing, mostly based on the self-attention mechanism. For example, restricting the receptive field across heads or extracting local fine-grained features via convolutions. However, most of existing works directly modeled local features but ignored the word-boundary information. This results in redundant and ambiguous attention distributions, which lacks of interpretability. In this work, we define those scales in different linguistic units, including sub-words, words and phrases. We built a multiscale Transformer model by establishing relationships among scales based on word-boundary information and phrase-level prior knowledge. The proposed extbf{U}niversal extbf{M}ulti extbf{S}cale extbf{T}ransformer, namely extsc{Umst}, was evaluated on two sequence generation tasks. Notably, it yielded consistent performance gains over the strong baseline on several test sets without sacrificing the efficiency.

Motivation & Objective

• To address the limitation of existing multiscale Transformers in NLP, which ignore word boundaries and phrase-level structure when modeling local features.
• To improve interpretability and reduce ambiguity in self-attention distributions by incorporating linguistic-scale hierarchies.
• To design a flexible, universal framework that integrates prior knowledge such as word boundaries and phrase structure into the Transformer architecture.
• To achieve consistent performance gains in sequence generation tasks without sacrificing computational efficiency.

Proposed method

• The paper redefines scale in NLP as sub-word, word, and phrase levels, using word boundaries and phrase-level prior knowledge to structure multiscale representations.
• It introduces a three-fold relationship modeling mechanism: inter-individual (within-scale), intra-group (within word/phrase), and inter-group (between scales) to capture hierarchical dependencies.
• The Umst model integrates these scale-aware relationships into the self-attention mechanism, allowing attention to focus more naturally on meaningful linguistic units.
• It uses a modified Transformer block where multi-head attention is enhanced with scale-aware attention computation that respects word and phrase boundaries.
• The model maintains efficiency by avoiding additional parameters or complex operations, making it orthogonal to existing techniques like relative positional embeddings.
• The framework is designed to be extensible, allowing integration of other linguistic priors beyond word boundaries and phrase structure.

Experimental results

Research questions

• RQ1 Can modeling sub-word, word, and phrase-level scales explicitly improve the performance of sequence generation models?
• RQ2 How does incorporating word boundaries and phrase-level knowledge affect attention distribution interpretability and quality?
• RQ3 Does a multiscale Transformer that respects linguistic structure outperform standard and existing multiscale Transformers in sequence generation tasks?
• RQ4 Can the proposed method achieve performance gains without increasing computational cost or model complexity?
• RQ5 To what extent can Umst be extended to incorporate additional linguistic priors beyond word boundaries and phrase structure?

Key findings

• Umst achieved a +0.88 BLEU point improvement over the standard Transformer on the base configuration in machine translation.
• On the big model configuration, Umst outperformed the baseline by +0.44 BLEU points, demonstrating consistent gains across model sizes.
• In abstractive summarization, Umst achieved an average of +1.0 ROUGE point gain over the Transformer baseline.
• The attention maps of Umst showed clearer, more interpretable patterns that aligned with word boundaries, unlike the ambiguous and redundant distributions in standard models.
• When combined with relative positional embeddings, Umst gained an additional +0.4 BLEU points, indicating orthogonality and compatibility with existing improvements.
• The model’s performance gains were consistent across multiple test sets, confirming robustness and generalization in sequence generation tasks.