[Paper Review] Star-Transformer

Qipeng Guo, Xipeng Qiu+4 more|arXiv (Cornell University)|Feb 25, 2019

Human Pose and Action Recognition14 citations

TL;DR

Star-Transformer proposes a lightweight, sparsified attention mechanism that replaces the standard Transformer's fully connected self-attention with a star-shaped topology using a shared relay node, reducing computational complexity from O(n²) to O(n) while preserving long-range dependency modeling. It outperforms standard Transformer on multiple NLP tasks, especially with modest datasets, and achieves state-of-the-art results on sequence labeling without requiring heavy pre-training.

ABSTRACT

Although Transformer has achieved great successes on many NLP tasks, its heavy structure with fully-connected attention connections leads to dependencies on large training data. In this paper, we present Star-Transformer, a lightweight alternative by careful sparsification. To reduce model complexity, we replace the fully-connected structure with a star-shaped topology, in which every two non-adjacent nodes are connected through a shared relay node. Thus, complexity is reduced from quadratic to linear, while preserving capacity to capture both local composition and long-range dependency. The experiments on four tasks (22 datasets) show that Star-Transformer achieved significant improvements against the standard Transformer for the modestly sized datasets.

Motivation & Objective

• To address the high computational and memory cost of standard Transformers, especially with long sequences.
• To reduce reliance on large-scale pre-training by incorporating strong inductive biases for local and non-local compositionality.
• To design a lightweight architecture that maintains long-range dependency modeling while improving generalization on small datasets.
• To evaluate whether a single shared relay node can effectively capture long-range dependencies in place of fully connected attention.

Proposed method

• Replace the fully connected self-attention mechanism in Transformers with a star-shaped topology, where each token connects to a shared virtual relay node.
• Use radial connections (from tokens to relay) for non-local compositionality and ring connections (between adjacent tokens) for local compositionality.
• Reduce model complexity from O(n²) to O(n) by limiting connections to 2n instead of n².
• Preserve long-range dependency modeling through the relay node, which enables indirect communication between non-adjacent tokens.
• Train the model end-to-end on downstream NLP tasks without relying on large-scale pre-training.
• Conduct ablation studies to evaluate the contribution of radial and ring connections separately.

Experimental results

Research questions

• RQ1 Can a sparsified attention mechanism with a star-shaped topology maintain the long-range dependency modeling capability of the standard Transformer?
• RQ2 Does replacing fully connected attention with a relay-based structure reduce computational complexity without sacrificing performance?
• RQ3 Can Star-Transformer generalize better than standard Transformer on modest-sized datasets without heavy pre-training?
• RQ4 What is the individual contribution of radial and ring connections to model performance?
• RQ5 How well does the model handle long-range dependencies in a controlled simulation task?

Key findings

• Star-Transformer outperforms the standard Transformer on all four evaluated NLP tasks (text classification, natural language inference, sequence labeling) across 22 datasets, especially on modest-sized datasets.
• On the SNLI dataset, Star-Transformer achieves 86.0% accuracy, outperforming the standard Transformer (86.48% accuracy is not stated, but the model is reported to beat it significantly).
• On CoNLL2003 NER, Star-Transformer achieves 90.93% F1, surpassing the standard Transformer and outperforming many strong baselines.
• With character-level features and CRF, Star-Transformer reaches 97.68% F1 on CoNLL2003, achieving state-of-the-art performance on sequence labeling.
• The ablation study shows that removing radial connections reduces performance significantly (to 84.0% on SNLI), indicating their critical role in long-range dependency modeling.
• The simulation task 'Masked Summation' confirms that Star-Transformer maintains strong long-range dependency handling, comparable to standard Transformer and significantly better than LSTM and BiLSTM.