[Paper Review] R-Transformer: Recurrent Neural Network Enhanced Transformer

Zhiwei Wang, Yao Ma+2 more|arXiv (Cornell University)|Jul 12, 2019

Neural Networks and Applications32 references89 citations

TL;DR

R-Transformer combines LocalRNNs for local structure with multi-head attention for global dependencies, achieving strong performance without position embeddings and outperforming Transformer and TCN on several sequence modeling tasks.

ABSTRACT

Recurrent Neural Networks have long been the dominating choice for sequence modeling. However, it severely suffers from two issues: impotent in capturing very long-term dependencies and unable to parallelize the sequential computation procedure. Therefore, many non-recurrent sequence models that are built on convolution and attention operations have been proposed recently. Notably, models with multi-head attention such as Transformer have demonstrated extreme effectiveness in capturing long-term dependencies in a variety of sequence modeling tasks. Despite their success, however, these models lack necessary components to model local structures in sequences and heavily rely on position embeddings that have limited effects and require a considerable amount of design efforts. In this paper, we propose the R-Transformer which enjoys the advantages of both RNNs and the multi-head attention mechanism while avoids their respective drawbacks. The proposed model can effectively capture both local structures and global long-term dependencies in sequences without any use of position embeddings. We evaluate R-Transformer through extensive experiments with data from a wide range of domains and the empirical results show that R-Transformer outperforms the state-of-the-art methods by a large margin in most of the tasks. We have made the code publicly available at \url{https://github.com/DSE-MSU/R-transformer}.

Motivation & Objective

• Motivate improving sequence modeling by capturing local structures while preserving long-range dependencies.
• Propose a hybrid architecture combining LocalRNN with multi-head attention.
• Demonstrate that the model can operate without position embeddings and still outperform baselines across diverse domains.

Proposed method

• Introduce LocalRNN to process local windows ending at each position, producing position representations that encode local sequential information.
• Apply a pooling sub-layer using multi-head attention to capture global long-term dependencies across positions.
• Follow a residual and layer normalization scheme across LocalRNN, attention, and feedforward sub-layers to form each layer of the R-Transformer.
• Use a three-tier layer structure: LocalRNN (local), Multi-Head Attention (global), and Position-wise Feedforward, with shared parameters and parallelizable computation.
• Compare R-Transformer against RNNs, TCN, and Transformer on multiple datasets to assess performance gains.

Experimental results

Research questions

• RQ1 Can LocalRNN effectively encode local sequential structure to enhance global attention-based models?
• RQ2 Does removing position embeddings degrade performance, or can LocalRNN and attention compensate?
• RQ3 How does R-Transformer perform relative to RNNs, TCN, and Transformer across tasks requiring varying balances of locality and long-range dependencies?
• RQ4 Is the model training and inference efficiently parallelizable similar to non-recurrent architectures?

Key findings

• R-Transformer achieves higher test accuracy than Transformer and TCN on pixel-by-pixel MNIST (99.1% vs. 98.2% for Transformer and 99.0% for TCN).
• On polyphonic Nottingham music modeling, R-Transformer attains NLL of 2.37, outperforming LSTM (3.29), GRU (3.46), TCN (3.07) and Transformer (3.34).
• For Penn Treebank character-level language modeling, R-Transformer achieves NLL of 1.24, better than Transformer (1.45) and comparable or better than RNN-based baselines.
• In word-level PTB language modeling, R-Transformer attains perplexity of 84.38, outperforming Transformer (122.37) and other baselines (RNN/GRU/LSTM/TCN).
• Across tasks, R-Transformer consistently outperforms both TCN and Transformer, leveraging LocalRNN for locality and multi-head attention for long-range dependencies.
• The model is implemented to enable full parallelization over sequence positions and does not rely on position embeddings.