[Paper Review] RoFormer: Enhanced Transformer with Rotary Position Embedding

Jianlin Su, Yu Lu+4 more|arXiv (Cornell University)|Apr 20, 2021

Topic ModelingComputer Science35 references229 citations

TL;DR

RoFormer introduces Rotary Position Embedding (RoPE) to encode absolute and relative position information via rotation matrices, improving performance across translation, pre-training, and GLUE tasks, with compatibility to linear attention.

ABSTRACT

Position encoding recently has shown effective in the transformer architecture. It enables valuable supervision for dependency modeling between elements at different positions of the sequence. In this paper, we first investigate various methods to integrate positional information into the learning process of transformer-based language models. Then, we propose a novel method named Rotary Position Embedding(RoPE) to effectively leverage the positional information. Specifically, the proposed RoPE encodes the absolute position with a rotation matrix and meanwhile incorporates the explicit relative position dependency in self-attention formulation. Notably, RoPE enables valuable properties, including the flexibility of sequence length, decaying inter-token dependency with increasing relative distances, and the capability of equipping the linear self-attention with relative position encoding. Finally, we evaluate the enhanced transformer with rotary position embedding, also called RoFormer, on various long text classification benchmark datasets. Our experiments show that it consistently overcomes its alternatives. Furthermore, we provide a theoretical analysis to explain some experimental results. RoFormer is already integrated into Huggingface: \url{https://huggingface.co/docs/transformers/model_doc/roformer}.

Motivation & Objective

• Investigate how to incorporate positional information into transformer learning beyond additive absolute encodings.
• Propose Rotary Position Embedding (RoPE) that encodes absolute position with a rotation matrix and encodes relative position in self-attention.
• Demonstrate RoPE benefits on long-text classification, machine translation, and pre-training tasks.
• Provide theoretical analysis of RoPE properties such as sequence-length flexibility and decay of inter-token dependency with distance.

Proposed method

• Derive RoPE by modeling the inner product between query and key as a function of embeddings and relative positions.
• Represent f_q and f_k as rotated affine transformations of token embeddings using a rotation matrix R_Θ,m (Equation 14/15).
• Show that q_m^T k_n equals (W_q x_m)^T R^d_Θ,m^T R^d_Θ,n (W_k x_n), which simplifies to a relative-position dependent term (Equation 16).
• Integrate RoPE with self-attention and, for linear attention, multiply rotated Q and K with non-negative feature maps (Equation 19).
• Prove long-term decay of RoPE via angular terms θ_i and show computationally efficient rotation implementation (Equations 15–16, 34).
• Evaluate RoFormer on machine translation (WMT14 En-De), MLM pre-training (BERT baseline), GLUE fine-tuning, and PerFormer with RoPE.

Experimental results

Research questions

• RQ1 How can absolute and relative position information be effectively encoded in transformers using rotation?
• RQ2 Does RoPE improve performance and convergence over additive position encodings across translation, language modeling, and downstream tasks?
• RQ3 Can RoPE be efficiently integrated with linear attention without sacrificing stability or performance?
• RQ4 What are the theoretical properties (e.g., decay with distance) of RoPE and how do they relate to natural language structure?

Key findings

• RoFormer achieves higher BLEU than Transformer-base on WMT 2014 English-German translation (BLEU: 27.5 vs 27.3).
• RoFormer converges faster than vanilla BERT during MLM pre-training on BookCorpus+Wikipedia.
• RoFormer improves several GLUE tasks over BERT in fine-tuning (MRPC, SST-2, QNLI, STS-B, QQP, MNLI) with mixed gains (e.g., MRPC 89.5 vs 88.9; SST-2 90.7 vs 93.5; QNLI 88.0 vs 90.5; STS-B 87.0 vs 85.8; QQP 86.4 vs 71.2; MNLI 80.2/79.8 vs 84.6/83.4).
• RoPE is compatible with linear attention (PerFormer), maintaining its linear complexity while encoding relative position information.
• RoPE provides sequence-length flexibility and a decaying inter-token dependency with increasing relative distance, aligning with linguistic structure.