[Paper Review] Rethinking Positional Encoding in Language Pre-training

Guolin Ke, Di He+1 more|arXiv (Cornell University)|Jun 28, 2020

Topic Modeling31 references67 citations

TL;DR

TUPE proposes untied positional encoding, separating word and positional correlations and untying the CLS token, improving GLUE performance and enabling faster pre-training.

ABSTRACT

In this work, we investigate the positional encoding methods used in language pre-training (e.g., BERT) and identify several problems in the existing formulations. First, we show that in the absolute positional encoding, the addition operation applied on positional embeddings and word embeddings brings mixed correlations between the two heterogeneous information resources. It may bring unnecessary randomness in the attention and further limit the expressiveness of the model. Second, we question whether treating the position of the symbol exttt{[CLS]} the same as other words is a reasonable design, considering its special role (the representation of the entire sentence) in the downstream tasks. Motivated from above analysis, we propose a new positional encoding method called extbf{T}ransformer with extbf{U}ntied extbf{P}ositional extbf{E}ncoding (TUPE). In the self-attention module, TUPE computes the word contextual correlation and positional correlation separately with different parameterizations and then adds them together. This design removes the mixed and noisy correlations over heterogeneous embeddings and offers more expressiveness by using different projection matrices. Furthermore, TUPE unties the exttt{[CLS]} symbol from other positions, making it easier to capture information from all positions. Extensive experiments and ablation studies on GLUE benchmark demonstrate the effectiveness of the proposed method. Codes and models are released at https://github.com/guolinke/TUPE.

Motivation & Objective

• Motivate reexamining absolute/relative positional encodings in language pre-training.
• Propose a new TUPE method that unties word–position correlations in self-attention.
• Untie the CLS symbol from regular positions to better capture global sentence information.
• Demonstrate improvements on GLUE benchmarks with TUPE across BERT-base settings.

Proposed method

• Compute word contextual correlation and positional correlation separately with distinct projections in self-attention.
• Replace input-level addition of word and absolute positional embeddings with separate correlation terms in attention.
• Untie CLS by resetting its position-related correlations using learnable parameters.
• Provide TUPE-A (untied absolute) and TUPE-R (untied with relative) variants.
• Share positional correlation terms across layers for efficiency.
• Evaluate on GLUE with BERT-Base, and extend analyses to BERT-Large and ELECTRA in appendices.

Experimental results

Research questions

• RQ1 Can untied, separate word and positional correlations improve Transformer pre-training compared to standard absolute/relative encodings?
• RQ2 Does untying the CLS symbol from regular positions improve sentence-level representations?
• RQ3 Do TUPE-A and TUPE-R provide complementary benefits when combined with existing relative encodings?
• RQ4 What is the impact of TUPE on GLUE benchmark performance and pre-training efficiency?

Key findings

• TUPE-A and TUPE-R outperform BERT-A and BERT-R baselines on GLUE tasks.
• TUPE-R achieves a GLUE average of 85.04, versus 83.66 for BERT-R, a gain of about 1.38 points.
• TUPE-R outperforms TUPE-A by 0.57 points on average.
• TUPE-A and TUPE-R converge faster during pre-training and can reach better downstream performance with around 30% of pre-training steps.
• Untying CLS provides notable gains on low-resource tasks (e.g., CoLA, RTE), while untied correlations help high-resource tasks (e.g., MNLI).
• TUPE adds minimal parameters (~1% of BERT-Base) and negligible additional computational cost; positional terms can be reused across layers.