[Paper Review] Transformer Language Models without Positional Encodings Still Learn Positional Information

Adi Haviv, Ori Ram+3 more|arXiv (Cornell University)|Mar 30, 2022

Topic Modeling4 citations

TL;DR

This paper demonstrates that causal transformer language models without explicit positional encodings still learn implicit absolute positional information through the causal attention mechanism, achieving competitive performance across model sizes, datasets, and sequence lengths. The authors show that the causal mask enables models to infer token positions by counting attendable predecessors, explaining why NoPos models outperform bidirectional models in the absence of positional encoding.

ABSTRACT

Causal transformer language models (LMs), such as GPT-3, typically require some form of positional encoding, such as positional embeddings. However, we show that LMs without any explicit positional encoding are still competitive with standard models, and that this phenomenon is robust across different datasets, model sizes, and sequence lengths. Probing experiments reveal that such models acquire an implicit notion of absolute positions throughout the network, effectively compensating for the missing information. We conjecture that causal attention enables the model to infer the number of predecessors that each token can attend to, thereby approximating its absolute position. Our findings indicate that causal LMs might derive positional awareness not only from the explicit positioning mechanism, but also from the effects of the causal mask.

Motivation & Objective

• To investigate whether transformer language models can learn positional information without explicit positional encodings.
• To evaluate the robustness of NoPos models across different model sizes, datasets, and sequence lengths.
• To analyze whether implicit positional awareness emerges in the hidden representations of NoPos models.
• To determine whether the causal attention mechanism is responsible for implicit positional learning.
• To contrast the behavior of autoregressive (causal) models with bidirectional (masked) models in the absence of positional encoding.

Proposed method

• Trained transformer language models without any positional encoding (NoPos) and compared them to models with sinusoidal, learned, and ALiBi positional encodings.
• Used probing classifiers to predict token positions from hidden representations across different network layers.
• Measured performance via validation set perplexity on WikiText-103 and a large-scale Pile corpus excerpt.
• Conducted ablation studies on masked language models (MLMs) to test whether bidirectional attention can learn positions without explicit encoding.
• Analyzed attention patterns to assess whether the causal mask enables models to infer the number of preceding tokens.
• Evaluated model behavior across multiple model sizes (125M to 1.3B parameters) and sequence lengths (up to 512 tokens).

Experimental results

Research questions

• RQ1 Can transformer language models learn absolute positional information without explicit positional encodings?
• RQ2 Does the causal attention mechanism enable implicit positional learning in autoregressive models?
• RQ3 How does the performance of NoPos models compare to position-aware models across different datasets and model sizes?
• RQ4 Why do bidirectional masked language models fail to learn positions without explicit positional encoding?
• RQ5 To what extent can hidden representations in NoPos models predict the true position of a token?

Key findings

• NoPos models achieve perplexity scores competitive with those of models using learned, sinusoidal, or ALiBi positional encodings across multiple datasets and model sizes.
• Probing experiments show that NoPos models learn a meaningful representation of absolute positions, with mean absolute prediction error comparable to models with learned positional embeddings.
• The causal attention mask enables models to implicitly infer the number of preceding tokens, effectively approximating absolute position.
• Bidirectional masked language models fail to converge when trained without positional encodings, indicating that causal attention is essential for implicit positional learning.
• The performance gap between NoPos and position-aware models narrows with increasing model size, suggesting that larger models may rely less on explicit positional inductive bias.
• NoPos models trained on the Pile corpus achieved a validation perplexity of 17.8, compared to 17.6 for the best baseline, indicating near-equal performance despite no explicit positional encoding.