[Paper Review] Learning to Encode Position for Transformer with Continuous Dynamical Model

Xuanqing Liu, Hsiang‐Fu Yu+2 more|arXiv (Cornell University)|Mar 12, 2020

Natural Language Processing TechniquesComputer Science56 citations

TL;DR

The paper introduces FLOATER, a flow-based continuous dynamical position encoder for Transformers, enabling inductive, data-driven, and parameter-efficient position encoding that improves MT, language understanding, and QA tasks.

ABSTRACT

We introduce a new way of learning to encode position information for non-recurrent models, such as Transformer models. Unlike RNN and LSTM, which contain inductive bias by loading the input tokens sequentially, non-recurrent models are less sensitive to position. The main reason is that position information among input units is not inherently encoded, i.e., the models are permutation equivalent; this problem justifies why all of the existing models are accompanied by a sinusoidal encoding/embedding layer at the input. However, this solution has clear limitations: the sinusoidal encoding is not flexible enough as it is manually designed and does not contain any learnable parameters, whereas the position embedding restricts the maximum length of input sequences. It is thus desirable to design a new position layer that contains learnable parameters to adjust to different datasets and different architectures. At the same time, we would also like the encodings to extrapolate in accordance with the variable length of inputs. In our proposed solution, we borrow from the recent Neural ODE approach, which may be viewed as a versatile continuous version of a ResNet. This model is capable of modeling many kinds of dynamical systems. We model the evolution of encoded results along position index by such a dynamical system, thereby overcoming the above limitations of existing methods. We evaluate our new position layers on a variety of neural machine translation and language understanding tasks, the experimental results show consistent improvements over the baselines.

Motivation & Objective

• Motivate the need for learnable, inductive position encodings in non-recurrent Transformers.
• Propose FLOATER, a continuous dynamical system to generate position encodings.
• Ensure FLOATER is data-driven, parameter-efficient, and compatible with standard Transformer architectures.
• Demonstrate FLOATER's improvements across MT, language understanding, and QA benchmarks.

Proposed method

• Model position encodings as a continuous dynamical system p(t) driven by a neural network h(t, p(t); θ_h).
• Discretize p(i) via evaluation at increasing t_i with a fixed Δt to obtain position vectors for each token.
• Share dynamics h(·) across Transformer blocks to reduce parameters while allowing different initial p(0) per block.
• Show FLOATER degenerates to the original sinusoidal encoding when h(·)=0 for compatibility.
• Optionally inject the dynamical encoding at all Transformer blocks for improved performance.
• Provide warm-start strategies by initializing FLOATER from pretrained Transformers and fine-tuning.

Experimental results

Research questions

• RQ1 Can a continuous dynamical system for position encoding provide inductive, data-driven, and parameter-efficient enhancements over fixed sinusoidal or layer-wise embeddings?
• RQ2 How does FLOATER perform across neural machine translation, language understanding, and question answering tasks compared to baselines?
• RQ3 What is the impact of applying FLOATER at all blocks versus only at the input block?
• RQ4 How compatible is FLOATER with pre-trained Transformer models and how does warm-start training affect performance?

Key findings

• FLOATER achieves consistent improvements over baselines on MT, GLUE, RACE, and SQuAD tasks.
• Using FLOATER at all Transformer blocks yields better performance than applying it only at the input block.
• FLOATER remains compatible with vanilla Transformer by reducing to sinusoidal encoding when h(·)=0, enabling warm-start from pretrained models.
• On WMT En-De, FLOATER achieves BLEU of 28.57 with 526.3K parameters, outperforming several RNN-based encoders with varying parameter budgets.
• FLOATER demonstrates inductive behavior, performing well on longer sequences not seen during training, especially in MT.
• Training FLOATER incurs overhead, but with warm-start and parameter-sharing strategies, overhead can be kept modest (~20-30%); inference overhead is avoided by storing positional biases.