[Paper Review] Rethinking Positional Encoding

Jianqiao Zheng, Sameera Ramasinghe+1 more|arXiv (Cornell University)|Jul 6, 2021

Multimodal Machine Learning ApplicationsComputer Science38 references30 citations

TL;DR

The paper generalizes positional encoding beyond Fourier features by using shifted continuous basis functions, linking performance to the stable rank of embeddings and distance preservation, and validates with Gaussian embedders.

ABSTRACT

It is well noted that coordinate based MLPs benefit -- in terms of preserving high-frequency information -- through the encoding of coordinate positions as an array of Fourier features. Hitherto, the rationale for the effectiveness of these positional encodings has been solely studied through a Fourier lens. In this paper, we strive to broaden this understanding by showing that alternative non-Fourier embedding functions can indeed be used for positional encoding. Moreover, we show that their performance is entirely determined by a trade-off between the stable rank of the embedded matrix and the distance preservation between embedded coordinates. We further establish that the now ubiquitous Fourier feature mapping of position is a special case that fulfills these conditions. Consequently, we present a more general theory to analyze positional encoding in terms of shifted basis functions. To this end, we develop the necessary theoretical formulae and empirically verify that our theoretical claims hold in practice. Codes available at https://github.com/osiriszjq/Rethinking-positional-encoding.

Motivation & Objective

• Expand the understanding of positional encoding beyond Fourier mappings.
• Propose and analyze embedding schemes based on shifted continuous basis functions.
• Characterize performance via stable rank and distance preservation of embedded coordinates.
• Demonstrate a practical Gaussian embedder as an effective alternative to Fourier features.

Proposed method

• Model positional encoding as systematic sampling of shifted basis functions.
• Define embedder functions and analyze their rank and distance preservation properties.
• Use stable rank as a practical surrogate for rank in empirical evaluations.
• Show Gaussian embedder as a controllable trade-off between memorization and generalization.
• Relate Gaussian embedder to Random Fourier Features and derive equivalence under certain conditions.
• Experiment with 1D and 2D signals to compare embedders.

Experimental results

Research questions

• RQ1 Can non-Fourier embedding functions be effectively used for positional encoding in coordinate-MLPs?
• RQ2 How do the stable rank of embeddings and distance preservation govern the performance of positional encoding?
• RQ3 What is the practical performance of Gaussian-based embedders relative to Fourier feature mappings and RFF?
• RQ4 How can high-dimensional embeddings be managed efficiently (e.g., via separable embedders) while preserving performance?
• RQ5 Do theoretical relationships between embedder bandwidth, stable rank, and distance preservation hold empirically?

Key findings

• Positional encoding performance is governed by the trade-off between approximate rank (stable rank) and distance preservation.
• Any continuous function can serve as a positional embedder under the rank-distance trade-off framework.
• Gaussian embedders deliver competitive performance with controlled stable rank and robust distance preservation.
• Gaussian embedders are more stable at low embedding dimensions compared to Random Fourier Features.
• Separability of embedders enables efficient scaling to higher-dimensional signals while maintaining performance.