[Paper Review] Learnable Fourier Features for Multi-Dimensional Spatial Positional Encoding

Yang Li, Si Si+3 more|arXiv (Cornell University)|Jun 5, 2021

Advanced Image and Video Retrieval Techniques44 references30 citations

TL;DR

Introduces a learnable Fourier feature-based positional encoding for multi-dimensional spatial data, integrated with an MLP, enabling inductive, scalable, and distance-preserving position representations for Transformer-based models across images and UI-like structures. Demonstrates improved convergence and accuracy on multiple vision and UI tasks.

ABSTRACT

Attentional mechanisms are order-invariant. Positional encoding is a crucial component to allow attention-based deep model architectures such as Transformer to address sequences or images where the position of information matters. In this paper, we propose a novel positional encoding method based on learnable Fourier features. Instead of hard-coding each position as a token or a vector, we represent each position, which can be multi-dimensional, as a trainable encoding based on learnable Fourier feature mapping, modulated with a multi-layer perceptron. The representation is particularly advantageous for a spatial multi-dimensional position, e.g., pixel positions on an image, where $L_2$ distances or more complex positional relationships need to be captured. Our experiments based on several public benchmark tasks show that our learnable Fourier feature representation for multi-dimensional positional encoding outperforms existing methods by both improving the accuracy and allowing faster convergence.

Motivation & Objective

• Motivate the need for effective, scalable positional encoding in multi-dimensional spatial domains (e.g., images, UI layouts) for Transformer-based models.
• Propose a learnable Fourier feature-based positional encoding that captures Euclidean-like distances and complex spatial relations.
• Show that the proposed encoding is inductive, parameter-efficient, and scalable to unseen positions and higher dimensions.
• Demonstrate improved accuracy and faster convergence over established PE methods across image generation, object detection, image classification, and UI widget captioning.

Proposed method

• Represent multi-dimensional positions x in R^M via learnable Fourier features r_x with r_x = (1/sqrt{D}) [ cos(x W_r^T) || sin(x W_r^T) ], where W_r is trainable and initialized from N(0, gamma^{-2}).
• Compute a shift-invariant dot product r_x · r_y that approximates a Gaussian kernel over positions (k(x,y) ≈ exp(-||x-y||^2 / gamma^2)).
• Pass the Fourier features through a multi-layer perceptron (phi) and a linear projection to produce the final positional embedding PE_x = phi(r_x, theta) W_p.
• Handle multi-dimensional positions holistically by grouping coordinates and applying the same encoding pipeline to each group, then concatenating results.
• The approach is inductive (handles unseen positions) and parameter-efficient (does not scale with sequence length).
• Integrate the encoding with Transformer-based models by adding the generated PE_X to content embeddings in downstream attention computations.

Experimental results

Research questions

• RQ1 Can learnable Fourier feature-based positional encodings capture meaningful multi-dimensional spatial relationships (e.g., 2D Euclidean distances) better than fixed sinusoidal or discrete embeddings?
• RQ2 Do the proposed encodings improve convergence speed and accuracy for Transformer-based tasks involving spatial data (images, object detection, UI layouts) and generalize to unseen positions/sizes?
• RQ3 Is the combination Learnable-Fourier Features + MLP more effective than using Fourier features or MLP alone across diverse tasks?
• RQ4 How does the multi-group (partitioned) encoding strategy affect performance in high-dimensional spatial settings like UI widget bounding boxes?

Key findings

• The Learnable-Fourier + MLP encoding consistently outperformed baseline positional encodings across image generation, object detection, image classification, and widget captioning benchmarks.
• The combination of learnable Fourier features with an MLP yields faster convergence and higher accuracy than using either component alone.
• For unseen image sizes and positions, the Learnable-Fourier + MLP method generalizes better than discrete embedding or sinusoidal approaches, reducing performance gaps on out-of-distribution positions.
• Partitioning multi-dimensional positions into groups and encoding each group with shared Fourier features can model more complex spatial relationships than simple L2 distance, benefiting tasks like UI widget captioning.
• In image classification with Vision Transformer, Learnable-Fourier + MLP achieved higher top-1 accuracy than Embed-1D, demonstrating practical gains on standard benchmarks.