[Paper Review] Long-Range Transformers for Dynamic Spatiotemporal Forecasting

Jake Grigsby, Zhe Wang+2 more|arXiv (Cornell University)|Sep 24, 2021

Time Series Analysis and Forecasting56 references78 citations

TL;DR

The paper introduces Spacetimeformer, a spatiotemporal Transformer that treats each variable at each time as a separate input token, learns dynamic variable relationships without predefined graphs, and achieves competitive results across traffic, electricity, weather, and metro ridership tasks.

ABSTRACT

Multivariate time series forecasting focuses on predicting future values based on historical context. State-of-the-art sequence-to-sequence models rely on neural attention between timesteps, which allows for temporal learning but fails to consider distinct spatial relationships between variables. In contrast, methods based on graph neural networks explicitly model variable relationships. However, these methods often rely on predefined graphs that cannot change over time and perform separate spatial and temporal updates without establishing direct connections between each variable at every timestep. Our work addresses these problems by translating multivariate forecasting into a "spatiotemporal sequence" formulation where each Transformer input token represents the value of a single variable at a given time. Long-Range Transformers can then learn interactions between space, time, and value information jointly along this extended sequence. Our method, which we call Spacetimeformer, achieves competitive results on benchmarks from traffic forecasting to electricity demand and weather prediction while learning spatiotemporal relationships purely from data.

Motivation & Objective

• Motivate multivariate forecasting and the limitations of single-token-per-timestep Transformers in capturing spatial relationships.
• Propose a spatiotemporal sequence formulation where each token represents a variable at a given time.
• Develop a Spacetimeformer encoder–decoder that learns dynamic variable relationships via spatiotemporal attention.
• Demonstrate competitive performance across benchmarks from traffic, electricity, and weather to metro ridership.
• Open-source codebase with datasets and model implementations for future research.

Proposed method

• Flatten multivariate inputs into a long sequence of tokens, each token representing a single variable at a given time.
• Embed time and variable information with Time2Vec and learned time/variable embeddings, plus position embeddings.
• Employ spatiotemporal attention over the extended token sequence to learn dynamic spatial and temporal interactions.
• Incorporate local attention biases (per-variable) alongside global spatiotemporal attention within a Transformer encoder–decoder architecture.
• Use scalable attention mechanisms (e.g., Performer FAVOR+) and optional convolutions/shifted windows to handle long sequences.
• Handle missing data with a dedicated embedding strategy that signals when values are originally missing.

Experimental results

Research questions

• RQ1 Can Spacetimeformer match or exceed seq2seq models on long-horizon forecasting tasks?
• RQ2 Can the model compete with graph-based spatial models without reliance on predefined graphs?
• RQ3 Does the spatiotemporal sequence formulation enable learning meaningful variable relationships directly from data?
• RQ4 How well does the approach scale to long sequences and high-dimensional spatial problems?

Key findings

• Spacetimeformer achieves competitive results across benchmarks from traffic forecasting to electricity production, temperature prediction, and metro ridership.
• It performs well against graph-based methods without requiring predefined spatial graphs, and against standard TSF baselines.
• Attention patterns reveal learned variable relationships that align with ground-truth spatial structures in qualitative analyses.
• The model scales to long spatiotemporal sequences using fast attention and architectural biases, while handling missing data through dedicated embeddings.