[Paper Review] Machine Learning for Spatiotemporal Sequence Forecasting: A Survey

Xingjian Shi, Dit‐Yan Yeung|arXiv (Cornell University)|Aug 21, 2018

Traffic Prediction and Management Techniques70 references78 citations

TL;DR

A systematic survey of machine learning methods for spatiotemporal sequence forecasting (STSF), categorizing problems and reviewing both classical and deep learning approaches.

ABSTRACT

Spatiotemporal systems are common in the real-world. Forecasting the multi-step future of these spatiotemporal systems based on the past observations, or, Spatiotemporal Sequence Forecasting (STSF), is a significant and challenging problem. Although lots of real-world problems can be viewed as STSF and many research works have proposed machine learning based methods for them, no existing work has summarized and compared these methods from a unified perspective. This survey aims to provide a systematic review of machine learning for STSF. In this survey, we define the STSF problem and classify it into three subcategories: Trajectory Forecasting of Moving Point Cloud (TF-MPC), STSF on Regular Grid (STSF-RG) and STSF on Irregular Grid (STSF-IG). We then introduce the two major challenges of STSF: 1) how to learn a model for multi-step forecasting and 2) how to adequately model the spatial and temporal structures. After that, we review the existing works for solving these challenges, including the general learning strategies for multi-step forecasting, the classical machine learning based methods for STSF, and the deep learning based methods for STSF. We also compare these methods and point out some potential research directions.

Motivation & Objective

• Define STSF and its subproblems: TF-MPC, STSF-RG, and STSF-IG.
• Identify core challenges in multi-step forecasting and modeling spatiotemporal structure.
• Review and contrast classical and deep learning methods for STSF.
• Summarize learning strategies (IMS, DMS, boosting, scheduled sampling) and guide future research directions.

Proposed method

• Classify STSF problems by coordinate and measurement characteristics (TF-MPC, STSF-RG, STSF-IG).
• Survey learning strategies for multi-step forecasting: Iterative Multi-step (IMS), Direct Multi-step (DMS), boosting, and scheduled sampling.
• Review classical methods (feature-based, state-space models, Gaussian processes) with architectural adaptations for STSF.
• Review deep learning approaches including temporal generative models and feed-forward/recurrent networks for STSF.

Experimental results

Research questions

• RQ1 What are the canonical subcategories of STSF and their distinctive challenges?
• RQ2 How do IMS and DMS compare for multi-step STSF, and what strategies bridge their gaps?
• RQ3 What classical methods adapt to STSF, and what are their limitations?
• RQ4 What deep learning architectures are most effective for capturing spatiotemporal dependencies in STSF?
• RQ5 What future research directions are suggested for STSF?

Key findings

• STSF problems split into three categories: TF-MPC (changing coordinates), STSF-RG (regular grid), and STSF-IG (irregular grid).
• Two main challenges are learning multi-step forecasts and modeling spatial-temporal structures to manage high-dimensional data.
• IMS is easy to train and suited for long horizons, but accumulates errors; DMS can be more accurate for shorter horizons but is computationally heavier.
• Boosting and scheduled sampling are effective hybrids bridging IMS and DMS, enhancing prediction quality in practice.
• Classical methods (feature-based, SSMs, GPs) require task-specific engineering and have limitations in nonlinear or large-scale STSF; deep learning methods offer more flexible modeling of complex STSF relationships.