[Paper Review] Structured Time Series Prediction without Structural Prior

Darko Drakulić, Jean‐Marc Andreoli|arXiv (Cornell University)|Feb 7, 2022

Traffic Prediction and Management Techniques4 citations

TL;DR

This paper proposes ADN, a data-driven attention-based model for structured time series prediction that eliminates reliance on structural priors like graphs. By leveraging learnable multi-head attention over spatial and temporal dimensions, ADN achieves state-of-the-art performance with minimal data and excels in missing data handling and few-shot domain adaptation, outperforming graph-based models with only two days of fine-tuning data.

ABSTRACT

Time series prediction is a widespread and well studied problem with applications in many domains (medical, geoscience, network analysis, finance, econometry etc.). In the case of multivariate time series, the key to good performances is to properly capture the dependencies between the variates. Often, these variates are structured, i.e. they are localised in an abstract space, usually representing an aspect of the physical world, and prediction amounts to a form of diffusion of the information across that space over time. Several neural network models of diffusion have been proposed in the literature. However, most of the existing proposals rely on some a priori knowledge on the structure of the space, usually in the form of a graph weighing the pairwise diffusion capacity of its points. We argue that this piece of information can often be dispensed with, since data already contains the diffusion capacity information, and in a more reliable form than that obtained from the usually largely hand-crafted graphs. We propose instead a fully data-driven model which does not rely on such a graph, nor any other prior structural information. We conduct a first set of experiments to measure the impact on performance of a structural prior, as used in baseline models, and show that, except at very low data levels, it remains negligible, and beyond a threshold, it may even become detrimental. We then investigate, through a second set of experiments, the capacity of our model in two respects: treatment of missing data and domain adaptation.

Motivation & Objective

• To address the limitations of structural priors (e.g., hand-crafted graphs) in multivariate time series forecasting, which can hinder performance and generalization.
• To develop a model that learns dependencies purely from data, without assuming prior knowledge of spatial or temporal relationships.
• To evaluate the impact of structural priors on model performance and demonstrate that they can be detrimental when sufficient data is available.
• To investigate the model's robustness in handling missing data and its effectiveness in few-shot domain adaptation.

Proposed method

• The ADN model uses a standard encoder-decoder architecture with multi-head attention mechanisms applied sequentially to the spatial and temporal dimensions of event grids (locations × time points).
• Attention is applied by reorganizing tensor dimensions: spatial (N) and temporal (T) dimensions are alternately merged into the batch dimension (B) to enable attention across each dimension independently.
• The model uses no positional encoding; instead, the causal mask in decoder self-attention enforces temporal order, ensuring predictions depend only on past events.
• For domain adaptation, only the location embeddings in the target domain are retrained, while all other parameters (including attention weights and time embeddings) are kept fixed.
• The model is trained end-to-end using standard regression loss on short-term prediction tasks, with no inductive bias on structure.
• The architecture is lightweight and fully differentiable, enabling efficient fine-tuning on small target-domain data.

Experimental results

Research questions

• RQ1 Does the use of a structural prior (e.g., a graph) significantly improve time series prediction performance when sufficient training data is available?
• RQ2 Can a data-driven model without any structural prior match or exceed the performance of state-of-the-art models that rely on such priors?
• RQ3 How well does the proposed model handle missing data in multivariate time series?
• RQ4 To what extent can the model generalize to new domains with minimal labeled data from the target domain?
• RQ5 Can the model learn temporal patterns such as cyclic daily rhythms without explicit positional encoding?

Key findings

• The performance of models relying on structural priors diminishes or becomes detrimental beyond a certain data threshold, indicating that such priors are not essential when data is abundant.
• ADN achieves comparable or better performance than graph-based models like STGAT even without any structural prior, especially when fine-tuned on just a few days of target data.
• With only two days of fine-tuning data, ADN matches the performance of STGAT in zero-shot adaptation (no fine-tuning), and continues to improve beyond that point.
• The model learns the cyclic nature of daily time patterns without positional encoding, relying solely on attention mechanisms to infer temporal order.
• In domain adaptation, only the location embeddings in the target domain need to be retrained, enabling efficient and effective transfer with minimal parameter updates.
• ADN demonstrates superior robustness in handling missing data, as it learns dependencies directly from observed data patterns rather than relying on a fixed graph structure.