[Paper Review] Graph Attention Recurrent Neural Networks for Correlated Time Series Forecasting -- Full version

Razvan-Gabriel Cirstea, Chenjuan Guo+1 more|arXiv (Cornell University)|Mar 19, 2021

Time Series Analysis and Forecasting36 references4 citations

TL;DR

This paper proposes Graph Attention Recurrent Neural Networks (GA-RNNs) to improve correlated time series forecasting by learning dynamic, adaptive adjacency matrices via multi-head attention on a spatial proximity-based graph. The method integrates these time-varying attention weights into RNNs to model evolving spatio-temporal dependencies, significantly outperforming state-of-the-art methods, especially in short-term predictions on real-world traffic speed data.

ABSTRACT

We consider a setting where multiple entities inter-act with each other over time and the time-varying statuses of the entities are represented as multiple correlated time series. For example, speed sensors are deployed in different locations in a road network, where the speed of a specific location across time is captured by the corresponding sensor as a time series, resulting in multiple speed time series from different locations, which are often correlated. To enable accurate forecasting on correlated time series, we proposes graph attention recurrent neural networks.First, we build a graph among different entities by taking into account spatial proximity and employ a multi-head attention mechanism to derive adaptive weight matrices for the graph to capture the correlations among vertices (e.g., speeds at different locations) at different timestamps. Second, we employ recurrent neural networks to take into account temporal dependency while taking into account the adaptive weight matrices learned from the first step to consider the correlations among time series.Experiments on a large real-world speed time series data set suggest that the proposed method is effective and outperforms the state-of-the-art in most settings. This manuscript provides a full version of a workshop paper [1].

Motivation & Objective

• To address the limitation of static graph structures in modeling dynamic spatio-temporal correlations among correlated time series.
• To improve forecasting accuracy in complex cyber-physical systems, such as intelligent transportation systems, by capturing evolving interactions between entities.
• To develop a generic framework that integrates adaptive attention-based adjacency matrices with graph-based RNNs for enhanced spatio-temporal modeling.
• To enable more robust and accurate predictions by replacing fixed adjacency matrices with attention-derived, time-varying ones in RNN architectures.

Proposed method

• A graph is constructed based on spatial proximity between entities (e.g., speed sensors), with vertices representing entities and edges connecting nearby ones.
• Multi-head self-attention is applied on the graph to learn an attention matrix that adaptively assigns importance to neighboring vertices at each timestamp.
• The learned attention matrix replaces the static adjacency matrix in graph-based RNNs, creating dynamic, time-varying adjacency matrices that reflect evolving correlations.
• Recurrent neural networks (e.g., GRU or LSTM) are modified to incorporate these adaptive adjacency matrices via graph convolution or diffusion convolution operations.
• The model is trained end-to-end to predict future time series values over a P-step horizon using historical data from a window of L timestamps.
• The approach is generic and can be integrated with any graph convolution method relying on adjacency matrices, such as GCN or DCNN.

Experimental results

Research questions

• RQ1 Can attention mechanisms effectively learn dynamic, time-varying relationships between correlated time series in a graph-structured setting?
• RQ2 How does replacing a static adjacency matrix with an attention-derived, adaptive one improve forecasting performance in RNN-based models?
• RQ3 To what extent does the proposed method outperform state-of-the-art models in short- and long-term correlated time series forecasting?
• RQ4 Can the attention mechanism capture meaningful, context-dependent interactions that reflect real-world dynamics, such as traffic congestion spreading between roads?

Key findings

• The proposed GA-RNN model outperforms state-of-the-art methods on a large real-world traffic speed dataset, particularly in short-term forecasting tasks.
• The integration of graph attention with RNNs leads to more accurate predictions by capturing dynamic, evolving correlations among time series.
• The attention mechanism introduces additional computational cost, with inference taking approximately 401 seconds, compared to 350 seconds for the baseline GCRNN.
• The model demonstrates robustness and generalization, especially when modeling complex, non-linear spatio-temporal dependencies in real-world systems.
• The use of adaptive adjacency matrices significantly improves performance over models using fixed, pre-computed graphs, highlighting the importance of dynamic interaction modeling.