[Paper Review] Multi-variable LSTM neural network for autoregressive exogenous model

Tian Guo, Tao Lin|arXiv (Cornell University)|Jun 17, 2018

Stock Market Forecasting Methods25 references15 citations

TL;DR

This paper proposes a multi-variable LSTM (MV-LSTM) with tensorized hidden states to enable joint forecasting and variable importance interpretation in autoregressive exogenous (ARX) time series models. By introducing a mixture of temporal and variable-wise attention mechanisms, MV-LSTM quantifies variable importance through posterior inference, outperforming baselines in both prediction accuracy and interpretability on real and synthetic datasets.

ABSTRACT

In this paper, we propose multi-variable LSTM capable of accurate forecasting and variable importance interpretation for time series with exogenous variables. Current attention mechanism in recurrent neural networks mostly focuses on the temporal aspect of data and falls short of characterizing variable importance. To this end, the multi-variable LSTM equipped with tensorized hidden states is developed to learn hidden states for individual variables, which give rise to our mixture temporal and variable attention. Based on such attention mechanism, we infer and quantify variable importance. Extensive experiments using real datasets with Granger-causality test and the synthetic dataset with ground truth demonstrate the prediction performance and interpretability of multi-variable LSTM in comparison to a variety of baselines. It exhibits the prospect of multi-variable LSTM as an end-to-end framework for both forecasting and knowledge discovery.

Motivation & Objective

• To address the lack of variable-level interpretability in standard RNNs like LSTM when modeling multivariate time series with exogenous variables.
• To develop a unified end-to-end framework that simultaneously enables accurate forecasting and knowledge discovery through variable importance quantification.
• To overcome the limitation of existing attention mechanisms, which focus only on temporal attention and fail to capture variable-specific contributions.
• To design a tensorized hidden state structure that enables individual variable representation within the LSTM cell for improved interpretability.
• To validate the model’s performance and interpretability using real-world datasets with Granger causality tests and synthetic data with ground truth

Proposed method

• Introduce a multi-variable LSTM (MV-LSTM) with tensorized hidden states, where each element of the hidden state tensor encodes information specific to a particular input variable.
• Develop a mixture attention mechanism that jointly models temporal attention (across time steps) and variable attention (across input variables) using the variable-specific hidden states.
• Formulate a probabilistic posterior inference framework to quantify variable importance based on the learned variable attention weights.
• Use the posterior attention distribution to derive interpretable importance scores, which are more distinguishable than prior or unnormalized attention weights.
• Train the model end-to-end using a combination of prediction loss and attention regularization to improve both forecasting accuracy and interpretability.
• Apply the model to real datasets (PM2.5, Plant, Energy) and a synthetic dataset with known ground truth to evaluate performance and interpretability

Experimental results

Research questions

• RQ1 Can MV-LSTM achieve superior forecasting performance compared to standard RNNs, statistical models, and deep learning baselines on multivariate time series with exogenous variables?
• RQ2 Can the proposed mixture attention mechanism effectively capture and quantify variable importance at the input level, beyond just temporal attention?
• RQ3 Does the posterior inference of variable attention in MV-LSTM lead to more accurate and distinguishable variable importance rankings than prior or unnormalized attention?
• RQ4 Can MV-LSTM correctly identify Granger-causal variables in real-world datasets, as validated by statistical tests?
• RQ5 In synthetic data with known ground truth, can MV-LSTM recover the true importance of input variables, especially those with known causal influence?

Key findings

• MV-LSTM achieved the best prediction performance on all real datasets, with a test RMSE of 132.54 ± 0.21 on the PM2.5 dataset, outperforming all baselines.
• On the synthetic dataset, MV-LSTM correctly identified variables 2 and 3 as top contributors, aligning with ground truth, while baselines failed to recover the correct importance order.
• In the PM2.5 dataset, MV-LSTM correctly ranked dew point, cumulated wind speed, and pressure as top three important variables, consistent with domain knowledge and Granger causality results.
• The posterior attention mechanism in MV-LSTM shifted attention toward more relevant variables (e.g., dew point) and suppressed irrelevant ones (e.g., cumulated hours of rain), improving interpretability over prior or unnormalized attention.
• Compared to DUAL and RETAIN, MV-LSTM produced more distinguishable and meaningful variable importance scores, as shown in histogram visualizations of attention weights.
• cLSTM, which relies on sparse regularization for variable importance, failed to identify the correct variables in both synthetic and real datasets, highlighting the advantage of MV-LSTM’s attention-based interpretability