[Paper Review] A Memory-Network Based Solution for Multivariate Time-Series Forecasting

Yen‐Yu Chang, Fan-Yun Sun+2 more|arXiv (Cornell University)|Sep 6, 2018

Time Series Analysis and Forecasting24 references54 citations

TL;DR

MTNet introduces a memory-augmented neural network for multivariate time-series forecasting, using a memory module and three encoders with an autoregressive component to capture long-term patterns and periodicity with interpretable attention.

ABSTRACT

Multivariate time series forecasting is extensively studied throughout the years with ubiquitous applications in areas such as finance, traffic, environment, etc. Still, concerns have been raised on traditional methods for incapable of modeling complex patterns or dependencies lying in real word data. To address such concerns, various deep learning models, mainly Recurrent Neural Network (RNN) based methods, are proposed. Nevertheless, capturing extremely long-term patterns while effectively incorporating information from other variables remains a challenge for time-series forecasting. Furthermore, lack-of-explainability remains one serious drawback for deep neural network models. Inspired by Memory Network proposed for solving the question-answering task, we propose a deep learning based model named Memory Time-series network (MTNet) for time series forecasting. MTNet consists of a large memory component, three separate encoders, and an autoregressive component to train jointly. Additionally, the attention mechanism designed enable MTNet to be highly interpretable. We can easily tell which part of the historic data is referenced the most.

Motivation & Objective

• Motivate improved multivariate time-series forecasting by addressing long-term dependencies and cross-variable interactions.
• Propose MTNet, a memory-network based model with a memory module, three encoders, and an autoregressive component.
• Enable interpretability via block-wise attention over memory sections to reveal historic patterns influencing predictions.
• Validate MTNet against state-of-the-art baselines on diverse univariate and multivariate datasets.

Proposed method

• Memory component stores long-term history and is attended to by input short-term representations.
• Three independent encoders produce embedding maps for memory, input, and memory output.
• Attention weights are computed via inner products between input and memory embeddings and then applied to output embeddings.
• An autoregressive (AR) linear component is combined with a neural network output for final prediction.
• The final prediction is the sum of the neural network output and the AR component.
• Training uses mean absolute error (L1 loss) with Adam optimizer.

Experimental results

Research questions

• RQ1 Can a memory-augmented architecture better capture long-term and periodic dependencies in multivariate time-series data compared to RNN-based approaches?
• RQ2 Does attending over memory chunks (as opposed to only past timestamps) improve predictive accuracy and interpretability across datasets with periodic patterns?
• RQ3 How does MTNet perform relative to established baselines (RNN-GRU, DA-RNN, LSTNet, etc.) on both univariate and multivariate tasks?

Key findings

• MTNet outperforms competing models on both univariate and multivariate benchmarks across multiple horizons.
• The memory-augmented approach provides superior handling of long-term and periodic patterns, improving accuracy over DA-RNN and LSTNet baselines.
• Block-wise attention over memory enables visualization of which historical periods most influence a forecast, showcasing interpretability.
• Integration of AR in parallel with neural components enhances robustness across datasets.
• MTNet demonstrates clear gains on Beijing PM2.5, GefCom2014 Electricity Price, Traffic, Solar-Energy, Electricity, and Exchange-Rate datasets compared to multiple baselines.