[Paper Review] Time Series Forecasting Based on Augmented Long Short-Term Memory

Daniel Hsu|arXiv (Cornell University)|Jul 3, 2017

Time Series Analysis and ForecastingComputer Science16 references34 citations

TL;DR

This paper proposes an augmented Long Short-Term Memory (LSTM) model using a hierarchical latent variable framework with Bayesian neural networks and Horseshoe priors to improve multi-period time series forecasting. By modeling temporal dependencies across multiple resolutions and automatically pruning redundant inputs, the method achieves superior prediction accuracy on both synthetic and real-world datasets.

ABSTRACT

In this paper, we use augmented the hierarchical latent variable model to model multi-period time series, where the dynamics of time series are governed by factors or trends in multiple periods. Previous methods based on stacked recurrent neural network (RNN) and deep belief network (DBN) models cannot model the tendencies in multiple periods, and no models for sequential data pay special attention to redundant input variables which have no or even negative impact on prediction and modeling. Applying hierarchical latent variable model with multiple transition periods, our proposed algorithm can capture dependencies in different temporal resolutions. Introducing Bayesian neural network with Horseshoe prior as input network, we can discard the redundant input variables in the optimization process, concurrently with the learning of other parts of the model. Based on experiments with both synthetic and real-world data, we show that the proposed method significantly improves the modeling and prediction performance on multi-period time series.

Motivation & Objective

• To address the limitation of existing RNN and DBN models in capturing multi-period temporal trends.
• To model time series dynamics using hierarchical latent variables across multiple transition periods.
• To reduce noise from redundant or irrelevant input variables through Bayesian regularization with Horseshoe priors.
• To improve forecasting performance by jointly learning model parameters and identifying non-informative inputs.

Proposed method

• The model employs a hierarchical latent variable structure to represent time series dynamics across multiple temporal resolutions.
• It integrates a Bayesian neural network with Horseshoe prior to perform automatic input variable selection during optimization.
• The Horseshoe prior enables shrinkage of irrelevant input weights toward zero, effectively discarding redundant features.
• The architecture uses stacked LSTM layers to capture long-term dependencies in sequential data.
• The model is trained end-to-end using stochastic gradient descent with variational inference for approximate posterior estimation.
• Temporal dependencies across different periods are explicitly modeled through structured transition matrices in the latent space.

Experimental results

Research questions

• RQ1 Can a hierarchical latent variable model effectively capture multi-period temporal dependencies in time series data?
• RQ2 How does incorporating Horseshoe priors improve model robustness by eliminating redundant input variables?
• RQ3 Does the proposed method outperform standard RNN and DBN models in multi-period forecasting tasks?
• RQ4 To what extent does the Bayesian regularization enhance generalization on real-world time series?

Key findings

• The proposed method significantly improves prediction accuracy on both synthetic and real-world time series datasets compared to baseline RNN and DBN models.
• The Horseshoe prior successfully identifies and suppresses irrelevant input variables during training, enhancing model interpretability.
• The hierarchical latent structure captures long-range dependencies across multiple temporal resolutions more effectively than standard RNNs.
• The model demonstrates robust performance even when input data contains noisy or irrelevant features.
• Empirical results show consistent improvement in forecasting metrics such as RMSE and MAE across diverse time series benchmarks.