[Paper Review] Time Series Forecasting Using LSTM Networks: A Symbolic Approach

Steven Elsworth, Stefan Güttel|arXiv (Cornell University)|Mar 12, 2020

Time Series Analysis and ForecastingComputer Science37 references91 citations

TL;DR

The paper proposes ABBA symbolic representation of time series combined with LSTM (ABBA-LSTM) to speed training and reduce hyperparameter sensitivity while maintaining forecast performance.

ABSTRACT

Machine learning methods trained on raw numerical time series data exhibit fundamental limitations such as a high sensitivity to the hyper parameters and even to the initialization of random weights. A combination of a recurrent neural network with a dimension-reducing symbolic representation is proposed and applied for the purpose of time series forecasting. It is shown that the symbolic representation can help to alleviate some of the aforementioned problems and, in addition, might allow for faster training without sacrificing the forecast performance.

Motivation & Objective

• Motivate and address limitations of LSTM on raw time series data (sensitivity to hyperparameters and initialization).
• Introduce ABBA symbolic representation with a patching procedure to compress and reconstruct time series.
• Demonstrate that ABBA-LSTM can achieve comparable forecasts with faster training.
• Provide reproducible Python code and a comparative analysis with raw LSTM.

Proposed method

• Describe ABBA symbolic representation: compression into piecewise linear segments, digitization into k symbols, and patch-based reconstruction.
• Integrate ABBA with LSTM by training on symbolic sequences and using a final softmax layer for symbol prediction (ABBA-LSTM).
• Compare two training regimes (stateful vs stateless) for LSTM on time series forecasting.
• Use two-layer LSTM architectures with two initial layers of c cells, and tailor output layers for raw vs symbolic models.
• Evaluate with mean squared error for raw LSTM and categorical cross-entropy for ABBA-LSTM, with Adam optimizer and early stopping.

Experimental results

Research questions

• RQ1 Can ABBA symbolic representation reduce training time while preserving forecast accuracy compared to raw LSTM?
• RQ2 Does ABBA-LSTM exhibit reduced sensitivity to hyperparameters and initialization?
• RQ3 How does stateful versus stateless training impact ABBA-LSTM and raw LSTM performance?
• RQ4 What behavioral differences emerge in forecasts when using ABBA patching versus direct numeric reconstruction?

Key findings

• ABBA-LSTM trains more easily and can achieve forecast performance similar to raw LSTM.
• Symbolic ABBA representation reduces sensitivity to hyperparameters and initial weights.
• ABBA patching yields visually more appealing forecasts by constraining outputs to observed patch shapes.
• Experiments indicate stateless training often yields larger DTW distances than stateful training, with raw LSTM generally less robust.