[Paper Review] Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case

Neo Wu, Bradley Green+2 more|arXiv (Cornell University)|Jan 23, 2020

Anomaly Detection Techniques and ApplicationsComputer Science21 references349 citations

TL;DR

The paper introduces a Transformer-based approach for time series forecasting, evaluated on influenza-like illness (ILI) data, and compares against ARIMA, LSTM, and Seq2Seq models, showing competitive or superior performance.

ABSTRACT

In this paper, we present a new approach to time series forecasting. Time series data are prevalent in many scientific and engineering disciplines. Time series forecasting is a crucial task in modeling time series data, and is an important area of machine learning. In this work we developed a novel method that employs Transformer-based machine learning models to forecast time series data. This approach works by leveraging self-attention mechanisms to learn complex patterns and dynamics from time series data. Moreover, it is a generic framework and can be applied to univariate and multivariate time series data, as well as time series embeddings. Using influenza-like illness (ILI) forecasting as a case study, we show that the forecasting results produced by our approach are favorably comparable to the state-of-the-art.

Motivation & Objective

• Motivate time series forecasting as a critical task in health monitoring and disease surveillance.
• Propose a general Transformer-based framework for univariate and multivariate time series forecasting.
• Demonstrate that Transformers can model both observed data and phase-space dynamics via embeddings.
• Benchmark against ARIMA, LSTM, and Seq2Seq to establish performance on ILI forecasting.

Proposed method

• Use a Transformer architecture with an encoder and decoder to forecast multi-step ahead ILI ratios.
• Input mapping via a dense layer followed by positional encoding and four encoder/decoder layers.
• Train with look-ahead masking and a custom learning-rate schedule on 4-step predictions from a 10-step input window.
• Evaluate using Pearson correlation and RMSE, and compare to ARIMA, LSTM, and Seq2Seq baselines.
• Experiment with time delay embeddings (TDE) to capture phase-space information and assess effects on performance.

Experimental results

Research questions

• RQ1 Can a Transformer-based model match or exceed state-of-the-art ILI forecasting methods on official CDC data?
• RQ2 Does incorporating multivariate features (e.g., week number, first/second differences) improve Transformer forecasts for ILI?
• RQ3 Can time delay embeddings enhance Transformer performance by capturing phase-space structure in the data?
• RQ4 How does the Transformer compare to ARIMA, LSTM, and Seq2Seq with attention on one-step-ahead ILI forecasting?
• RQ5 Is a single global Transformer model capable of generalizing across US states for country-level ILI forecasting?

Key findings

• Transformer-based forecasting achieves high correlation (e.g., US-level: 0.984) and low RMSE (0.3318) for one-step-ahead ILI prediction.
• Transformer generally outperforms ARIMA in correlation and RMSE, and surpasses LSTM and Seq2Seq+attention in RMSE.
• Including week number and difference features yields modest improvements, suggesting self-attention captures dependencies without extra features.
• Time delay embeddings (TDE) with optimal dimensionality (8 in experiments) provide comparable or slightly improved performance over scalar inputs.
• Compared with ARGONet, the Transformer achieves similar or slightly better correlation and RMSE on average across states.
• A single global Transformer model trained on concatenated state data can generalize to country-level forecasting.