[Paper Review] Long-term Forecasting with TiDE: Time-series Dense Encoder

Abhimanyu Das, Weihao Kong+4 more|arXiv (Cornell University)|Apr 17, 2023

Neural Networks and Applications157 citations

TL;DR

TiDE presents an MLP-based encoder-decoder for long-horizon forecasting that handles covariates, achieves competitive accuracy with Transformers, and is significantly faster. It includes a linear residual connection and a temporal decoder to incorporate future covariates.

ABSTRACT

Recent work has shown that simple linear models can outperform several Transformer based approaches in long term time-series forecasting. Motivated by this, we propose a Multi-layer Perceptron (MLP) based encoder-decoder model, Time-series Dense Encoder (TiDE), for long-term time-series forecasting that enjoys the simplicity and speed of linear models while also being able to handle covariates and non-linear dependencies. Theoretically, we prove that the simplest linear analogue of our model can achieve near optimal error rate for linear dynamical systems (LDS) under some assumptions. Empirically, we show that our method can match or outperform prior approaches on popular long-term time-series forecasting benchmarks while being 5-10x faster than the best Transformer based model.

Motivation & Objective

• Motivate the need for accurate long-term forecasting in multivariate time series with covariates.
• Propose a simple yet effective MLP-based encoder-decoder architecture (TiDE) that avoids self-attention and recurrence.
• Theoretically analyze a linear analogue showing near-optimal error rates for linear dynamical systems.
• Empirically compare TiDE against state-of-the-art Transformer-based and linear models on standard benchmarks.
• Demonstrate improved training and inference efficiency while maintaining competitive accuracy.

Proposed method

• TiDE encodes past observations and covariates using a dense MLP encoder after a feature projection step that reduces dynamic covariate dimensionality.
• A dense MLP decoder generates horizon-specific representations, followed by a temporal decoder that fuses these representations with future covariates to produce predictions.
• A global linear residual connection from the look-back to the horizon is added to ensure linear models are a subclass of TiDE.
• TemporalDecoder hyperparameters control the hidden size used for processing horizon steps.
• Training uses mini-batch gradient descent with MSE loss and rolling evaluation on a standardized look-back/horizon setup.

Figure 1 : Overview of TiDE architecture. The dynamic covariates per time-point are mapped to a lower dimensional space using a feature projection step. Then the encoder combines the look-back along with the projected covariates with the static attributes to form an encoding. The decoder maps this e

Experimental results

Research questions

• RQ1 Can an ML P-based Dense Encoder (TiDE) match or exceed Transformer-based approaches on long-horizon multivariate forecasting benchmarks?
• RQ2 How does TiDE perform when covariates (dynamic and static) are available, and can it leverage them effectively without self-attention?
• RQ3 Does the linear analogue of TiDE achieve near-optimal error for linear dynamical systems under reasonable spectral conditions?
• RQ4 What are the practical trade-offs in accuracy, training time, and inference time between TiDE and Transformer-based models across standard datasets?

Key findings

• TiDE achieves comparable or superior MSE on popular long-term forecasting benchmarks compared to state-of-the-art Transformer-based models.
• TiDE is 5x faster in inference and more than 10x faster in training than the best Transformer-based model on large datasets like Traffic.
• Incorporating future covariates via the temporal decoder provides tangible gains, particularly when covariates have direct effects at specific horizon steps.
• A linear analogue of TiDE can be near-optimal for linear dynamical systems, providing theoretical support for the effectiveness of simple linear/bias-enabled architectures in long-horizon settings.
• TiDE effectively handles both dynamic covariates and static attributes, outperforming baselines in the M5 demand forecasting benchmark when covariates are used.

(a) Inference time per batch in microseconds