[Paper Review] Are Self-Attentions Effective for Time Series Forecasting?

Dongbin Kim, J.-G. Park+2 more|arXiv (Cornell University)|May 27, 2024

Forecasting Techniques and ApplicationsDecision Sciences3 citations

TL;DR

This paper proposes CATS, a novel time series forecasting model that replaces self-attention with cross-attention by treating future forecasting horizons as learnable queries and past sequences as keys/values. By eliminating self-attention and enforcing horizon-specific parameter sharing, CATS achieves state-of-the-art performance with lower MSE and significantly fewer parameters than existing Transformer-based models across multiple datasets.

ABSTRACT

Time series forecasting is crucial for applications across multiple domains and various scenarios. Although Transformer models have dramatically advanced the landscape of forecasting, their effectiveness remains debated. Recent findings have indicated that simpler linear models might outperform complex Transformer-based approaches, highlighting the potential for more streamlined architectures. In this paper, we shift the focus from evaluating the overall Transformer architecture to specifically examining the effectiveness of self-attention for time series forecasting. To this end, we introduce a new architecture, Cross-Attention-only Time Series transformer (CATS), that rethinks the traditional Transformer framework by eliminating self-attention and leveraging cross-attention mechanisms instead. By establishing future horizon-dependent parameters as queries and enhanced parameter sharing, our model not only improves long-term forecasting accuracy but also reduces the number of parameters and memory usage. Extensive experiment across various datasets demonstrates that our model achieves superior performance with the lowest mean squared error and uses fewer parameters compared to existing models. The implementation of our model is available at: https://github.com/dongbeank/CATS.

Motivation & Objective

• To investigate whether self-attention mechanisms are truly necessary for effective time series forecasting.
• To address the inefficiency and potential information loss in self-attention due to permutation invariance and quadratic complexity.
• To design a streamlined Transformer architecture that maintains high forecasting accuracy while reducing model size and memory usage.
• To improve long-term forecasting by enabling horizon-dependent parameter sharing through cross-attention.
• To provide interpretability by allowing individual attention maps per forecasting horizon.

Proposed method

• CATS replaces all self-attention layers in the Transformer with cross-attention, using future forecasting horizons as queries.
• Each forecasting horizon is treated as an independent query vector, enabling horizon-specific attention computation.
• The model uses a patch-based input representation with non-overlapping patches and channel-independent processing.
• Parameter sharing is enhanced by tying the query embeddings across different horizons, reducing model complexity.
• Multi-head cross-attention is applied between the query (future horizon) and the key/value (past time series patches).
• The architecture is trained end-to-end with standard regression loss, and attention maps are visualized to interpret periodic pattern learning.

Figure 1: Experimental results illustrating the mean squared error (MSE) and the number of parameters with varying input sequence lengths on ETTm1. Each bubble represents a different model, with the bubble size indicating the number of parameters in millions—larger bubbles denote models with more pa

Experimental results

Research questions

• RQ1 Is self-attention essential for accurate time series forecasting, or can it be replaced with a simpler mechanism?
• RQ2 Can cross-attention alone achieve superior long-term forecasting performance compared to self-attention-based Transformers?
• RQ3 Does eliminating self-attention reduce model complexity without sacrificing predictive accuracy?
• RQ4 Can horizon-specific queries improve interpretability and performance in long-horizon forecasting?
• RQ5 How effectively can cross-attention capture periodic patterns in time series data?

Key findings

• CATS achieves the lowest mean squared error (MSE) across all evaluated datasets, including ETTm1 and Electricity, outperforming existing Transformer-based models.
• On the ETTm1 dataset with a 720-step horizon, CATS achieves an MSE of 0.400, significantly lower than models with self-attention (e.g., 0.442 for two self-attention layers).
• CATS uses fewer parameters than all compared models, with the smallest bubble size in Figure 1, indicating superior parameter efficiency.
• The model successfully captures periodic patterns in synthetic data, as shown by attention score maps highlighting 24-step periodicity and shock detection.
• Visualizations confirm that high-attention patches exhibit similar temporal patterns, demonstrating the model's ability to detect sequential and periodic structures.
• CATS provides interpretable attention maps per forecasting horizon, enabling clear analysis of how each prediction is derived from past data.

Figure 3: Illustration on the proposed model architecture. Our model removes all self-attentions from the original Transformer structure and focuses on cross-attentions. To fully utilize the cross-attention, we conceptualize the future horizon as queries and use the input time series (i.e., past tim