[Paper Review] TCNFormer: Temporal Convolutional Network Former for Short-Term Wind Speed Forecasting

Abid Hasan Zim, Aquib Iqbal+3 more|arXiv (Cornell University)|Aug 27, 2024

Energy Load and Power Forecasting4 citations

TL;DR

This paper proposes TCNFormer, a hybrid deep learning model combining Temporal Convolutional Networks (TCN) and a dual-attention transformer encoder for 12-hour short-term wind speed forecasting. By integrating causal temporal multi-head self-attention (CT-MSA) and temporal external attention (TEA), TCNFormer captures long-range dependencies and spatio-temporal patterns more effectively than state-of-the-art models, achieving superior prediction accuracy on real-world wind speed data from Bangladesh.

ABSTRACT

Global environmental challenges and rising energy demands have led to extensive exploration of wind energy technologies. Accurate wind speed forecasting (WSF) is crucial for optimizing wind energy capture and ensuring system stability. However, predicting wind speed remains challenging due to its inherent randomness, fluctuation, and unpredictability. This study proposes the Temporal Convolutional Network Former (TCNFormer) for short-term (12-hour) wind speed forecasting. The TCNFormer integrates the Temporal Convolutional Network (TCN) and transformer encoder to capture the spatio-temporal features of wind speed. The transformer encoder consists of two distinct attention mechanisms: causal temporal multi-head self-attention (CT-MSA) and temporal external attention (TEA). CT-MSA ensures that the output of a step derives only from previous steps, i.e., causality. Locality is also introduced to improve efficiency. TEA explores potential relationships between different sample sequences in wind speed data. This study utilizes wind speed data from the NASA Prediction of Worldwide Energy Resources (NASA POWER) of Patenga Sea Beach, Chittagong, Bangladesh (latitude 22.2352° N, longitude 91.7914° E) over a year (six seasons). The findings indicate that the TCNFormer outperforms state-of-the-art models in prediction accuracy. The proposed TCNFormer presents a promising method for spatio-temporal WSF and may achieve desirable performance in real-world applications of wind power systems.

Motivation & Objective

• To address the challenge of accurate short-term wind speed forecasting due to wind's inherent volatility and nonlinearity.
• To overcome limitations of existing models—such as RNNs' vanishing gradients and transformers' reliance on positional encoding—by designing a more robust architecture.
• To enhance feature learning by integrating causality, locality, and inter-sequence relationships in attention mechanisms.
• To evaluate the model’s performance across diverse seasonal conditions using real-world data from Patenga Sea Beach, Bangladesh.

Proposed method

• The TCNFormer model integrates a Temporal Convolutional Network (TCN) with a transformer encoder to preserve sequence length and enable causal processing.
• The transformer encoder employs two attention mechanisms: causal temporal multi-head self-attention (CT-MSA), which ensures causality and improves efficiency via local receptive fields.
• CT-MSA enhances standard multi-head attention by enforcing autoregressive constraints and incorporating local context, improving temporal feature learning.
• Temporal external attention (TEA) uses two external memory units to model relationships across different sample sequences, enriching representation learning.
• The model is trained on one year of hourly wind speed data from NASA POWER for Patenga Sea Beach, covering six distinct seasons.
• Ablation studies validate the contributions of CT-MSA and TEA by replacing them with standard multi-head attention in controlled experiments.

Figure 1: Govt. considered energy mix for 2041.

Experimental results

Research questions

• RQ1 Can a hybrid architecture combining TCN and transformer components improve short-term wind speed forecasting accuracy compared to existing models?
• RQ2 How do causality and locality in CT-MSA affect the model’s ability to learn temporal dependencies in wind speed sequences?
• RQ3 To what extent does temporal external attention (TEA) enhance feature representation by capturing inter-sequence relationships in wind speed data?
• RQ4 Does the TCNFormer maintain high performance across diverse seasonal patterns and varying wind regimes?
• RQ5 How does the model compare to standard RNNs, CNNs, and vanilla transformers in terms of accuracy and computational efficiency?

Key findings

• The TCNFormer outperforms state-of-the-art models, including Autoformer, Pyraformer, and vanilla Transformers, in 12-hour wind speed forecasting accuracy.
• The integration of CT-MSA significantly improves performance over standard multi-head attention, particularly in maintaining causality and reducing information leakage.
• The removal of TEA leads to a notable drop in model performance, confirming its critical role in capturing inter-sequence relationships.
• The ablation study confirms that both CT-MSA and TEA are essential components, with CT-MSA enhancing local temporal modeling and TEA improving global feature representation.
• The model demonstrates robust performance across all six seasons in the dataset, indicating strong generalization to varying climatic and wind conditions.
• TCNFormer achieves superior computational efficiency compared to traditional RNNs and standard transformers, making it suitable for real-time wind power applications.

Figure 2: Model Overview: TCNFormer.