[Paper Review] Application of BERT in Wind Power Forecasting-Teletraan's Solution in Baidu KDD Cup 2022

Longxing Tan, Hongying Yue|arXiv (Cornell University)|Jul 18, 2023

Energy Load and Power ForecastingEngineering3 citations

TL;DR

This paper presents a BERT-based model for long-term wind power forecasting in the Baidu KDD Cup 2022, leveraging temporal dependencies through a single BERT block and enhancing predictions with post-processing to enforce daily periodicity. The solution achieved 3rd place among 2,490 teams by combining sequence modeling with a simple yet effective daily fluctuation adjustment.

ABSTRACT

Nowadays, wind energy has drawn increasing attention as its important role in carbon neutrality and sustainable development. When wind power is integrated into the power grid, precise forecasting is necessary for the sustainability and security of the system. However, the unpredictable nature and long sequence prediction make it especially challenging. In this technical report, we introduce the BERT model applied for Baidu KDD Cup 2022, and the daily fluctuation is added by post-processing to make the predicted results in line with daily periodicity. Our solution achieves 3rd place of 2490 teams. The code is released athttps://github.com/LongxingTan/KDD2022-Baidu

Motivation & Objective

• To address the challenge of long-sequence wind power forecasting with high temporal variability and unpredictability.
• To leverage the BERT model’s ability to capture long-range temporal dependencies in wind power time series.
• To improve forecast accuracy by incorporating daily periodicity through post-processing, as the model alone failed to learn this pattern.
• To achieve high performance under strict inference time constraints (10 hours for test inference).
• To evaluate the effectiveness of BERT versus other deep learning and traditional models in a real-world wind power forecasting setting.

Proposed method

• Applied a single BERT block to model 14-day historical wind power sequences at 10-minute intervals for each turbine.
• Used wind speed and wind direction as input features, with min-max normalization applied to all features except the target.
• Employed a masked autoencoder pretraining strategy on the wind power sequence to learn temporal patterns.
• Implemented post-processing by adding a normalized, scaled daily fluctuation pattern to predictions, aligned with the start time of each forecast.
• Used a multiplier of 36 (and 1.1 for larger values) to amplify the daily fluctuation effect, calibrated via local validation.
• Trained the model using RMSE loss and Adam optimizer with a batch size of 1024 and 3 training epochs.

Experimental results

Research questions

• RQ1 Can a single BERT model effectively capture long-range temporal dependencies in wind power time series for 48-hour forecasting?
• RQ2 Does post-processing with a learned daily fluctuation pattern improve forecast accuracy when the model fails to capture periodicity?
• RQ3 How does BERT-based modeling compare to other deep learning models (e.g., LSTM, TCN, Transformer) in terms of accuracy and inference efficiency?
• RQ4 To what extent do spatial features or spatial-temporal models improve performance in wind power forecasting with homogeneous turbine data?
• RQ5 Can a minimal feature set (wind speed and direction) achieve competitive results without including temperature or lag/rolling features?

Key findings

• The BERT model achieved a local validation score of 58.1 and a leaderboard score of 44.6 in phase II, securing 3rd place among 2,490 teams.
• Post-processing with a scaled, time-aligned daily fluctuation pattern significantly improved forecast periodicity and outperformed end-to-end model learning of this pattern.
• Among deep learning models, BERT outperformed LSTM, TCN, and Transformer in both local and leaderboard scores, with a lower RMSE (365) and MAE (299).
• Spatial-temporal models like GCN-LSTM achieved lower local scores (54.5) but were less effective on the leaderboard, suggesting limited benefit from spatial modeling in this homogeneous dataset.
• Statistical models like KNN and LightGBM performed worse than deep learning models, indicating that complex temporal patterns require deep architectures.
• The final solution was robust to concept drift, with performance maintained by retraining based on the best model from phase II during the final phase.