[Paper Review] Predict+Optimize Problem in Renewable Energy Scheduling

Christoph Bergmeir, Frits de Nijs+26 more|arXiv (Cornell University)|Dec 21, 2022

Energy Load and Power Forecasting4 citations

TL;DR

This paper evaluates top solutions to the IEEE-CIS Technical Challenge on Predict+Optimize for renewable energy scheduling, using data from the Monash Microgrid. It proposes a hybrid approach combining gradient-boosted trees and random forests for forecasting, followed by a fix-and-optimize MIP strategy with sample average approximation, achieving significant cost reductions in energy scheduling under uncertainty.

ABSTRACT

Predict+Optimize frameworks integrate forecasting and optimization to address real-world challenges such as renewable energy scheduling, where variability and uncertainty are critical factors. This paper benchmarks solutions from the IEEE-CIS Technical Challenge on Predict+Optimize for Renewable Energy Scheduling, focusing on forecasting renewable production and demand and optimizing energy cost. The competition attracted 49 participants in total. The top-ranked method employed stochastic optimization using LightGBM ensembles, and achieved at least a 2% reduction in energy costs compared to deterministic approaches, demonstrating that the most accurate point forecast does not necessarily guarantee the best performance in downstream optimization. The published data and problem setting establish a benchmark for further research into integrated forecasting-optimization methods for energy systems, highlighting the importance of considering forecast uncertainty in optimization models to achieve cost-effective and reliable energy management. The novelty of this work lies in its comprehensive evaluation of Predict+Optimize methodologies applied to a real-world renewable energy scheduling problem, providing insights into the scalability, generalizability, and effectiveness of the proposed solutions. Potential applications extend beyond energy systems to any domain requiring integrated forecasting and optimization, such as supply chain management, transportation planning, and financial portfolio optimization.

Motivation & Objective

• Address the challenge of integrating forecasting and optimization in renewable energy scheduling under uncertainty.
• Provide a benchmark for the Predict+Optimize paradigm in energy systems using real-world microgrid data.
• Evaluate and compare state-of-the-art methods that jointly optimize scheduling and forecasting for cost minimization.
• Demonstrate the superiority of end-to-end or integrated forecasting-optimization approaches over traditional two-stage pipelines.
• Highlight practical challenges such as scenario selection, model robustness, and computational efficiency in real deployments.

Proposed method

• Uses gradient-boosted trees and random forests to forecast solar PV generation and building electricity demand using weather and calendar features.
• Applies quantile regression with Random Forests to estimate predictive marginal densities for scenario generation.
• Employs a fix-and-optimize heuristic in mixed-integer programming (MIP) to iteratively improve solutions by fixing subsets of variables.
• Implements sample average approximation (SAA) to optimize over multiple forecast scenarios, reducing expected cost under uncertainty.
• Solves the optimization problem using Gurobi with time limits and optimality gap constraints to balance solution quality and runtime.
• Uses iterative local search with activity sampling (10 recurring, 5 one-off per iteration) and patience-based early stopping to improve convergence.

Experimental results

Research questions

• RQ1 How do integrated Predict+Optimize methods compare to traditional two-stage forecasting-optimization pipelines in renewable energy scheduling?
• RQ2 What is the impact of scenario generation quality—particularly based on a single building’s forecast—on final scheduling cost?
• RQ3 To what extent does the fix-and-optimize heuristic improve solution quality and speed compared to full MIP solving?
• RQ4 How do different forecasting models (e.g., XGBoost, Random Forest) affect downstream optimization performance in energy scheduling?
• RQ5 What are the key bottlenecks in deploying robust, scalable Predict+Optimize systems in real-world microgrid operations?

Key findings

• The winning method achieved significant cost reductions by jointly optimizing over multiple forecast scenarios using sample average approximation.
• Gradient-boosted trees and random forests produced the most accurate forecasts for solar PV and building demand across the dataset.
• The fix-and-optimize approach reduced the final cost substantially, especially on larger problem instances, with improvements visible in objective value progression.
• The optimization phase was bounded by the solution quality of the full MIP, indicating that better MIP formulations could further improve results.
• Scenarios based solely on Building 3’s predictive density limited generalizability, and incorporating cross-source correlations could improve performance.
• Modeling errors and mismatched optimization windows during Phase 2 hindered evaluation and highlighted the need for robust, modular code in real-world deployments.