[Paper Review] Gradient boosting machines and careful pre-processing work best: ASHRAE Great Energy Predictor III lessons learned

Clayton Miller, Hao Liu+1 more|arXiv (Cornell University)|Feb 7, 2022

Energy Load and Power Forecasting4 citations

TL;DR

This paper analyzes the top-performing solutions in the ASHRAE Great Energy Predictor III competition, revealing that gradient boosting machines—particularly LightGBM—combined with meticulous feature engineering and pre-processing yielded the highest accuracy in long-term building energy prediction. The study identifies data preprocessing and feature extraction as the most critical phases, with Python and Jupyter notebooks as dominant tools among top teams.

ABSTRACT

The ASHRAE Great Energy Predictor III (GEPIII) competition was held in late 2019 as one of the largest machine learning competitions ever held focused on building performance. It was hosted on the Kaggle platform and resulted in 39,402 prediction submissions, with the top five teams splitting $25,000 in prize money. This paper outlines lessons learned from participants, mainly from teams who scored in the top 5% of the competition. Various insights were gained from their experience through an online survey, analysis of publicly shared submissions and notebooks, and the documentation of the winning teams. The top-performing solutions mostly used ensembles of Gradient Boosting Machine (GBM) tree-based models, with the LightGBM package being the most popular. The survey participants indicated that the preprocessing and feature extraction phases were the most important aspects of creating the best modeling approach. All the survey respondents used Python as their primary modeling tool, and it was common to use Jupyter-style Notebooks as development environments. These conclusions are essential to help steer the research and practical implementation of building energy meter prediction in the future.

Motivation & Objective

• To identify the most effective machine learning techniques and workflows used by top-performing teams in the ASHRAA Great Energy Predictor III competition.
• To understand the role of data preprocessing and feature engineering in achieving high accuracy for long-term building energy prediction.
• To assess the tools, programming languages, and development environments preferred by high-scoring teams in the competition.
• To explore barriers to entry for built environment professionals in data science and suggest pathways for broader adoption of ML in building energy research.
• To provide actionable insights for future machine learning competitions and real-world applications in building energy performance prediction.

Proposed method

• A post-competition survey was administered to 27 individuals from 50 contestants in the top 5% of the ASHRAE GEPIII leaderboard, focusing on demographics, modeling workflows, and tool preferences.
• Publicly shared notebooks, competition submissions, and winning team documentation were analyzed to validate survey findings and extract technical details of top solutions.
• The analysis focused on the five-stage machine learning workflow: data preprocessing, feature engineering, model selection, hyperparameter tuning, and ensemble stacking.
• LightGBM was identified as the most widely used gradient boosting framework, with ensemble methods combining multiple GBM models showing superior performance.
• The study evaluated the impact of data diversity, including inclusion of additional behavioral and contextual data sources, on model generalization and accuracy.
• Qualitative feedback from participants was used to assess competition design preferences and identify challenges in adopting ML tools for non-expert practitioners in the built environment.

Experimental results

Research questions

• RQ1 Which machine learning models and algorithms were most effective in achieving high accuracy in long-term building energy prediction across diverse building types and climates?
• RQ2 To what extent did data preprocessing and feature engineering contribute to model performance compared to model architecture or hyperparameter tuning?
• RQ3 What tools and development environments were most commonly used by top-performing teams, and how did they influence solution quality?
• RQ4 What are the key barriers preventing built environment professionals from engaging in data science and machine learning applications in building energy research?
• RQ5 How can open datasets and shared code notebooks help bridge the gap between data science and building energy professionals?

Key findings

• The top-performing solutions were predominantly ensembles of Gradient Boosting Machine (GBM) models, with LightGBM being the most popular and effective implementation.
• Survey respondents consistently ranked data preprocessing and feature engineering as the most critical phases in the modeling pipeline, surpassing model selection and hyperparameter tuning in importance.
• All survey respondents used Python as their primary programming language, and Jupyter-style notebooks were the dominant development environment for model prototyping and sharing.
• Despite the availability of open-source code and datasets, built environment professionals remained underrepresented among top competitors, indicating a skills gap in data science adoption.
• The winning solutions were largely open-sourced, providing valuable learning resources for beginners and practitioners seeking to apply modern data science techniques to building energy data.
• The study found that model accuracy was highly sensitive to data quality and feature representation, with improvements in data breadth and context significantly boosting predictive performance.