[Paper Review] Evaluating Soccer Match Prediction Models: A Deep Learning Approach and Feature Optimization for Gradient-Boosted Trees

Calvin Yeung, Rory Bunker+2 more|arXiv (Cornell University)|Sep 26, 2023

Sports Analytics and Performance4 citations

TL;DR

This study evaluates deep learning and gradient-boosted tree models for soccer match outcome prediction using the 2023 Soccer Prediction Challenge dataset. It proposes a hybrid deep learning model combining TimesNet, Transformer, and feedforward networks, while optimizing features via selection algorithms; results show pi-ratings remain optimal for CatBoost, but deep learning underperforms betting consensus by 6% on W/D/L probability prediction.

ABSTRACT

Machine learning models have become increasingly popular for predicting the results of soccer matches, however, the lack of publicly-available benchmark datasets has made model evaluation challenging. The 2023 Soccer Prediction Challenge required the prediction of match results first in terms of the exact goals scored by each team, and second, in terms of the probabilities for a win, draw, and loss. The original training set of matches and features, which was provided for the competition, was augmented with additional matches that were played between 4 April and 13 April 2023, representing the period after which the training set ended, but prior to the first matches that were to be predicted (upon which the performance was evaluated). A CatBoost model was employed using pi-ratings as the features, which were initially identified as the optimal choice for calculating the win/draw/loss probabilities. Notably, deep learning models have frequently been disregarded in this particular task. Therefore, in this study, we aimed to assess the performance of a deep learning model and determine the optimal feature set for a gradient-boosted tree model. The model was trained using the most recent five years of data, and three training and validation sets were used in a hyperparameter grid search. The results from the validation sets show that our model had strong performance and stability compared to previously published models from the 2017 Soccer Prediction Challenge for win/draw/loss prediction.

Motivation & Objective

• To assess the performance of a deep learning model in predicting soccer match outcomes, particularly win/draw/loss probabilities and exact scores.
• To identify the optimal feature set for gradient-boosted tree models using feature selection algorithms.
• To address the lack of publicly available benchmark datasets by leveraging the 2023 Soccer Prediction Challenge data and augmenting the training set with matches from April 4–13, 2023.
• To compare the predictive performance of deep learning models against established baselines, including betting odds consensus and statistical models.
• To explore model interpretability and the potential of incorporating expert-derived features such as betting odds and player data in future work.

Proposed method

• Trained a deep learning model combining TimesNet for temporal modeling, Transformer architecture for sequence attention, and a feedforward neural network for final prediction.
• Used a hybrid architecture with embedding layers for team IDs and multi-layer perceptrons with dropout for non-sequential features.
• Applied grid search with three train/validation splits to tune hyperparameters, including embedding dimensions, feedforward layer sizes, and dropout rates.
• Augmented the original training set with matches played between April 4 and April 13, 2023—after the training set end but before the evaluation period.
• Employed feature selection algorithms to identify the most predictive features for CatBoost, focusing on pi-ratings, goal statistics, and performance metrics.
• Evaluated models on both exact score prediction and win/draw/loss probability prediction tasks using the challenge’s official evaluation protocol.

Experimental results

Research questions

• RQ1 Can a deep learning model outperform gradient-boosted tree models in predicting soccer match outcomes, particularly win/draw/loss probabilities?
• RQ2 What is the optimal set of features for training gradient-boosted tree models in soccer match prediction, especially when using pi-ratings and derived statistics?
• RQ3 How does model performance vary between exact score prediction and probability-based outcome forecasting tasks?
• RQ4 To what extent do betting odds consensus models outperform machine learning models in W/D/L probability prediction?
• RQ5 Can feature augmentation using post-training-period matches improve model generalization and predictive stability?

Key findings

• The deep learning model underperformed the betting odds consensus model by 6% in win/draw/loss probability prediction, indicating limited gains from deep learning in this context.
• Pi-ratings remained the most effective feature set for CatBoost models, outperforming other feature combinations in W/D/L probability prediction.
• For exact score prediction, Berrar ratings and simple statistical baselines outperformed both deep learning and gradient-boosted tree models.
• The optimal feature set for CatBoost included team-specific metrics such as EG_HT, GS_avg_HT, point_per_match_HT, win_pct_AT, and pi_rating_HT/AT.
• The deep learning model’s hyperparameter search identified 10 layers for the MLP, 1-dimensional team embedding, and 0.2 dropout as optimal settings.
• Model stability and performance were enhanced through data augmentation with matches from April 4–13, 2023, which bridged the gap between training and evaluation periods.