[Paper Review] Luck is Hard to Beat: The Difficulty of Sports Prediction

Raquel Aoki, Renato Assunção+1 more|arXiv (Cornell University)|Jun 8, 2017

Sports Analytics and Performance26 references8 citations

TL;DR

This paper introduces a skill coefficient $\phi$ to quantify the relative influence of luck versus skill in sports leagues, using data from 1,503 seasons across 198 leagues in basketball, soccer, volleyball, and handball. It proposes a probabilistic graphical model that decomposes outcomes into skill and luck components, revealing that underdog teams win with a 36% probability in the NBA—highlighting that luck is a dominant factor even in highly competitive sports.

ABSTRACT

Predicting the outcome of sports events is a hard task. We quantify this difficulty with a coefficient that measures the distance between the observed final results of sports leagues and idealized perfectly balanced competitions in terms of skill. This indicates the relative presence of luck and skill. We collected and analyzed all games from 198 sports leagues comprising 1503 seasons from 84 countries of 4 different sports: basketball, soccer, volleyball and handball. We measured the competitiveness by countries and sports. We also identify in each season which teams, if removed from its league, result in a completely random tournament. Surprisingly, not many of them are needed. As another contribution of this paper, we propose a probabilistic graphical model to learn about the teams' skills and to decompose the relative weights of luck and skill in each game. We break down the skill component into factors associated with the teams' characteristics. The model also allows to estimate as 0.36 the probability that an underdog team wins in the NBA league, with a home advantage adding 0.09 to this probability. As shown in the first part of the paper, luck is substantially present even in the most competitive championships, which partially explains why sophisticated and complex feature-based models hardly beat simple models in the task of forecasting sports' outcomes.

Motivation & Objective

• To develop a general framework for measuring the relative roles of skill and luck in sports league outcomes.
• To quantify the extent to which observed league results deviate from a purely random (luck-based) distribution.
• To identify which teams, if removed, would render a league's outcome statistically random, indicating their outsized influence.
• To build a probabilistic graphical model that estimates team skills and decomposes the contributions of luck and skill in match outcomes.
• To investigate how team-level features (e.g., salary, roster coherence) relate to skill and predictability in sports competitions.

Proposed method

• Introduce a skill coefficient $\phi$ that measures the distance between observed league final score distributions and an idealized, purely random (no-skill) distribution.
• Apply a significance test on $\phi$ to determine whether skill plays a statistically significant role in a given league.
• Use a probabilistic graphical model to estimate team-specific skill parameters ($\alpha_i$) based on match outcomes and contextual features.
• Model the probability of an underdog win ($\mathbb{P}(U)$) and condition it on home advantage ($\mathbb{P}(U|H)$) and away play ($\mathbb{P}(U|A)$), with additional conditioning on removed teams ($\mathbb{P}(U|H,R+)$).
• Incorporate features such as conference, average salary of top 5 players, team PER, volatility, roster coherence, and team size to predict skill and validate model performance via correlation with wins.

Experimental results

Research questions

• RQ1 To what extent does skill, as opposed to luck, determine outcomes in professional sports leagues across different sports?
• RQ2 Which teams, if removed from a league, would render the competition's outcome statistically indistinguishable from random?
• RQ3 What is the baseline probability of an underdog winning in a major league such as the NBA, and how does home advantage affect this?
• RQ4 How do team-level features (e.g., salary, roster coherence) correlate with estimated skill and actual win rates?
• RQ5 Can a probabilistic graphical model effectively decompose the relative contributions of luck and skill in match outcomes?

Key findings

• The skill coefficient $\phi$ reveals that basketball is the most competitive sport among the four studied, followed by volleyball, soccer, and handball.
• In the NBA, the probability that an underdog wins a game is 0.36, indicating that luck accounts for approximately 36% of match outcomes.
• Home advantage increases the underdog’s win probability by 0.18, from 0.27 when playing away to 0.45 when playing at home.
• Removing only 3 teams from the Primera División Spanish soccer league renders the competition statistically random, indicating their outsized influence.
• The probabilistic graphical model achieves a correlation of 0.7399 between estimated team skill and actual number of wins in the NBA regular season.
• When the most skilled teams are removed, the home advantage effect on underdog wins diminishes significantly, with $\mathbb{P}(U|H,R+)$ dropping to 0.13–0.17, suggesting that home advantage is less impactful in more balanced leagues.