[Paper Review] iWinRNFL: A Simple, Interpretable & Well-Calibrated In-Game Win Probability Model for NFL

Konstantinos Pelechrinis|arXiv (Cornell University)|Apr 1, 2017

Sports Analytics and Performance7 references3 citations

TL;DR

This paper introduces iWinRNFL, a simple, interpretable, and well-calibrated logistic regression model for in-game win probability in the NFL, using 10 game-state features from play-by-play data across 7 seasons. It achieves 75% accuracy in predicting game winners, significantly outperforming baseline pre-game models, and maintains well-calibrated probabilities across all confidence levels, with no meaningful improvement from more complex models like neural networks or Bayesian classifiers.

ABSTRACT

During the last few sports seasons a lot of discussion has been generated for the several, high-profile, "comebacks" that were observed in almost all sports. The Cavaliers won the championship after being down 3-1 in the 2016 NBA finals' series against the Golden State Warriors, which was exactly the case for Chicago Cubs and the World Series. The Patriots won the Super Bowl in 2016 even though they were trailing by 25 points late in the third quarter, while FC Barcelona in the top-16 round of the 2016-17 Champions League scored 3 goals during the last 7 minutes of the game (including stoppage time) against PSG to advance in the tournament. This has brought the robustness and accuracy of the various probabilistic prediction models under high scrutiny. Many of these models are proprietary, which makes it hard to evaluate. In this paper, we build a simple and open, yet robust and well-calibrated, in-game probability model for predicting the winner in an NFL (iWinRNFL) game. In particular, we build a logistic regression model that utilizes a set of 10 variables to predict the running win probability for the home team. We train our model using detailed play-by-play data from the last 7 NFL seasons obtained through the league's API. Our results indicate that in 75% of the cases iWinRNFL provides an accurate winner projection, as compared to a 63% accuracy of a baseline pre-game win probability model. Most importantly the probabilities that iWinRNFL provides are well-calibrated. Finally, we have also evaluated more complex, non-linear, models using the same set of features, without any significant improvement in performance.

Motivation & Objective

• To develop a transparent, open, and interpretable in-game win probability model for NFL that avoids overconfidence in trailing teams' chances.
• To evaluate whether a simple generalized linear model can achieve performance comparable to complex, non-linear models in predicting NFL game outcomes.
• To ensure the model's predicted probabilities are well-calibrated across the full range of win probabilities, especially in extreme cases like late-game comebacks.
• To provide a benchmark for evaluating and improving future win probability models through open, community-reviewable methodology.

Proposed method

• The model uses a logistic regression framework with 10 key game-state variables: time remaining, score differential, possession, down, distance to go, field position, team strength, home advantage, and game state (first/second half, end of half/game).
• Model training is based on detailed play-by-play data from 7 NFL seasons (2009–2015) obtained via the NFL’s official API.
• The model is evaluated using standard metrics: accuracy, Brier score, and calibration reliability across probability bins.
• Non-linear models (Bayesian classifier and feedforward neural network) are trained using the same 10 features to compare performance against the logistic regression baseline.
• Model calibration is assessed by comparing predicted probabilities to actual win frequencies in binned probability intervals.
• Anecdotal evaluation focuses on Super Bowl 51, where the Patriots overcame a 25-point deficit, to test model behavior under extreme conditions.

Experimental results

Research questions

• RQ1 Can a simple, interpretable logistic regression model achieve high accuracy and well-calibrated probabilities in NFL in-game win probability prediction?
• RQ2 How does the performance of iWinRNFL compare to baseline pre-game win probability models in terms of accuracy and calibration?
• RQ3 Do more complex, non-linear models (e.g., neural networks, Bayesian classifiers) significantly outperform the simple logistic model when using the same input features?
• RQ4 How well does the model handle extreme scenarios, such as late-game comebacks like the 2017 Super Bowl 51 Patriots comeback?
• RQ5 Is the model’s calibration robust across the entire probability spectrum, especially in low-probability events like 1-in-1000 chances?

Key findings

• iWinRNFL achieves 75% accuracy in predicting the correct game winner, significantly outperforming the 63% accuracy of a baseline pre-game win probability model.
• The model’s predicted probabilities are well-calibrated across all probability bins, meaning that teams assigned a 20% win probability win approximately 20% of the time.
• Even in extreme cases like the 2017 Super Bowl 51 comeback, iWinRNFL assigned the Patriots a conservative 2.1% win probability at their lowest point, avoiding overconfidence seen in some proprietary models.
• Despite evidence of non-linear dynamics near the end of quarters, more complex models like the feedforward neural network (FNN) only provided marginal improvements in Brier score and accuracy, especially in later game stages.
• The model’s performance improves as the game progresses, with accuracy approaching state-of-the-art levels by the end of the game, consistent with the increasing importance of in-game dynamics over team ratings.
• The study concludes that simple, interpretable models like iWinRNFL are sufficient for robust in-game win probability prediction, and do not require the complexity of non-linear models for meaningful performance gains.