[Paper Review] Estimation of Climbing Route Difficulty using Whole-History Rating

Dean Scarff|arXiv (Cornell University)|Jan 15, 2020

Winter Sports Injuries and Performance4 references4 citations

TL;DR

This paper proposes a data-driven approach to estimating rock climbing route difficulty using the Whole-History Rating (WHR) system, adapting the Bradley-Terry model to infer route ratings from 236,095 ascent records. The method achieved 85% accuracy in 10-fold cross-validation, demonstrating that empirical ratings correlate strongly with conventional grades while adjusting for inconsistencies in subjective grading.

ABSTRACT

Existing grading systems for rock climbing routes assign a difficulty grade to a route based on the opinions of a few people. An objective approach to estimating route difficulty on an interval scale was obtained by adapting the Whole-History Rating (WHR) system to rock climbing. WHR's model was fitted to a database of 236,095 ascents recorded by users on an established climbing website. 73% of the ascents used in the dataset were classified as successful. Predictions were on average 85% accurate with 10-fold cross-validation. The results suggest that an empirical rating system is accurate at assessing route difficulty and is viable for revising conventional route grades.

Motivation & Objective

• To develop an objective, statistically grounded method for estimating climbing route difficulty, moving beyond subjective grading systems.
• To evaluate the applicability of the Whole-History Rating (WHR) system—originally designed for two-player games—to rock climbing as a dynamic rating system.
• To assess whether WHR can produce accurate, stable, and interpretable route ratings using real-world ascent data from a large online climbing database.
• To investigate whether conventional grading (e.g., Ewbank grades) improves model performance or if the data alone suffice for reliable estimation.

Proposed method

• Adapts the WHR model to climbing by treating climbers and routes as players with time-varying ratings, where successful ascents are treated as 'wins'.
• Uses the Bradley-Terry model to compute the probability of a successful ascent based on the difference in climber and route ratings: $ P(A=1) = \frac{\exp(r_i)}{\exp(r_i) + \exp(r_j)} $.
• Implements normal priors for route and climber ratings, with route priors informed by conventional grades via $ \mu_i = b(g_i - g_0) $, where $ b $ is a learned scaling parameter.
• Applies a Wiener process to model temporal changes in climber ratings, allowing ratings to evolve over time based on performance history.
• Employs Newton’s method for iterative optimization of the posterior log-likelihood, using the Hessian for efficient computation.
• Calibrates hyperparameters (e.g., $ b $, $ \sigma_C^2 $, $ \sigma_R^2 $) through cross-validation to balance accuracy and numerical stability.

Experimental results

Research questions

• RQ1 Can the Whole-History Rating system be effectively adapted to estimate climbing route difficulty using real ascent data?
• RQ2 How well do WHR-derived ratings correlate with conventional Ewbank grades, and what adjustments do they make across grade boundaries?
• RQ3 To what extent does incorporating conventional grades into the prior distribution improve model accuracy?
• RQ4 Is the model robust and stable when trained on large-scale, real-world ascent data from a public climbing database?
• RQ5 Can the resulting ratings provide a reliable, interval-scale measure of route difficulty that is more objective than current subjective grading practices?

Key findings

• The WHR model achieved 85.1% accuracy in 10-fold cross-validation and 88.3% accuracy without cross-validation on the Australian dataset of 236,095 ascents.
• The model's precision and recall were 90.5% and 93.7%, respectively, indicating strong performance on the binary classification of ascent success.
• Residuals were tightly distributed around zero with no detectable bias, confirming the model’s technical soundness and lack of systematic error.
• The estimated route ratings showed a moderate but meaningful linear relationship with Ewbank grades ($ R^2 = 0.640 $), with significant overlap between adjacent grades due to model adjustments.
• Using Ewbank grades in the prior improved accuracy only marginally—models with $ b = 0.4 $ were less than 0.1% more accurate than those with $ b = 0 $, suggesting the data alone carry substantial informative value.
• The final rating distributions were non-normal, skewed, and had non-zero means, indicating that the model’s posterior estimates diverged meaningfully from the initial priors.