[Paper Review] Calibration Scoring Rules for Practical Prediction Training

Spencer Greenberg|arXiv (Cornell University)|Aug 22, 2018

Statistics Education and Methodologies2 references3 citations

TL;DR

This paper introduces 'Practical' scoring rules—specifically the 'Distance' and 'Order of Magnitude' rules—for improving calibration training in real-world prediction systems. Designed to enhance usability and psychological intuitiveness, these rules modify standard proper scoring rules to avoid zero or unbounded negative scores, ensuring clearer feedback and faster learning, even at the cost of sacrificing strict properness.

ABSTRACT

In situations where forecasters are scored on the quality of their probabilistic predictions, it is standard to use `proper' scoring rules to perform such scoring. These rules are desirable because they give forecasters no incentive to lie about their probabilistic beliefs. However, in the real world context of creating a training program designed to help people improve calibration through prediction practice, there are a variety of desirable traits for scoring rules that go beyond properness. These potentially may have a substantial impact on the user experience, usability of the program, or efficiency of learning. The space of proper scoring rules is too broad, in the sense that most proper scoring rules lack these other desirable properties. On the other hand, the space of proper scoring rules is potentially also too narrow, in the sense that we may sometimes choose to give up properness when it conflicts with other properties that are even more desirable from the point of view of usability and effective training. We introduce a class of scoring rules that we call `Practical' scoring rules, designed to be intuitive to users in the context of `right' vs. `wrong' probabilistic predictions. We also introduce two specific scoring rules for prediction intervals, the `Distance' and `Order of magnitude' rules. These rules are designed to satisfy a variety of properties that, based on user testing, we believe are desirable for applied calibration training.

Motivation & Objective

• To address the limitations of standard proper scoring rules in real-world calibration training by incorporating psychological and usability factors beyond mathematical properness.
• To design scoring rules that make feedback intuitive and actionable for users, enabling faster learning and better user engagement in prediction training apps.
• To resolve common user confusions caused by zero or unbounded negative scores in traditional scoring systems for probabilistic predictions.
• To develop prediction interval scoring rules that reward narrower intervals when the true value is near the center, while ensuring minimal positive scores even when the true value lies exactly on the interval boundary.
• To balance mathematical rigor with practical usability by introducing parameterized rules that maintain consistency across different prediction types

Proposed method

• Proposes 'Practical' scoring rules as a class of modified proper scoring rules that prioritize user intuition and psychological clarity over strict mathematical purity.
• Introduces the 'Distance' scoring rule for prediction intervals, which assigns higher scores when the true value is closer to the interval center and narrower intervals receive higher rewards.
• Introduces the 'Order of Magnitude' scoring rule, which uses logarithmic distance to assess prediction error and rewards predictions that are closer in order of magnitude.
• Applies a bounded score function with a minimum score $ s_{min} = -57.26893683880667 $, ensuring no unbounded losses and preventing counterintuitive zero-point outcomes.
• Incorporates a prediction interval expansion factor $ au = 0.4 $, so values exactly on the boundary receive positive points, improving user perception of fairness.
• Calibrates parameters such as $ s_{max} = 10 $, $ p_{max} = 0.99 $, and scale parameters $ c = 100 $ (Distance) and $ c = \ln(100) \approx 4.605 $ (Order of Magnitude) to align with user expectations and training efficiency.

Experimental results

Research questions

• RQ1 How can scoring rules for probabilistic predictions be redesigned to improve usability and psychological clarity in real-world calibration training applications?
• RQ2 What modifications to standard proper scoring rules are necessary to avoid counterintuitive outcomes such as zero or unbounded negative scores?
• RQ3 To what extent can sacrificing strict properness improve the learning experience and feedback clarity in a prediction training system?
• RQ4 How do bounded and expanded scoring rules affect user perception and learning speed when predicting numerical values via prediction intervals?
• RQ5 What parameter choices optimize the balance between mathematical consistency, user intuition, and training efficiency in a practical calibration app?

Key findings

• The 'Distance' and 'Order of Magnitude' scoring rules successfully resolve the issue of zero-point rewards when the true value lies exactly on the prediction interval boundary, which users previously found confusing.
• By introducing a minimum score $ s_{min} = -57.26893683880667 $, the system prevents unbounded negative scores, eliminating psychological discomfort from extreme point losses.
• The expansion factor $ au = 0.4 $ ensures that predictions with the true value on the boundary receive positive points, improving user satisfaction and reinforcing correct behavior.
• Setting $ p_{max} = 0.99 $ and $ s_{max} = 10 $ improves usability by preventing users from selecting extreme probabilities that are hard to calibrate and lead to numerical outliers.
• The scale parameters $ c = 100 $ and $ c = \ln(100) $ define a 'moderate' error as a 100-fold deviation or two orders of magnitude, respectively, aligning with user expectations of what constitutes a significant mistake.
• The final scoring rules, while not strictly proper, outperform standard linear and logarithmic rules in real-world training contexts due to superior feedback clarity and faster cognitive processing by users.