[Paper Review] Eliciting forecasts from self-interested experts: scoring rules for decision makers

Craig Boutilier|arXiv (Cornell University)|Jun 4, 2012

Sports Analytics and Performance18 references32 citations

TL;DR

This paper develops compensation rules that align expert incentives with truthful forecasting when experts have personal stakes in the decision maker's choice. By combining compensation with expert utility, it constructs a net scoring rule that ensures truthfulness, even under uncertainty about expert preferences, and provides bounds on misreporting and decision loss.

ABSTRACT

Scoring rules for eliciting expert predictions of random variables are usually developed assuming that experts derive utility only from the quality of their predictions. We study more realistic settings in which (a) the principal is a decision maker who takes a decision based on the expert's prediction; and (b) the expert has an inherent interest in the decision. Not surprisingly, in such situations, the expert usually has an incentive to misreport her forecast to influence the choice of the decision maker. We develop a general model for this setting and introduce the concept of a compensation rule. When combined with the expert's inherent utility for decisions, a compensation rule induces a net scoring rule that behaves like a traditional scoring rule. Assuming full knowledge of expert utility, we provide a complete characterization of all (strictly) proper compensation rules. We then analyze the case when the expert's utility function is not fully known to the decision maker. We show bounds on: (a) expert incentive to misreport; (b) the degree to which an expert will misreport; and (c) decision maker loss in utility due to such uncertainty. These bounds depend in natural ways on the degree of uncertainty, the local degree of convexity of net scoring function, and properties of the decision maker's utility function. Finally, we briefly discuss the use of compensation rules in prediction markets.

Motivation & Objective

• Address the gap in scoring rules that assume experts care only about forecast accuracy, not decision outcomes.
• Model settings where experts have intrinsic utility from the decision, creating incentives to misreport forecasts.
• Develop compensation rules that, when combined with expert utility, yield a net scoring rule that incentivizes truthful reporting.
• Characterize all strictly proper compensation rules under full knowledge of expert utility functions.
• Analyze the impact of uncertainty about expert utility on forecasting incentives and decision maker utility loss.

Proposed method

• Introduce a general model where the decision maker uses expert forecasts to make a decision, and experts derive utility from both forecast accuracy and decision outcomes.
• Define a compensation rule that adjusts payments based on forecast and decision, counterbalancing expert's inherent interest in the decision.
• Construct a net scoring rule as the sum of the expert's forecast scoring rule and the compensation rule, ensuring truthfulness when properly calibrated.
• Use convex analysis to characterize strictly proper compensation rules under full knowledge of expert utility.
• Derive bounds on expert misreporting and decision maker utility loss when expert utility is partially known, based on local convexity of the net scoring function.
• Apply the framework to prediction markets by discussing how compensation rules can be implemented in market mechanisms.

Experimental results

Research questions

• RQ1 How can compensation rules be designed to ensure truthful forecasting when experts have personal stakes in the decision outcome?
• RQ2 What conditions on the compensation rule guarantee that the resulting net scoring rule is strictly proper under full knowledge of expert utility?
• RQ3 How does uncertainty about the expert's utility function affect the expert's incentive to misreport forecasts?
• RQ4 What are the bounds on the degree of misreporting and decision maker utility loss due to uncertainty in expert preferences?
• RQ5 Can the proposed compensation rules be adapted for use in prediction markets to elicit accurate forecasts?

Key findings

• All strictly proper compensation rules are completely characterized when the decision maker has full knowledge of the expert's utility function.
• Under uncertainty about expert utility, the expert's incentive to misreport is bounded by the local degree of convexity of the net scoring function.
• The decision maker's expected utility loss due to expert misreporting is bounded and depends on the degree of uncertainty and the curvature of the net scoring function.
• The bounds on misreporting and utility loss are tighter when the net scoring function is more convex locally.
• The framework can be extended to prediction markets, where compensation rules can be used to align incentives in market-based forecasting mechanisms.
• The net scoring rule, formed by combining compensation and forecast scoring, behaves like a traditional proper scoring rule when expert utility is properly accounted for.