[Paper Review] Evaluating Forecasts with scoringutils in R

Nikos I Bosse, Hugo Gruson+4 more|arXiv (Cornell University)|May 14, 2022

Forecasting Techniques and Applications26 citations

TL;DR

The paper demonstrates how to evaluate forecasts in R using the scoringutils package, including summarising scores, coverage estimation, and converting sample-based forecasts to quantile-based formats. It also discusses cautions when aggregating scores and interpreting calibration metrics.

ABSTRACT

Evaluating forecasts is essential to understand and improve forecasting and make forecasts useful to decision makers. A variety of R packages provide a broad variety of scoring rules, visualisations and diagnostic tools. One particular challenge, which scoringutils aims to address, is handling the complexity of evaluating and comparing forecasts from several forecasters across multiple dimensions such as time, space, and different types of targets. scoringutils extends the existing landscape by offering a convenient and flexible data.table-based framework for evaluating and comparing probabilistic forecasts (forecasts represented by a full predictive distribution). Notably, scoringutils is the first package to offer extensive support for probabilistic forecasts in the form of predictive quantiles, a format that is currently used by several infectious disease Forecast Hubs. The package is easily extendable, meaning that users can supply their own scoring rules or extend existing classes to handle new types of forecasts. scoringutils provides broad functionality to check the data and diagnose issues, to visualise forecasts and missing data, to transform data before scoring, to handle missing forecasts, to aggregate scores, and to visualise the results of the evaluation. The paper presents the package and its core functionality and illustrates common workflows using example data of forecasts for COVID-19 cases and deaths submitted to the European COVID-19 Forecast Hub.

Motivation & Objective

• Demonstrate how to use scoringutils for forecast evaluation in R.
• Show how to summarise forecast scores and visualize them across models and target types.
• Provide guidance on calibration metrics and coverage using empirical forecast intervals.
• Illustrate data preparation steps for different forecast formats (quantile- vs sample-based).

Proposed method

• Use summarise_scores to aggregate and display forecast scores by model and target type.
• Apply add_coverage to quantify empirical coverage of central prediction intervals (e.g., 50% or 90%).
• Convert sample-based forecasts to quantile-based form using sample_to_quantile, enabling score() and add_coverage().
• Demonstrate plotting with plotscoretable and by/grouping options (e.g., by targettype).
• Highlight cautions when aggregating scores across heterogeneous forecast types or horizons to avoid dominance by large-magnitude targets.
• Note the use of a proxy for calibration through empirical coverage and the role of alternative summary functions (e.g., signif) with care.

Experimental results

Research questions

• RQ1 How can scoringutils be used to compute and visualise forecast evaluation metrics across models and target types?
• RQ2 What are appropriate aggregation and visualization strategies for different forecast formats (quantile-based vs sample-based)?
• RQ3 How do empirical coverage metrics reflect calibration for forecast intervals, and how can they be added to score tables?
• RQ4 What cautions are needed when aggregating scores across heterogeneous forecast targets or horizons?

Key findings

• scoringutils enables summarising and visualising forecast scores with functions like summarise_scores and plotscoretable.
• add_coverage provides empirical coverage estimates for specified central prediction intervals (e.g., 50% or 90%).
• sample_to_quantile allows converting sample-based forecasts to a quantile-based format suitable for scoring and coverage analysis.
• Aggregating scores across heterogeneous targets or horizons can lead to dominated aggregates and misinterpretation; relative or stratified analyses are recommended.
• The example demonstrates a multi-quantity (e.g., cases, deaths) and multi-model scenario with results including per-model and per-target-type summaries.
• Users are advised to exercise caution with non-mean summary functions, as propriety of scores under certain aggregations may be violated.