[论文解读] Evaluating forecasts for high-impact events using transformed kernel scores

Sam Allen, David Ginsbourger+1 more|arXiv (Cornell University)|Feb 25, 2022

Meteorological Phenomena and Simulations被引用 5

一句话总结

本文提出基于核得分的加权多变量评分规则，用于评估高影响事件（如复合天气现象）的预报。研究表明，阈值加权连续 ranked probability score（twCRPS）是一种核得分，从而实现对集合预报的高效评估，并可推广至新的加权评分规则，如阈值加权能量得分与变差分值得分，这些规则能更准确识别出预测高影响结果的预报，即使单个事件本身并不极端。

ABSTRACT

It is informative to evaluate a forecaster's ability to predict outcomes that have a large impact on the forecast user. Although weighted scoring rules have become a well-established tool to achieve this, such scores have been studied almost exclusively in the univariate case, with interest typically placed on extreme events. However, a large impact may also result from events not considered to be extreme from a statistical perspective: the interaction of several moderate events could also generate a high impact. Compound weather events provide a good example of this. To assess forecasts made for high-impact events, this work extends existing results on weighted scoring rules by introducing weighted multivariate scores. To do so, we utilise kernel scores. We demonstrate that the threshold-weighted continuous ranked probability score (twCRPS), arguably the most well-known weighted scoring rule, is a kernel score. This result leads to a convenient representation of the twCRPS when the forecast is an ensemble, and also permits a generalisation that can be employed with alternative kernels, allowing us to introduce, for example, a threshold-weighted energy score and threshold-weighted variogram score. To illustrate the additional information that these weighted multivariate scoring rules provide, results are presented for a case study in which the weighted scores are used to evaluate daily precipitation accumulation forecasts, with particular interest on events that could lead to flooding.

研究动机与目标

• 解决未加权评分规则在评估由复合非极端条件引发的高影响事件预报时的局限性。
• 将加权评分规则从单变量情形扩展至多变量结果，利用核得分框架。
• 构建一个灵活的框架，以强调结果空间中的特定区域，特别是与高影响事件相关的区域。
• 聚焦于洪水风险，使用加权与未加权评分规则，评估累积降水预报的性能。
• 证明加权核得分可得出与未加权评分不同的结论，凸显在高影响事件中表现最佳的预报策略。

提出的方法

• 利用基于条件负定（c.n.d.）核的核得分——一种合适的评分规则——构建加权多变量评分规则。
• 证明阈值加权连续 ranked probability score（twCRPS）为核得分，从而实现对集合预报的高效计算。
• 将twCRPS推广至其他核函数，提出阈值加权能量得分与阈值加权变差分值得分。
• 引入垂直重缩放核得分作为替代加权机制，同时保持核得分框架的完整性。
• 采用逆多二次核作为新的严格合适的核函数，适用于单变量与多变量情形，具备强大的区分能力。
• 将所提出的加权评分规则应用于MeteoSwiss的每日累积降水预报，通过加权与未加权评估比较多种多变量后处理方法。

Figure 1 : Common weight functions (top row) with corresponding chaining functions (middle row) and the resulting kernels to be employed in the twCRPS (bottom row). $\Phi(x;\mu,\sigma)$ and $\phi(x;\mu,\sigma)$ represent the distribution and density functions, respectively, of a normal distribution

实验结果

研究问题

• RQ1 能否在核得分框架内构建加权多变量评分规则，以更有效地评估高影响事件的预报？
• RQ2 阈值加权CRPS是否为核得分？该性质是否能实现对集合预报的更高效评估？
• RQ3 阈值加权能量得分与变差分值得分相较于其未加权版本，在检测高影响事件预报性能方面表现如何？
• RQ4 在实际应用中，加权核得分是否会导致与未加权评分不同的预报质量判断？
• RQ5 核函数的选择是否会影响评分规则对结果空间特定区域（尤其是与复合高影响事件相关的区域）的敏感性？

主要发现

• 正式证明阈值加权CRPS为核得分，从而可通过闭式表达式用于集合预报。
• 该框架可将twCRPS推广至其他核函数，从而导出新型加权评分规则，如阈值加权能量得分与变差分值得分。
• 垂直重缩放核得分提供了一种替代加权机制，在特定条件下等价于阈值加权，但可扩展至更广泛的权重函数范围。
• 在每日累积降水预报的案例研究中，加权评分识别出与未加权评分不同的最优后处理方法，尤其在洪水风险事件中表现显著差异。
• 证明逆多二次核对R^d上所有概率测度均为严格合适，且具备优异的区分能力，适用于多变量预报评估。
• 加权核得分可检测预报在结果空间不同区域的行为差异，尤其在高影响区域表现突出，而未加权评分则无法捕捉此类差异。

(a) $w(z)=\mathbbm{1}\{z_{1}+z_{2}\geq t\}$