[论文解读] Analysis and Comparison of Classification Metrics

Luciana Ferrer|arXiv (Cornell University)|Sep 12, 2022

Imbalanced Data Classification Techniques被引用 18

一句话总结

本文主张使用期望损失（EC）和 proper scoring rules（PSRs）作为评估分类系统的更优、更合乎原则的指标。它表明，EC 广义化了标准误差率和平衡误差率，而基于 PSR 的校准损失在可解释性、鲁棒性和多分类适用性方面优于广泛使用的期望校准误差（ECE），从而对硬分类决策和校准后的得分提供了更全面的评估。

ABSTRACT

A variety of different performance metrics are commonly used in the machine learning literature for the evaluation of classification systems. Some of the most common ones for measuring quality of hard decisions are standard and balanced accuracy, standard and balanced error rate, F-beta score, and Matthews correlation coefficient (MCC). In this document, we review the definition of these and other metrics and compare them with the expected cost (EC), a metric introduced in every statistical learning course but rarely used in the machine learning literature. We show that both the standard and balanced error rates are special cases of the EC. Further, we show its relation with F-beta score and MCC and argue that EC is superior to these traditional metrics for being based on first principles from statistics, and for being more general, interpretable, and adaptable to any application scenario. The metrics mentioned above measure the quality of hard decisions. Yet, most modern classification systems output continuous scores for the classes which we may want to evaluate directly. Metrics for measuring the quality of system scores include the area under the ROC curve, equal error rate, cross-entropy, Brier score, and Bayes EC or Bayes risk, among others. The last three metrics are special cases of a family of metrics given by the expected value of proper scoring rules (PSRs). We review the theory behind these metrics, showing that they are a principled way to measure the quality of the posterior probabilities produced by a system. Finally, we show how to use these metrics to compute a system's calibration loss and compare this metric with the widely-used expected calibration error (ECE), arguing that calibration loss based on PSRs is superior to the ECE for being more interpretable, more general, and directly applicable to the multi-class case, among other reasons.

研究动机与目标

• 评估并比较常见分类指标，特别是那些用于评估硬决策和系统得分的指标。
• 论证期望损失（EC）优于标准指标如 F-beta 和准确率，因其具有坚实的统计基础和灵活性。
• 证明基于 proper scoring rules（PSRs）的校准损失是期望校准误差（ECE）的更鲁棒、更可解释的替代方案。
• 通过提供理论依据和开源代码，推动在机器学习中采用 EC 和基于 PSR 的指标。
• 为在多样化应用场景下评估分类系统提供统一、合乎原则的框架。

提出的方法

• 本文从统计决策理论出发，正式将期望损失（EC）定义为标准误差率和平衡误差率的广义化形式。
• 通过 proper scoring rules（PSRs）来框架化对系统得分的评估，PSRs 具有严格正确性且对校准质量敏感。
• 作者提出校准损失为实际期望 PSR 与其在最优校准后可达到的最小值之间的差值。
• 将基于 PSR 的校准损失与 ECE 进行比较，突出 ECE 在可解释性、分箱敏感性和多分类处理方面的局限性。
• 理论推导表明，EC 和 PSRs 是更广泛统计原理的特例，因此更具普遍性和可解释性。
• 提供了一个开源仓库以计算所有指标，支持结果的可复现性和实际应用。

实验结果

研究问题

• RQ1 在可解释性和泛化能力方面，期望损失（EC）与标准指标如 F-beta 和准确率相比如何？
• RQ2 基于 proper scoring rules（PSRs）的校准损失在哪些方面优于期望校准误差（ECE）？
• RQ3 为何 PSRs 比仅关注区分能力的指标（如 AUC 和 EER）更适合评估后验概率质量？
• RQ4 EC 如何作为统一框架，用于评估硬决策和校准后的得分？
• RQ5 在多分类和实际部署场景中，基于 PSR 的指标相比 ECE 具有哪些实际优势？

主要发现

• 期望损失（EC）广义化了标准误差率和平衡误差率，使其成为评估硬决策的更灵活、更合乎原则的指标。
• 在类别不平衡或不确定性较高的场景下，F-beta 分数和标准准确率均不如 EC 在可解释性和统计基础方面表现优越。
• 通过 proper scoring rules（PSRs）计算的校准损失比依赖脆弱直方图分箱的期望校准误差（ECE）更具可解释性和鲁棒性。
• PSR 基于的指标对区分能力和校准性均敏感，而 AUC 和 EER 对校准问题不敏感。
• 校准后期望 PSR 的最小值为系统性能提供了下界，从而可精确测量校准损失。
• 开源仓库支持所有结果的完全可复现性，推动了 EC 和基于 PSR 的指标的实际应用。