[Paper Review] Analysis and Comparison of Classification Metrics

Luciana Ferrer|arXiv (Cornell University)|Sep 12, 2022

Imbalanced Data Classification TechniquesComputer Science18 citations

TL;DR

This paper advocates for the use of expected cost (EC) and proper scoring rules (PSRs) as superior, principled metrics for evaluating classification systems. It demonstrates that EC generalizes standard and balanced error rates, while PSR-based calibration loss outperforms the widely used expected calibration error (ECE) in interpretability, robustness, and multi-class applicability, offering a more comprehensive evaluation of both hard decisions and calibrated scores.

ABSTRACT

A variety of different performance metrics are commonly used in the machine learning literature for the evaluation of classification systems. Some of the most common ones for measuring quality of hard decisions are standard and balanced accuracy, standard and balanced error rate, F-beta score, and Matthews correlation coefficient (MCC). In this document, we review the definition of these and other metrics and compare them with the expected cost (EC), a metric introduced in every statistical learning course but rarely used in the machine learning literature. We show that both the standard and balanced error rates are special cases of the EC. Further, we show its relation with F-beta score and MCC and argue that EC is superior to these traditional metrics for being based on first principles from statistics, and for being more general, interpretable, and adaptable to any application scenario. The metrics mentioned above measure the quality of hard decisions. Yet, most modern classification systems output continuous scores for the classes which we may want to evaluate directly. Metrics for measuring the quality of system scores include the area under the ROC curve, equal error rate, cross-entropy, Brier score, and Bayes EC or Bayes risk, among others. The last three metrics are special cases of a family of metrics given by the expected value of proper scoring rules (PSRs). We review the theory behind these metrics, showing that they are a principled way to measure the quality of the posterior probabilities produced by a system. Finally, we show how to use these metrics to compute a system's calibration loss and compare this metric with the widely-used expected calibration error (ECE), arguing that calibration loss based on PSRs is superior to the ECE for being more interpretable, more general, and directly applicable to the multi-class case, among other reasons.

Motivation & Objective

• To evaluate and compare common classification metrics, especially those assessing hard decisions and system scores.
• To argue that expected cost (EC) is superior to standard metrics like F-beta and accuracy due to its statistical foundations and flexibility.
• To demonstrate that calibration loss based on proper scoring rules (PSRs) is a more robust and interpretable alternative to the expected calibration error (ECE).
• To promote the adoption of EC and PSR-based metrics in machine learning by providing theoretical justification and open-source code.
• To provide a unified, principled framework for evaluating classification systems across diverse application scenarios.

Proposed method

• The paper formally defines expected cost (EC) as a generalization of standard and balanced error rates, derived from statistical decision theory.
• It frames the evaluation of system scores through proper scoring rules (PSRs), which are strictly proper and sensitive to calibration quality.
• The authors introduce calibration loss as the difference between actual expected PSR and its minimum achievable value after optimal calibration.
• They compare PSR-based calibration loss with ECE, highlighting ECE’s limitations in interpretability, binning sensitivity, and multi-class handling.
• Theoretical derivations show that EC and PSRs are special cases of broader statistical principles, making them more general and interpretable.
• An open-source repository is provided to compute all metrics, enabling reproducibility and practical adoption.

Experimental results

Research questions

• RQ1 How does expected cost (EC) compare to standard metrics like F-beta and accuracy in terms of interpretability and generalization?
• RQ2 In what ways does calibration loss based on proper scoring rules (PSRs) improve upon the expected calibration error (ECE)?
• RQ3 Why are PSRs more suitable than discrimination-only metrics like AUC and EER for evaluating posterior probability quality?
• RQ4 How can EC serve as a unifying framework for evaluating both hard decisions and calibrated scores?
• RQ5 What are the practical advantages of using PSR-based metrics over ECE in multi-class and real-world deployment scenarios?

Key findings

• Expected cost (EC) generalizes both standard and balanced error rates, making it a more flexible and principled metric for evaluating hard decisions.
• The F-beta score and standard accuracy are outperformed by EC in terms of interpretability and statistical grounding, especially in imbalanced or uncertain scenarios.
• Calibration loss computed via proper scoring rules (PSRs) is more interpretable and robust than the expected calibration error (ECE), which relies on fragile histogram binning.
• PSR-based metrics are sensitive to both discrimination and calibration, unlike AUC and EER, which are invariant to calibration issues.
• The minimum value of the expected PSR after calibration provides a lower bound for system performance, enabling precise measurement of calibration loss.
• The open-source repository enables full reproducibility of all results, supporting the practical adoption of EC and PSR-based metrics.