[Paper Review] Technical Note: Towards ROC Curves in Cost Space

José Hernández‐Orallo, Peter Flach+1 more|arXiv (Cornell University)|Jul 29, 2011

Imbalanced Data Classification Techniques16 references3 citations

TL;DR

This paper introduces ROC cost curves—a novel visualization that maps ROC curves into cost space by selecting thresholds such that the proportion of positive predictions matches the operating condition (cost proportion or skew). It establishes a linear relationship between the area under the ROC cost curve and AUC, and shows that for evenly spaced scores, this area equals the Brier score, providing the first direct link between AUC and the Brier score.

ABSTRACT

ROC curves and cost curves are two popular ways of visualising classifier performance, finding appropriate thresholds according to the operating condition, and deriving useful aggregated measures such as the area under the ROC curve (AUC) or the area under the optimal cost curve. In this note we present some new findings and connections between ROC space and cost space, by using the expected loss over a range of operating conditions. In particular, we show that ROC curves can be transferred to cost space by means of a very natural way of understanding how thresholds should be chosen, by selecting the threshold such that the proportion of positive predictions equals the operating condition (either in the form of cost proportion or skew). We call these new curves {ROC Cost Curves}, and we demonstrate that the expected loss as measured by the area under these curves is linearly related to AUC. This opens up a series of new possibilities and clarifies the notion of cost curve and its relation to ROC analysis. In addition, we show that for a classifier that assigns the scores in an evenly-spaced way, these curves are equal to the Brier Curves. As a result, this establishes the first clear connection between AUC and the Brier score.

Motivation & Objective

• To clarify the relationship between ROC space and cost space, which has been historically inconsistent in the literature.
• To define a new class of cost curves—ROC cost curves—that correspond directly to points on the ROC curve.
• To establish a formal connection between the area under ROC cost curves and the AUC, demonstrating a linear relationship.
• To show that for classifiers with evenly spaced scores, the area under ROC cost curves equals the Brier score, linking two key performance metrics.
• To provide a principled framework for threshold choice in cost-sensitive learning using rate-based thresholding.

Proposed method

• Proposes a threshold choice method where the threshold is selected such that the proportion of positive predictions equals the cost proportion (or skew), ensuring equal x-axis segment lengths in cost space.
• Derives the expected loss over cost proportions and shows that the resulting curve—ROC cost curve—is a piecewise linear function in cost space.
• Uses a change of variable from cost proportion to skew to generalize the formulation and derive the linear relationship between AUC and the area under the ROC cost curve.
• Applies the method to classifiers with evenly spaced scores, showing that the ROC cost curve becomes identical to the Brier curve under this condition.
• Demonstrates that the area under the ROC cost curve is mathematically equivalent to a linear transformation of AUC, with exact equality in specific cases (e.g., random, perfect, worst classifiers).
• Validates the theoretical results through graphical examples and numerical computation, comparing ROC cost curves with optimal cost curves and Brier curves.

Experimental results

Research questions

• RQ1 How can ROC curves be meaningfully represented in cost space, given the lack of a direct correspondence in prior work?
• RQ2 What threshold choice method enables a one-to-one mapping between ROC curve points and cost space representations?
• RQ3 Is there a formal relationship between the area under the ROC cost curve and the AUC?
• RQ4 Can the Brier score be interpreted as an area under a cost curve, and under what conditions?
• RQ5 How do ROC cost curves compare to existing representations like the optimal cost curve and Brier curves?

Key findings

• The area under the ROC cost curve is linearly related to the AUC, with the exact relationship expressed as $ \text{Area} = 2\pi_1\pi_0(1 - \text{AUC}) + \frac{1}{3} - \pi_1\pi_0 $.
• For classifiers with evenly spaced scores, the ROC cost curve coincides with the Brier curve, and the area under the ROC cost curve equals the Brier score.
• This establishes the first formal and direct connection between the AUC and the Brier score, both of which are widely used performance metrics.
• The ROC cost curve provides a valid and interpretable alternative to the optimal cost curve, with each segment corresponding to a point on the ROC curve and reflecting the same performance information.
• In the case of a perfectly calibrated classifier with tied scores, the ROC cost curve may make sub-optimal threshold choices compared to the optimal cost curve, highlighting a trade-off in the rate-based method.
• Numerical validation shows near-identical areas under the ROC cost curve and Brier curve (0.2006 vs. 0.2047), with minor discrepancies due to tie-handling in implementation.