[Paper Review] A better Beta for the H measure of classification performance

David J. Hand, Christoforos Anagnostopoulos|arXiv (Cornell University)|Feb 12, 2012

Imbalanced Data Classification Techniques7 references4 citations

TL;DR

This paper proposes the Beta(π₁+1, π₀+1) distribution as a superior universal standard for the H measure of classification performance, replacing the previously suggested Beta(2,2). It ensures classifier-invariant performance evaluation by aligning the cost distribution with class imbalance, offering better coherence for unbalanced datasets while preserving symmetry and defaulting to Beta(2,2) in balanced cases.

ABSTRACT

The area under the ROC curve is widely used as a measure of performance of classification rules. However, it has recently been shown that the measure is fundamentally incoherent, in the sense that it treats the relative severities of misclassifications differently when different classifiers are used. To overcome this, Hand (2009) proposed the $H$ measure, which allows a given researcher to fix the distribution of relative severities to a classifier-independent setting on a given problem. This note extends the discussion, and proposes a modified standard distribution for the $H$ measure, which better matches the requirements of researchers, in particular those faced with heavily unbalanced datasets, the $Beta(π_1+1,π_0+1)$ distribution. [Preprint submitted at Pattern Recognition Letters]

Motivation & Objective

• Address the fundamental incoherence of the AUC measure in classification performance evaluation, particularly under class imbalance.
• Overcome the limitations of the AUC by introducing the H measure, which fixes the relative cost distribution in a classifier-independent manner.
• Provide a more appropriate, problem-specific universal standard distribution for the H measure that reflects real-world class imbalance.
• Replace the previous default Beta(2,2) distribution with a distribution that adapts to class priors, improving relevance and coherence in imbalanced settings.
• Enable researchers to incorporate domain knowledge about misclassification costs while maintaining a principled, default distribution for standardized reporting.

Proposed method

• Propose a modified cost distribution for the H measure based on the Beta(π₁+1, π₀+1) distribution, which centers the mode at the positive class prior π₁.
• Enforce symmetry in the cost distribution under class label swap by ensuring that swapping π₀ and π₁ reflects the distribution appropriately, satisfying the invariance requirement.
• Use a parameter k ≥ 3 to control dispersion around the mode, with k=3 as a default for minimal uncertainty, allowing flexibility for expert-estimated cost beliefs.
• Allow expert elicitation of relative misclassification cost ratios r = c/(1−c), which are transformed into a mode estimate c̃ = r/(1+r) to inform the Beta distribution parameters.
• Construct the H measure using a weighted average of performance across thresholds, with the cost distribution serving as the weighting measure to ensure invariance.
• Demonstrate that the proposed distribution reduces to Beta(2,2) when π₁ = π₀ = 0.5, ensuring consistency with the previous standard in balanced cases.

Experimental results

Research questions

• RQ1 How can the H measure be improved to better reflect the relative severity of misclassification errors in unbalanced classification problems?
• RQ2 Why is the previous default Beta(2,2) distribution for the H measure inadequate for imbalanced datasets?
• RQ3 What distributional form ensures both symmetry under class label reversal and proper alignment with class priors in the H measure?
• RQ4 Can a universal standard distribution for the H measure be defined that adapts to class imbalance while remaining coherent and interpretable?
• RQ5 How can domain-specific expert knowledge about misclassification costs be systematically incorporated into the H measure’s cost distribution?

Key findings

• The Beta(π₁+1, π₀+1) distribution is proposed as a more coherent and appropriate universal standard for the H measure than the previously suggested Beta(2,2).
• The new distribution ensures symmetry under class label reversal, satisfying the invariance requirement that the H measure must meet to be classifier-independent.
• The distribution is centered at c = π₁, meaning the relative cost of misclassifying a positive instance is set to the prior probability of the positive class.
• When π₁ = π₀ = 0.5, the proposed distribution reduces to Beta(2,2), maintaining consistency with the prior standard.
• The parameter k controls dispersion around the mode, with k=3 as a default, allowing researchers to adjust confidence in the cost estimate.
• The method enables researchers to incorporate expert knowledge about relative misclassification costs via the mode parameter, improving practical relevance and interpretability.