[Paper Review] A study on cost behaviors of binary classification measures in class-imbalanced problems

Bao-Gang Hu, Weiming Dong|arXiv (Cornell University)|Mar 26, 2014

Imbalanced Data Classification Techniques28 references22 citations

TL;DR

This paper introduces a cost-function-based theoretical framework to evaluate binary classification measures in class-imbalanced problems, revealing that measures like G-means of accuracy rates and balance error rate (BER) exhibit 'proper' cost behaviors—assigning higher cost to misclassifications from the minority class—while F1, G-means of precision/recall, MCC, and Kappa do not. The study establishes a meta-measure criterion for identifying 'proper' performance metrics in imbalanced learning scenarios.

ABSTRACT

This work investigates into cost behaviors of binary classification measures in a background of class-imbalanced problems. Twelve performance measures are studied, such as F measure, G-means in terms of accuracy rates, and of recall and precision, balance error rate (BER), Matthews correlation coefficient (MCC), Kappa coefficient, etc. A new perspective is presented for those measures by revealing their cost functions with respect to the class imbalance ratio. Basically, they are described by four types of cost functions. The functions provides a theoretical understanding why some measures are suitable for dealing with class-imbalanced problems. Based on their cost functions, we are able to conclude that G-means of accuracy rates and BER are suitable measures because they show "proper" cost behaviors in terms of "a misclassification from a small class will cause a greater cost than that from a large class". On the contrary, F1 measure, G-means of recall and precision, MCC and Kappa coefficient measures do not produce such behaviors so that they are unsuitable to serve our goal in dealing with the problems properly.

Motivation & Objective

• To address the theoretical gap in understanding why some performance measures work better than others in class-imbalanced learning.
• To investigate the intrinsic cost behaviors of binary classification measures as the class-imbalance ratio increases.
• To define and apply a 'meta-measure' criterion—where misclassifying a minority-class instance incurs higher cost than a majority-class misclassification—to evaluate measure suitability.
• To distinguish between 'proper' and 'improper' measures based on their implicit cost functions, independent of data or algorithms.
• To provide a theoretical foundation for selecting learning criteria in cost-free, imbalanced-data classification.

Proposed method

• The study derives exact or approximate cost functions for 12 binary classification measures, including F1, G-means, BER, MCC, and Kappa, with respect to the class-imbalance ratio.
• A new meta-measure is introduced to assess whether a performance measure implicitly assigns higher cost to minority-class misclassifications.
• Theoretical analysis classifies measures into four types of cost functions, with two distinct behaviors: 'proper' (minority misclassification penalized more) and 'improper' (equal or lower penalty for minority errors).
• Numerical examples are constructed using synthetic data with varying imbalance ratios (p2 = 0.5, 0.1, 0.01, 0.001, 0.0001, 0.00001) to validate theoretical findings.
• The function-based evaluation is prioritized over performance-based evaluation to isolate intrinsic properties of measures from data and algorithmic noise.
• Theoretical cost function analysis enables comparison of measure equivalences and differences without relying on empirical datasets or learning algorithms.

Experimental results

Research questions

• RQ1 Why do some performance measures perform better than others in highly imbalanced binary classification problems?
• RQ2 What are the implicit cost functions of common binary classification measures as the class-imbalance ratio increases?
• RQ3 Which measures inherently penalize minority-class misclassifications more heavily, aligning with the principle that such errors are more costly?
• RQ4 How can a meta-measure be defined to distinguish 'proper' from 'improper' performance measures in imbalanced learning?
• RQ5 To what extent do theoretical cost behaviors predict real-world performance differences across measures?

Key findings

• G-means of accuracy rates and balance error rate (BER) exhibit 'proper' cost behaviors, assigning higher cost to minority-class misclassifications, making them suitable for imbalanced problems.
• F1 measure, G-means of precision and recall, Matthews correlation coefficient (MCC), and Kappa coefficient do not show proper cost behaviors and are therefore 'improper' for imbalanced learning.
• The cost function analysis reveals that BER and G-means of accuracy rates are theoretically equivalent in their cost behavior, both satisfying the meta-measure criterion.
• Numerical results confirm that BER and G-means of accuracy rates maintain stable, optimal decision boundaries across increasing imbalance ratios, while F1 and G_PR show deteriorating performance.
• The study demonstrates that measures like A_T (overall accuracy) and F1 are 'improper' because they fail to increase penalty for minority-class errors, potentially degrading model quality in imbalanced settings.
• The theoretical framework provides a basis for selecting learning criteria independent of data or algorithm, emphasizing that 'what to learn' is the primary issue in machine learning.