[Paper Review] Optimal Binary Classification Beyond Accuracy

Shashank Kumar Singh, Justin Khim|arXiv (Cornell University)|Jul 5, 2021

Imbalanced Data Classification Techniques4 citations

TL;DR

This paper generalizes the Bayes-optimal classifier beyond accuracy to any performance metric derived from the confusion matrix, proving that stochastic classifiers can outperform deterministic ones under imbalanced data. It introduces Uniform Class Imbalance as a formalized sub-type of imbalance and provides finite-sample guarantees for k-NN classification, showing stochastic thresholding improves F1 scores in real-world fraud detection.

ABSTRACT

The vast majority of statistical theory on binary classification characterizes performance in terms of accuracy. However, accuracy is known in many cases to poorly reflect the practical consequences of classification error, most famously in imbalanced binary classification, where data are dominated by samples from one of two classes. The first part of this paper derives a novel generalization of the Bayes-optimal classifier from accuracy to any performance metric computed from the confusion matrix. Specifically, this result (a) demonstrates that stochastic classifiers sometimes outperform the best possible deterministic classifier and (b) removes an empirically unverifiable absolute continuity assumption that is poorly understood but pervades existing results. We then demonstrate how to use this generalized Bayes classifier to obtain regret bounds in terms of the error of estimating regression functions under uniform loss. Finally, we use these results to develop some of the first finite-sample statistical guarantees specific to imbalanced binary classification. Specifically, we demonstrate that optimal classification performance depends on properties of class imbalance, such as a novel notion called Uniform Class Imbalance, that have not previously been formalized. We further illustrate these contributions numerically in the case of $k$-nearest neighbor classification

Motivation & Objective

• To address the lack of finite-sample theoretical guarantees for imbalanced binary classification under non-accuracy metrics.
• To formalize sub-types of class imbalance, particularly Uniform Class Imbalance, which has not been previously distinguished in theory or practice.
• To demonstrate that stochastic classifiers can outperform deterministic ones when optimizing general performance metrics.
• To derive regret bounds in terms of regression function estimation error under uniform loss.
• To provide empirical validation of theoretical findings using real-world credit card fraud data.

Proposed method

• Derives a generalized Bayes-optimal classifier for any performance metric computed from the confusion matrix, relaxing unverifiable absolute continuity assumptions.
• Proves that optimal performance is only guaranteed via stochastic classifiers under general metrics, not deterministic ones.
• Introduces a novel notion of Uniform Class Imbalance to characterize how imbalance severity affects optimal classification.
• Establishes regret bounds in terms of L∞-loss estimation error of the class probability regression function.
• Proposes an efficient O(n log n) algorithm for threshold optimization over empirical risk under general metrics.
• Empirically evaluates k-NN with optimal deterministic and stochastic thresholding on the credit card fraud dataset, using F1 score as the metric.

Experimental results

Research questions

• RQ1 Can the Bayes-optimal classifier be generalized beyond accuracy to arbitrary performance metrics derived from the confusion matrix?
• RQ2 Under what conditions do stochastic classifiers outperform deterministic classifiers in optimal classification?
• RQ3 How does the nature of class imbalance—particularly Uniform Class Imbalance—affect optimal classification performance?
• RQ4 What finite-sample statistical guarantees can be derived for nonparametric classifiers like k-NN under non-accuracy metrics?
• RQ5 To what extent does stochastic thresholding improve performance on real imbalanced datasets like credit card fraud?

Key findings

• Stochastic classifiers can achieve optimal performance under general metrics, while deterministic classifiers may not, even when the latter are optimal under accuracy.
• The generalized Bayes classifier is derived without requiring empirically unverifiable absolute continuity assumptions, improving theoretical robustness.
• Uniform Class Imbalance is formally defined as a sub-type of imbalance that significantly affects k-NN performance, distinguishing it from other imbalance patterns.
• Finite-sample regret bounds are established in terms of L∞-loss estimation error of the regression function, enabling performance comparison across algorithms.
• In real-world credit card fraud detection, stochastic thresholding consistently outperforms deterministic thresholding in F1 score across varying imbalance ratios and k values.
• The proposed O(n log n) algorithm enables efficient threshold optimization for general performance metrics, making the approach scalable to large datasets.