[Paper Review] Proper likelihood ratio based ROC curves for general binary classification problems

Lidia Sacchetto, Mauro Gasparini|arXiv (Cornell University)|Sep 3, 2018

Advanced Statistical Methods and Models21 references3 citations

TL;DR

This paper proposes likelihood ratio (LR)-based ROC curves as a theoretically optimal alternative to conventional empirical ROC curves in binary classification. By deriving a general framework for LR-based classification applicable to diverse data types—including discrete, continuous, and complex data—it demonstrates that LR-based ROC curves are inherently concave, non-decreasing, and optimal under the Neyman-Pearson lemma, offering a more reliable and efficient alternative to standard empirical estimates, especially in multivariate and nonparametric settings.

ABSTRACT

Everybody writes that ROC curves, a very common tool in binary classification problems, should be optimal, and in particular concave, non-decreasing and above the 45-degree line. Everybody uses ROC curves, theoretical and especially empirical, which are not so. This work is an attempt to correct this schizophrenic behavior. Optimality stems from the Neyman-Pearson lemma, which prescribes using likelihood-ratio based ROC curves. Starting from there, we give the most general definition of a likelihood-ratio based classification procedure, which encompasses finite, continuous and even more complex data types. We point out a strict relationship with a general notion of concentration of two probability measures. We give some nontrivial examples of situations with non-monotone and non-continuous likelihood ratios. Finally, we propose the ROC curve of a likelihood ratio based Gaussian kernel flexible Bayes classifier as a proper default alternative to the usual empirical ROC curve.

Motivation & Objective

• To address the widespread use of improper, non-concave, and non-monotonic empirical ROC curves that lack theoretical optimality.
• To establish a general, mathematically sound definition of likelihood ratio-based classification applicable to arbitrary data types, including finite, continuous, and complex data.
• To demonstrate that LR-based ROC curves are inherently proper—concave, non-decreasing, and above the 45-degree line—thereby fulfilling optimal properties derived from the Neyman-Pearson lemma.
• To provide a practical, nonparametric alternative using a Gaussian kernel flexible Bayes classifier that produces a proper, smooth ROC curve superior to staircase-shaped empirical estimates.
• To highlight the limitations of common methods like QDA and linear discriminant analysis in multivariate settings and show that LR-based approaches dominate them in theoretical performance.

Proposed method

• Derives a general likelihood ratio-based classification rule applicable to any data space where the two population measures are mutually absolutely continuous.
• Defines the ROC curve as the parametric locus of (FPR(t), TPR(t)) where the decision rule is based on thresholding the likelihood ratio.
• Establishes a connection between the likelihood ratio and the general notion of concentration function, linking classification to measure-theoretic concepts.
• Applies the method to a real-world prostate cancer diagnosis case study using microRNA and PSA data, comparing parametric (QDA) and nonparametric (Flexible Bayes) approaches.
• Uses Algorithm 5.1 to estimate the nonparametric LR-based ROC curve via a Gaussian kernel flexible Bayes classifier, avoiding the staircase artifact of empirical methods.
• Compares the resulting ROC curves with standard empirical estimates (e.g., via R library Sing S2005) to demonstrate improved smoothness and efficiency.

Experimental results

Research questions

• RQ1 Can a general likelihood ratio-based classification framework be defined for arbitrary data types, including discrete, continuous, and complex data?
• RQ2 Why are conventional empirical ROC curves often improper (non-concave, non-monotonic), and what theoretical foundation justifies a better alternative?
• RQ3 How does the likelihood ratio-based ROC curve compare in performance to standard methods like QDA, linear discriminant analysis, and logistic regression in multivariate settings?
• RQ4 Can nonparametric estimation of the likelihood ratio yield a smooth, concave, and more efficient ROC curve than the standard empirical staircase estimate?
• RQ5 What is the relationship between the likelihood ratio and the concentration function in the context of two probability measures?

Key findings

• LR-based ROC curves are inherently concave and non-decreasing, satisfying the optimal properties derived from the Neyman-Pearson lemma, even when the likelihood ratio is not monotonic.
• The proposed method produces a proper ROC curve that is always above the 45-degree line and dominates commonly used empirical and parametric ROC curves in theoretical performance.
• In the prostate cancer case study, the nonparametric Flexible Bayes classifier based on LR estimation produced a smooth, concave ROC curve that was more efficient than the staircase-shaped empirical ROC curve.
• The LR-based ROC curve for the Flexible Bayes classifier was found to be lower than the QDA curve on average, reflecting a trade-off between generality and performance, but remained superior in estimation efficiency and smoothness.
• Theoretical analysis confirms that likelihood ratio-based classification is optimal under general conditions, regardless of data type, provided the measures are mutually absolutely continuous.
• The study demonstrates that the common use of empirical ROC curves is suboptimal and that proper, smooth, and concave ROC curves based on likelihood ratios are both theoretically justifiable and practically achievable.