[Paper Review] Calibrated Surrogate Maximization of Linear-fractional Utility in Binary Classification

Han Bao, Masashi Sugiyama|arXiv (Cornell University)|May 29, 2019

Machine Learning and Algorithms48 references4 citations

TL;DR

This paper proposes a calibrated surrogate utility for direct maximization of linear-fractional performance metrics—such as Fβ-measure and Jaccard index—in binary classification. By constructing a tractable lower bound with sufficient calibration conditions, the method enables efficient gradient-based optimization and ensures statistical consistency, outperforming plug-in methods especially in low-sample regimes.

ABSTRACT

Complex classification performance metrics such as the F${}_β$-measure and Jaccard index are often used, in order to handle class-imbalanced cases such as information retrieval and image segmentation. These performance metrics are not decomposable, that is, they cannot be expressed in a per-example manner, which hinders a straightforward application of M-estimation widely used in supervised learning. In this paper, we consider linear-fractional metrics, which are a family of classification performance metrics that encompasses many standard ones such as the F${}_β$-measure and Jaccard index, and propose methods to directly maximize performances under those metrics. A clue to tackle their direct optimization is a calibrated surrogate utility, which is a tractable lower bound of the true utility function representing a given metric. We characterize sufficient conditions which make the surrogate maximization coincide with the maximization of the true utility. Simulation results on benchmark datasets validate the effectiveness of our calibrated surrogate maximization especially if the sample sizes are extremely small.

Motivation & Objective

• Address the challenge of optimizing non-decomposable performance metrics like Fβ and Jaccard index in class-imbalanced binary classification.
• Overcome the limitations of traditional M-estimation due to the non-decomposability of these metrics.
• Develop a model-agnostic, computationally efficient method that directly maximizes utility without iterative retraining or high-sample posterior estimation.
• Ensure statistical consistency of the empirical estimator via Z-estimation theory.
• Provide sufficient calibration conditions under which surrogate maximization aligns with true utility maximization.

Proposed method

• Introduce a calibrated surrogate utility that serves as a tractable lower bound of the true linear-fractional utility function.
• Formulate the optimization as a two-step procedure combining concave and quasi-concave programs for computational efficiency.
• Derive sufficient conditions on the surrogate calibration that guarantee the surrogate maximizer coincides with the true utility maximizer.
• Utilize gradient-based optimization (U-GD and U-BFGS) on the surrogate due to its non-vanishing gradient and unbiased gradient estimator.
• Apply Z-estimation theory to establish consistency of the empirical estimator under finite samples.
• Use a split-sample approach with $\mathcal{S}_0$ for estimating the surrogate and $\mathcal{S}_1$ for optimization to reduce bias.

Experimental results

Research questions

• RQ1 Can a calibrated surrogate utility be constructed that enables direct maximization of linear-fractional metrics in binary classification?
• RQ2 Under what conditions does maximizing the surrogate utility yield the same result as maximizing the true utility?
• RQ3 How does the proposed method compare to plug-in and iterative retraining approaches in terms of computational efficiency and sample complexity?
• RQ4 What is the impact of the calibration parameter $\tau$ on performance, especially in low-sample regimes?
• RQ5 Is the empirical estimator based on the surrogate method statistically consistent as the sample size grows?

Key findings

• The proposed calibrated surrogate maximization outperforms plug-in methods in terms of F1 and Jaccard scores when sample sizes are small (e.g., 20–40), particularly on datasets like cod-rna, diabetes, and ionosphere.
• Gradient-based methods (U-GD and U-BFGS) achieve better performance than plug-in even at sample sizes of 100–400, contradicting results on full datasets where plug-in may dominate.
• Performance improves as $\tau$ approaches 1 for many metrics and datasets, though extrema exist, indicating sensitivity to $\tau$ choice.
• Theoretical analysis confirms that the empirical estimator is consistent under the proposed calibration conditions, ensuring convergence to the true utility maximizer.
• The method is model-agnostic and avoids the high sample complexity of posterior probability estimation and the computational burden of iterative retraining.
• In low-sample regimes, the surrogate-based approach provides a reliable alternative to plug-in methods, which fail due to poor probability estimation.