[Paper Review] Optimal Binary Classifier Aggregation for General Losses

Akshay Balsubramani, Yoav Freund|arXiv (Cornell University)|Oct 1, 2015

Face and Expression Recognition14 references3 citations

TL;DR

This paper proposes a minimax-optimal framework for aggregating binary classifiers in semi-supervised learning under general loss functions, including convex and non-convex losses. It derives a decision rule that applies sigmoid-like functions to a weighted ensemble margin, computed via convex optimization, achieving minimax optimality without relaxations or model assumptions.

ABSTRACT

We address the problem of aggregating an ensemble of predictors with known loss bounds in a semi-supervised binary classification setting, to minimize prediction loss incurred on the unlabeled data. We find the minimax optimal predictions for a very general class of loss functions including all convex and many non-convex losses, extending a recent analysis of the problem for misclassification error. The result is a family of semi-supervised ensemble aggregation algorithms which are as efficient as linear learning by convex optimization, but are minimax optimal without any relaxations. Their decision rules take a form familiar in decision theory -- applying sigmoid functions to a notion of ensemble margin -- without the assumptions typically made in margin-based learning.

Motivation & Objective

• Address the limitation of prior work restricted to misclassification error by extending to general loss functions, including log loss and cost-weighted losses.
• Formulate the ensemble aggregation problem as a two-player zero-sum game between a predictor and an adversary to ensure worst-case optimality.
• Develop a scalable, efficient algorithm that learns optimal classifier weights via convex optimization without requiring model-specific assumptions.
• Generalize the framework to handle heterogeneous ensembles, including specialist classifiers that abstain on subsets of data, by reweighting constraints based on prediction coverage.
• Establish theoretical guarantees that the optimal prediction rule depends only on the loss function and ensemble predictions, not on arbitrary hyperparameters or relaxations.

Proposed method

• Model the problem as a minimax game where the predictor selects predictions and the adversary sets unknown randomized labels within constraints derived from ensemble classifier performance.
• Use the partial loss functions ℓ₊(g) and ℓ₋(g) to define the expected loss ℓ(zⱼ, gⱼ) as a convex combination based on the label's randomized value zⱼ ∈ [−1, 1].
• Derive the dual formulation via convex duality, transforming the minimax problem into a convex optimization problem over dual variables σ ∈ ℝᵖ.
• The optimal prediction gⱼ* is computed as a sigmoid-like function of the weighted ensemble margin: gⱼ* = g(σ*ᵀxⱼ), where σ* minimizes a regularized dual objective.
• Introduce L₁ regularization on σ to handle label correlation deviations, enabling robustness and cross-validation over the regularization parameter ε.
• Generalize the framework to specialist classifiers by reweighting the feature matrix F into S using per-classifier coverage weights ρᵢ(xⱼ), preserving theoretical guarantees.

Experimental results

Research questions

• RQ1 Can we derive a minimax-optimal ensemble aggregation rule for general loss functions beyond classification error?
• RQ2 How can we maintain optimality and efficiency when the loss function is non-convex or asymmetric in false positives and false negatives?
• RQ3 What is the structure of the optimal prediction rule in terms of ensemble margins and loss geometry?
• RQ4 How can we extend the framework to handle specialist classifiers that abstain on subsets of test examples?
• RQ5 Can the resulting algorithm be efficiently implemented via convex optimization without relaxation or heuristic tuning?

Key findings

• The optimal prediction for each test example is a sigmoid-like function of a linear combination of ensemble predictions, derived from the dual variable σ*.
• The method achieves minimax optimality for all loss functions satisfying Assumption 1—i.e., monotonic partial losses—without requiring convexity or symmetry.
• The dual optimization problem is convex and scalable, with the optimal weights σ* obtained by minimizing a regularized objective involving the conjugate of the loss function.
• The framework generalizes to heterogeneous ensembles with specialist classifiers by reweighting the feature matrix S to reflect per-classifier coverage, preserving theoretical guarantees.
• The decision rule depends solely on the loss function and the ensemble’s prediction structure, with no free parameters or model assumptions.
• The approach subsumes and generalizes prior results on misclassification error and proper losses, such as log loss, by unifying them under a single minimax framework.