[Paper Review] Quit When You Can: Efficient Evaluation of Ensembles with Ordering Optimization

Serena Wang, Maya R. Gupta+1 more|arXiv (Cornell University)|Jun 28, 2018

Machine Learning and Data Classification19 references3 citations

TL;DR

This paper proposes Quit When You Can (QWYC), a greedy algorithm that jointly optimizes the evaluation order of base models in a classifier ensemble and early-stopping thresholds to minimize average evaluation time while preserving full ensemble accuracy. It achieves 2x–4x speedups in practice and provides a 4-approximation guarantee for certain cases, outperforming heuristic orderings and prior methods like Fan et al. (2002).

ABSTRACT

Given a classifier ensemble and a set of examples to be classified, many examples may be confidently and accurately classified after only a subset of the base models in the ensemble are evaluated. This can reduce both mean latency and CPU while maintaining the high accuracy of the original ensemble. To achieve such gains, we propose jointly optimizing a fixed evaluation order of the base models and early-stopping thresholds. Our proposed objective is a combinatorial optimization problem, but we provide a greedy algorithm that achieves a 4-approximation of the optimal solution for certain cases. For those cases, this is also the best achievable polynomial time approximation bound unless $P = NP$. Experiments on benchmark and real-world problems show that the proposed Quit When You Can (QWYC) algorithm can speed-up average evaluation time by $2$x--$4$x, and is around $1.5$x faster than prior work. QWYC's joint optimization of ordering and thresholds also performed better in experiments than various fixed orderings, including gradient boosted trees' ordering.

Motivation & Objective

• To reduce average inference latency and computational cost in ensemble classification without sacrificing accuracy.
• To address the inefficiency of evaluating all base models for every example, especially when some examples are easily classified by a subset.
• To jointly optimize model ordering and early-stopping thresholds rather than relying on heuristic ordering or fixed pruning strategies.
• To develop a practical, scalable algorithm for dynamic ensemble pruning that can be applied to real-world and benchmark datasets.
• To demonstrate that joint optimization of ordering and thresholds yields better performance than pre-selected orderings combined with existing early-stopping mechanisms.

Proposed method

• Proposes a combinatorial optimization objective that jointly selects an optimal evaluation order of T base models and 2T early-stopping thresholds (positive and negative) to minimize average number of models evaluated.
• Introduces a simple early-stopping rule: classify as positive if cumulative score exceeds the r-th positive threshold ε⁺_r after r models; otherwise continue.
• Develops a greedy algorithm called Quit When You Can (QWYC) to approximate the optimal solution, achieving a 4-approximation for certain ensemble types.
• The 4-approximation bound is tight under standard complexity assumptions (unless P=NP), making it the best possible polynomial-time approximation for the problem class.
• Applies the method to both independently-trained and jointly-trained ensembles, including real-world production systems with unlabeled test data.
• Extends the framework to QWYC*, which optimizes ordering and thresholds without requiring labeled test data, enabling deployment in data-scarce production environments.

Experimental results

Research questions

• RQ1 Can joint optimization of model ordering and early-stopping thresholds significantly reduce average evaluation time in ensemble models?
• RQ2 What is the theoretical approximation bound achievable for this joint optimization problem in polynomial time?
• RQ3 How does QWYC compare to heuristic ordering strategies (e.g., accuracy, diversity, gradient boosting order) combined with early stopping?
• RQ4 Can the method be applied effectively in real-world settings where labeled test data is unavailable?
• RQ5 Does joint optimization outperform state-of-the-art methods like Fan et al. (2002), especially when using more flexible early-stopping criteria?

Key findings

• QWYC achieves 2x–4x speedup in average evaluation time across benchmark and real-world datasets while maintaining full ensemble accuracy.
• QWYC’s joint optimization of ordering and thresholds outperforms all fixed orderings, including gradient-boosted tree ordering, in terms of evaluation time vs. accuracy trade-off.
• QWYC* achieves 1.8x–4x faster inference than Fan et al. (2002)* on real-world systems, even without labeled test data.
• The 4-approximation bound is optimal under standard complexity assumptions, meaning no better polynomial-time approximation is possible unless P=NP.
• The method remains effective even when base models are highly similar (e.g., independently trained ensembles), where random ordering performs poorly.
• QWYC’s joint optimization is more effective than combining Fan et al. (2002)’s flexible early-stopping thresholds with any pre-selected ordering, demonstrating the value of joint design.