[Paper Review] Ranking Data with Continuous Labels through Oriented Recursive Partitions

Stéphan Clémençon, Mastane Achab|arXiv (Cornell University)|Jan 17, 2018

Advanced Statistical Methods and Models3 citations

TL;DR

This paper introduces continuous ranking, a supervised learning problem where the goal is to order objects using a scoring function that preserves the stochastic order of continuous labels Y. It proposes the IROC curve and IAUC criterion for performance evaluation and develops CRank, a recursive partitioning algorithm that optimizes IAUC via oriented binary trees, showing superior performance over regression and ranking baselines in empirical tests.

ABSTRACT

We formulate a supervised learning problem, referred to as continuous ranking, where a continuous real-valued label Y is assigned to an observable r.v. X taking its values in a feature space $\mathcal{X}$ and the goal is to order all possible observations x in $\mathcal{X}$ by means of a scoring function $s:\mathcal{X} ightarrow \mathbb{R}$ so that s(X) and Y tend to increase or decrease together with highest probability. This problem generalizes bi/multi-partite ranking to a certain extent and the task of finding optimal scoring functions s(x) can be naturally cast as optimization of a dedicated functional criterion, called the IROC curve here, or as maximization of the Kendall $τ$ related to the pair (s(X), Y ). From the theoretical side, we describe the optimal elements of this problem and provide statistical guarantees for empirical Kendall $τ$ maximization under appropriate conditions for the class of scoring function candidates. We also propose a recursive statistical learning algorithm tailored to empirical IROC curve optimization and producing a piecewise constant scoring function that is fully described by an oriented binary tree. Preliminary numerical experiments highlight the difference in nature between regression and continuous ranking and provide strong empirical evidence of the performance of empirical optimizers of the criteria proposed.

Motivation & Objective

• To formalize continuous ranking as a generalization of bipartite and multipartite ranking, where the label Y is continuous rather than binary or discrete.
• To establish theoretical conditions for the existence of optimal scoring functions in continuous ranking.
• To propose the IROC curve and IAUC as a natural, distribution-free performance measure for continuous ranking.
• To develop a recursive statistical learning algorithm (CRank) that optimizes the empirical IAUC criterion through oriented binary partitions.
• To empirically demonstrate that continuous ranking is fundamentally different from regression and that IAUC-based learning yields superior ordering performance.

Proposed method

• Formulates continuous ranking as an optimization problem over a functional criterion, the IROC curve, which generalizes the ROC curve to continuous labels.
• Defines the IAUC (integrated AUC) as a summary scalar measure of performance, equivalent to the expected Kendall tau between s(X) and Y.
• Proposes the CRank algorithm, a recursive partitioning method that builds a piecewise constant scoring function via an oriented binary tree.
• At each node, CRank performs binary classification on a subset of data using a median-based threshold to split the feature space, optimizing IAUC locally.
• Uses a pruning strategy based on cross-validation to select the optimal subtree from the full tree sequence, balancing complexity and performance.
• Empirically evaluates the method using a unidimensional toy example with known Y = m(X), comparing IAUC, Kendall tau, and MSE across CRank, Kendall-optimized CRank, and CART.

Experimental results

Research questions

• RQ1 Under what conditions does an optimal scoring function exist for continuous ranking, and how can it be characterized in terms of stochastic order?
• RQ2 How can the performance of a scoring function be quantitatively measured in the continuous ranking setting, and what criteria generalize AUC from binary classification?
• RQ3 Can recursive partitioning be adapted to optimize IAUC in continuous ranking, and does it yield a stable, interpretable scoring function?
• RQ4 How does continuous ranking differ in nature from standard regression, and why is minimizing MSE suboptimal for ranking tasks?
• RQ5 What is the empirical performance of IAUC-optimized learning compared to regression and Kendall tau maximization in continuous ranking?

Key findings

• CRank achieves an IAUC of 0.95 on the toy example, significantly outperforming CART (0.61) and Kendall-optimized CRank (0.94), indicating superior IAUC optimization.
• The Kendall tau of CRank is 0.92, which is substantially higher than CART’s 0.58, showing better rank correlation with the true continuous label.
• CART achieves the lowest MSE (7.4×10⁻⁴), but its IAUC and Kendall tau are poor, demonstrating that MSE minimization fails to preserve ranking order.
• The Kendall-optimized version of CRank achieves a high Kendall tau of 0.93 and IAUC of 0.94, confirming that IAUC and Kendall tau are closely aligned in practice.
• Empirical results show that the least squares criterion (MSE) is inadequate for continuous ranking, as it fails to capture local oscillations critical for correct ordering.
• The CRank algorithm successfully learns a piecewise constant scoring function via oriented recursive partitions, producing a hierarchical, interpretable ranking model.