[Paper Review] Comparison-Based Random Forests

Siavash Haghiri, Damien Garreau+1 more|arXiv (Cornell University)|Jun 18, 2018

Face and Expression Recognition7 references4 citations

TL;DR

This paper proposes Comparison-Based Random Forests (CompRF), a novel random forest algorithm that performs classification and regression using only triplet comparisons—answering whether one item is closer to a second or third—without requiring explicit vector representations or distance values. The method achieves competitive performance across Euclidean, metric, and purely comparison-based settings, outperforming embedding-based baselines in efficiency and accuracy while proving statistical consistency under mild conditions.

ABSTRACT

Assume we are given a set of items from a general metric space, but we neither have access to the representation of the data nor to the distances between data points. Instead, suppose that we can actively choose a triplet of items (A,B,C) and ask an oracle whether item A is closer to item B or to item C. In this paper, we propose a novel random forest algorithm for regression and classification that relies only on such triplet comparisons. In the theory part of this paper, we establish sufficient conditions for the consistency of such a forest. In a set of comprehensive experiments, we then demonstrate that the proposed random forest is efficient both for classification and regression. In particular, it is even competitive with other methods that have direct access to the metric representation of the data.

Motivation & Objective

• To develop a random forest algorithm that operates solely on triplet comparisons, avoiding reliance on explicit data representations or distance metrics.
• To address the limitations of ordinal embedding methods, which are slow and introduce distortions in data representation.
• To establish theoretical consistency for a comparison-based random forest under a simplified model.
• To demonstrate empirical effectiveness of CompRF in classification and regression tasks across Euclidean, metric, and purely comparison-based settings.

Proposed method

• The method constructs comparison trees using random pivot points with opposing labels from within each node’s data subset.
• At each node, the algorithm splits data based on whether points are closer to one pivot or the other, using only triplet comparison queries.
• The forest aggregates M such trees, with predictions averaged over all trees (regression) or via majority vote (classification).
• Tree construction avoids vector space assumptions by relying solely on relative distance comparisons via oracle queries.
• The algorithm is designed to be consistent under mild conditions, with theoretical analysis focusing on a simplified variant of the tree-building process.
• In practice, the method uses $ n_0 = 1 $ and $ M = 200 $, and is evaluated on datasets with known distances and in purely comparison-based regimes.

Experimental results

Research questions

• RQ1 Can a random forest be consistently trained using only triplet comparison queries, without access to vector representations or distance values?
• RQ2 How does the performance of a comparison-based random forest compare to traditional methods like CART forests when only distances are available?
• RQ3 Does the proposed method outperform indirect approaches that first embed data into Euclidean space via ordinal embedding?
• RQ4 What is the impact of using actively selected triplets versus random triplets in the comparison-based setting?
• RQ5 Can the method maintain high accuracy and low computational cost in large-scale, comparison-only settings?

Key findings

• CompRF outperforms kernel SVM on the MUTAG and ENZYMES datasets in the distance-available setting, despite requiring no kernel construction.
• In the comparison-based setting, CompRF achieves lower or comparable classification error than TSTE-based embedding methods, especially when using the same triplet queries.
• When using the same set of triplets, CompRF consistently outperforms embedding-based methods, demonstrating superior robustness and efficiency.
• The method is significantly faster than ordinal embedding techniques like TSTE, which are computationally prohibitive on large datasets.
• In all tested settings—Euclidean, metric, and comparison-based—CompRF maintains strong performance, showing broad applicability.
• Theoretical analysis establishes consistency for a simplified variant of the algorithm, providing a foundation for future extensions to more realistic tree construction processes.