[Paper Review] Hyperbolic Random Forests

Lars Doorenbos, Pablo Márquez-Neila+2 more|arXiv (Cornell University)|Aug 25, 2023

Hydrocarbon exploration and reservoir analysisEngineering3 citations

TL;DR

This paper proposes HoroRF, a hyperbolic random forest classifier that generalizes random forests to hyperbolic space by using horospheres as splitting surfaces instead of hyperplanes. By combining a large-margin classifier to find informative horospheres and introducing hyperclasses and class-balanced loss for multi-class and imbalanced data, HoroRF achieves state-of-the-art performance on hierarchical and imbalanced benchmarks, outperforming both Euclidean and existing hyperbolic classifiers.

ABSTRACT

Hyperbolic space is becoming a popular choice for representing data due to the hierarchical structure - whether implicit or explicit - of many real-world datasets. Along with it comes a need for algorithms capable of solving fundamental tasks, such as classification, in hyperbolic space. Recently, multiple papers have investigated hyperbolic alternatives to hyperplane-based classifiers, such as logistic regression and SVMs. While effective, these approaches struggle with more complex hierarchical data. We, therefore, propose to generalize the well-known random forests to hyperbolic space. We do this by redefining the notion of a split using horospheres. Since finding the globally optimal split is computationally intractable, we find candidate horospheres through a large-margin classifier. To make hyperbolic random forests work on multi-class data and imbalanced experiments, we furthermore outline a new method for combining classes based on their lowest common ancestor and a class-balanced version of the large-margin loss. Experiments on standard and new benchmarks show that our approach outperforms both conventional random forest algorithms and recent hyperbolic classifiers.

Motivation & Objective

• To address the lack of tree-based classifiers in hyperbolic space, which is well-suited for hierarchical data due to its exponential volume growth.
• To overcome the limitations of existing hyperbolic classifiers—such as hyperbolic SVMs and logistic regression—on complex, hierarchical, and imbalanced datasets.
• To develop a scalable and effective method for constructing decision trees in hyperbolic space using horospheres as splitting surfaces.
• To enable effective multi-class and class-imbalanced learning in hyperbolic space through novel components: hyperclasses and a class-balanced loss function.

Proposed method

• Replaces Euclidean hyperplane splits with horospheres as splitting surfaces in decision trees, leveraging the geometric properties of hyperbolic space.
• Uses a large-margin binary classifier (HoroSplitter) to efficiently search for high-information-gain horospheres, avoiding intractable enumeration.
• Introduces a hyperclass heuristic that groups classes based on their lowest common ancestor to improve multi-class split quality.
• Employs a class-balanced version of the large-margin loss to mitigate performance degradation on imbalanced datasets.
• Constructs random forests by recursively applying HoroSplitter to build trees with horospherical splits, enabling hierarchical generalization.
• Supports end-to-end training and inference in hyperbolic space using learned embeddings and geometric operations.

Experimental results

Research questions

• RQ1 Can random forests be effectively generalized to hyperbolic space by replacing hyperplane splits with horospheric splits?
• RQ2 How can horospheric splits be efficiently found in practice, given the computational intractability of exhaustive search?
• RQ3 Can the performance of hyperbolic classifiers be improved on multi-class and imbalanced hierarchical datasets through structural modifications?
• RQ4 How does HoroRF compare to both Euclidean and existing hyperbolic classifiers on standard and novel benchmarks?

Key findings

• HoroRF outperforms conventional random forests (OblRF) and Euclidean SVMs on hierarchical and imbalanced datasets, particularly on complex, non-saturated benchmarks like 'occupation' and 'rodent' in WordNet.
• On the WordNet subtree classification task, HoroRF achieves the highest macro-F1 scores, with a 12.3% improvement over HoroSVM and 6.7% over HypMLR on the 'occupation' subtree.
• In multi-class WordNet experiments, HoroRF outperforms both Euclidean and hyperbolic baselines, achieving a 9.1% higher AUPR than OblRF on the 'solid' subtree and surpassing HypMLR and HoroSVM on all three multi-class tasks.
• Ablation studies confirm that both the hyperclass heuristic and class-balanced loss significantly improve performance, with their combination yielding the best results on the football dataset.
• Increasing embedding dimensionality from 2 to 10 dimensions leads to near-perfect AUPR on most WordNet subtrees, but HoroRF maintains strong performance even in the 'tree' subtree where higher dimensions degrade performance.