[Paper Review] Large Random Forests: Optimisation for Rapid Evaluation

Frederik Gossen, Bernhard Steffen|arXiv (Cornell University)|Dec 23, 2019

Machine Learning and Data Classification20 references4 citations

TL;DR

This paper proposes a method to compress large Random Forests into a single, semantically equivalent Algebraic Decision Diagram (ADD), enabling classification speeds up to several orders of magnitude faster while reducing memory usage by up to 99.99%. The approach uses ADD-based aggregation, majority vote abstraction, and unfeasible path elimination to eliminate redundancy without altering prediction accuracy or variance.

ABSTRACT

Random Forests are one of the most popular classifiers in machine learning. The larger they are, the more precise is the outcome of their predictions. However, this comes at a cost: their running time for classification grows linearly with the number of trees, i.e. the size of the forest. In this paper, we propose a method to aggregate large Random Forests into a single, semantically equivalent decision diagram. Our experiments on various popular datasets show speed-ups of several orders of magnitude, while, at the same time, also significantly reducing the size of the required data structure.

Motivation & Objective

• To address the high classification runtime of large Random Forests, which scales linearly with the number of trees.
• To reduce memory footprint and improve evaluation efficiency without altering prediction semantics or variance.
• To explore holistic aggregation of entire forests into compact decision diagrams using algebraic and semantic optimization techniques.
• To evaluate whether decision diagram-based optimization can outperform existing methods in both speed and size reduction.

Proposed method

• Transform a Random Forest into a single Algebraic Decision Diagram (ADD) using canonical aggregation based on a fixed predicate ordering.
• Apply compositional abstraction by pre-evaluating class frequencies at compile time to reduce structural complexity.
• Perform non-compositional abstraction by collapsing the majority vote outcome into a single value per path, preserving classification semantics.
• Eliminate unfeasible paths—those with contradictory predicates—using a don't-care-style minimization to further reduce size and improve evaluation speed.
• Leverage ADD-specific operations such as concatenation, addition, and reduction to optimize both runtime and memory usage.
• Use variable ordering heuristics to minimize the resulting diagram size, ensuring canonical and minimal representations.

Experimental results

Research questions

• RQ1 Can large Random Forests be compressed into a single decision diagram while preserving their classification accuracy and variance?
• RQ2 To what extent can ADD-based aggregation reduce classification time and memory footprint compared to standard Random Forest evaluation?
• RQ3 Does path elimination for unfeasible paths prevent exponential blowup and further improve performance in practice?
• RQ4 How do compositional and non-compositional abstractions compare in terms of size and speed gains?
• RQ5 Can this method be generalized to other classifiers or data types beyond standard UCI datasets?

Key findings

• The proposed method achieved speed-ups of several orders of magnitude, with classification times reduced by up to 1000x on standard UCI datasets.
• Memory usage was reduced by up to 99.99% in some cases, with the final decision diagram being significantly smaller than the original forest.
• On the Balance Scale dataset, the forest size of 2,158,330 nodes was compressed to just 144 nodes, a 99.99% reduction.
• The Breast Cancer dataset saw a reduction from 5,494,682 to 3,760 nodes, a 99.93% size decrease, with minimal loss in accuracy.
• Unfeasible path elimination prevented exponential blowup and further reduced diagram size, even making some diagrams smaller than the original forest.
• The method preserved the original Random Forest’s variance and prediction accuracy, confirming semantic equivalence across all test cases.