[Paper Review] Breadth-first, Depth-next Training of Random Forests

Andreea Anghel, Nikolas Ioannou+4 more|arXiv (Cornell University)|Oct 15, 2019

Machine Learning and Data Classification16 references4 citations

TL;DR

This paper proposes a hybrid breadth-first-search and depth-first-search (BFS-DFS) tree-building algorithm for Random Forests, optimized for modern multi-core CPUs. By dynamically switching between BFS and DFS modes based on runtime conditions and applying low-level system optimizations—such as cache-friendly memory layouts, explicit prefetching, and reduced heap allocations—it achieves an average 7.8× speedup over state-of-the-art RF solvers (scikit-learn, H2O, xgboost), with up to 100× speedup on certain workloads.

ABSTRACT

In this paper we analyze, evaluate, and improve the performance of training Random Forest (RF) models on modern CPU architectures. An exact, state-of-the-art binary decision tree building algorithm is used as the basis of this study. Firstly, we investigate the trade-offs between using different tree building algorithms, namely breadth-first-search (BFS) and depth-search-first (DFS). We design a novel, dynamic, hybrid BFS-DFS algorithm and demonstrate that it performs better than both BFS and DFS, and is more robust in the presence of workloads with different characteristics. Secondly, we identify CPU performance bottlenecks when generating trees using this approach, and propose optimizations to alleviate them. The proposed hybrid tree building algorithm for RF is implemented in the Snap Machine Learning framework, and speeds up the training of RFs by 7.8x on average when compared to state-of-the-art RF solvers (sklearn, H2O, and xgboost) on a range of datasets, RF configurations, and multi-core CPU architectures.

Motivation & Objective

• To improve the performance of Random Forest training on modern multi-core CPU architectures by analyzing and optimizing tree-building algorithms.
• To investigate the trade-offs between breadth-first-search (BFS) and depth-first-search (DFS) tree construction in terms of memory access patterns and cache efficiency.
• To design a hybrid tree-building algorithm that dynamically selects between BFS and DFS modes to adapt to varying workloads and hardware characteristics.
• To identify and resolve system-level performance bottlenecks in RF training, including memory access patterns and dynamic memory allocation overhead.
• To implement and evaluate the proposed optimizations within the Snap Machine Learning framework for real-world performance gains.

Proposed method

• Design a dynamic hybrid BFS-DFS algorithm that switches between breadth-first and depth-first traversal based on runtime workload characteristics and performance heuristics.
• Implement a single, read-only pre-sorted matrix in shared memory, reused across all trees in the forest to amortize the cost of sorting.
• Optimize memory access patterns to be CPU cache-friendly by reorganizing data structures and improving spatial locality.
• Employ explicit hardware prefetching to hide memory latency and improve instruction-level parallelism.
• Reduce dynamic memory allocation overhead by reusing memory pools and minimizing heap allocations during tree construction.
• Integrate the optimized tree-building pipeline into the Snap Machine Learning framework for end-to-end evaluation.

Experimental results

Research questions

• RQ1 How do BFS and DFS tree-building strategies compare in terms of performance and cache efficiency on modern multi-core CPUs?
• RQ2 Can a hybrid BFS-DFS algorithm dynamically adapt to different workloads and outperform both pure BFS and DFS strategies?
• RQ3 What system-level bottlenecks hinder high-performance Random Forest training on modern CPUs?
• RQ4 To what extent can low-level optimizations such as cache-aware memory layout and prefetching improve training throughput?
• RQ5 How does the proposed hybrid algorithm perform across diverse datasets, RF configurations, and CPU architectures compared to state-of-the-art solvers?

Key findings

• The hybrid BFS-DFS algorithm outperforms both pure BFS and DFS strategies by dynamically adapting to workload characteristics, resulting in better robustness and performance.
• On average, the proposed implementation achieves a 7.8× speedup over scikit-learn, H2O, and xgboost across multiple datasets and CPU architectures.
• On the IBM Power9 system, the speedup reaches up to 100× for certain configurations, particularly with larger ensembles of 100 trees.
• With 100 trees, the speedup over H2O reaches 33.3× on the Power9 system and 15.2× on the x86 system, demonstrating strong scaling with ensemble size.
• The system-level optimizations—cache-friendly memory layout, explicit prefetching, and reduced heap allocation—collectively contribute significantly to the performance gains.
• Test accuracy remains consistent across all frameworks, confirming that the performance improvements do not compromise model generalization.