[Paper Review] Faster Boosting with Smaller Memory

Julaiti Alafate, Yoav Freund|arXiv (Cornell University)|Jan 25, 2019

Machine Learning and Data Classification17 references4 citations

TL;DR

This paper proposes Sparrow, a memory-efficient boosting algorithm that achieves 10–100x speedup over XGBoost and LightGBM on large datasets by combining early stopping, effective sample size estimation, and stratified weighted sampling. The method dynamically maintains a small, representative training sample in memory, reducing I/O overhead while preserving model accuracy, especially when data exceeds available RAM.

ABSTRACT

State-of-the-art implementations of boosting, such as XGBoost and LightGBM, can process large training sets extremely fast. However, this performance requires that the memory size is sufficient to hold a 2-3 multiple of the training set size. This paper presents an alternative approach to implementing the boosted trees, which achieves a significant speedup over XGBoost and LightGBM, especially when the memory size is small. This is achieved using a combination of three techniques: early stopping, effective sample size, and stratified sampling. Our experiments demonstrate a 10-100 speedup over XGBoost when the training data is too large to fit in memory.

Motivation & Objective

• To address the performance bottleneck of gradient boosting on large datasets that exceed available main memory.
• To reduce training time in memory-constrained environments without sacrificing model accuracy.
• To design a boosting algorithm that dynamically maintains a small, representative sample of training data in memory.
• To enable efficient training on datasets larger than main memory using novel sampling and stopping strategies.

Proposed method

• Employs early stopping based on sequential analysis to minimize the number of examples scanned per boosting iteration.
• Introduces the effective number of examples ($n_{\text{eff}}$) to quantify the statistical reliability of gradient estimates under skewed weight distributions.
• Uses stratified weighted sampling to efficiently sample from weighted data with high skew, ensuring at most 50% rejection rate.
• Dynamically flushes and replaces the in-memory sample when $n_{\text{eff}}/n$ drops below a threshold, ensuring representativeness.
• Combines these techniques into a new boosting framework, Sparrow, which avoids bias from fixed sampling thresholds used in prior work.
• Implements a stopping rule that balances estimation accuracy and computational cost using confidence bounds.

Experimental results

Research questions

• RQ1 Can boosting be accelerated significantly when training data exceeds available main memory, without sacrificing accuracy?
• RQ2 How can the number of examples scanned per boosting iteration be reduced while maintaining reliable gradient estimation?
• RQ3 What metric can effectively quantify the statistical quality of a weighted sample in boosting?
• RQ4 Can a sampling strategy be designed that remains efficient even when weights are highly skewed?
• RQ5 How does dynamic sample replacement based on effective sample size improve training efficiency?

Key findings

• Sparrow achieves 10–20x faster training than XGBoost and LightGBM on the bathymetry dataset (600M examples), even when memory is larger than the dataset size.
• On the splice site dataset (50M examples), Sparrow trains successfully on 8GB memory, while XGBoost's external memory mode requires at least 15GB and runs 3x slower.
• LightGBM and in-memory XGBoost crash when training on less than 244GB memory, whereas Sparrow remains stable and efficient.
• Sparrow outperforms both baselines in AUROC and training speed across all memory settings, especially in low-memory regimes.
• The method maintains high accuracy comparable to full-data baselines, with no bias introduced by sampling, unlike prior approaches using fixed thresholds.