[Paper Review] Top-down particle filtering for Bayesian decision trees

Balaji Lakshminarayanan, Daniel M. Roy+1 more|arXiv (Cornell University)|Mar 3, 2013

Bayesian Modeling and Causal Inference27 references17 citations

TL;DR

This paper proposes a top-down sequential Monte Carlo (SMC) algorithm for Bayesian decision trees that mimics classic greedy tree learners like ID3 and C4.5, using particle filtering with resampling to approximate the posterior distribution. It achieves predictive accuracy comparable to state-of-the-art MCMC methods while being more than ten times faster, offering a superior computation-accuracy trade-off.

ABSTRACT

Decision tree learning is a popular approach for classification and regression in machine learning and statistics, and Bayesian formulations---which introduce a prior distribution over decision trees, and formulate learning as posterior inference given data---have been shown to produce competitive performance. Unlike classic decision tree learning algorithms like ID3, C4.5 and CART, which work in a top-down manner, existing Bayesian algorithms produce an approximation to the posterior distribution by evolving a complete tree (or collection thereof) iteratively via local Monte Carlo modifications to the structure of the tree, e.g., using Markov chain Monte Carlo (MCMC). We present a sequential Monte Carlo (SMC) algorithm that instead works in a top-down manner, mimicking the behavior and speed of classic algorithms. We demonstrate empirically that our approach delivers accuracy comparable to the most popular MCMC method, but operates more than an order of magnitude faster, and thus represents a better computation-accuracy tradeoff.

Motivation & Objective

• To develop a Bayesian inference method for decision trees that operates in a top-down, greedy manner like classical algorithms (e.g., ID3, C4.5), rather than via iterative MCMC updates.
• To improve the computational efficiency of Bayesian decision tree learning without sacrificing predictive accuracy.
• To enable posterior approximation in Bayesian decision trees using a sequential Monte Carlo framework that supports resampling and particle-based tree growth.
• To evaluate whether top-down particle filtering can achieve performance competitive with MCMC-based Bayesian tree learning while being significantly faster.

Proposed method

• The method uses a sequential Monte Carlo (SMC) framework to sample a population of decision trees in a top-down, greedy fashion, starting from the root and expanding nodes step-by-step.
• At each step, particles (trees) are propagated forward using a proposal distribution based on local likelihoods and data splits, with resampling to focus on high-posterior-probability trees.
• The algorithm employs a prior over decision trees that depends on the data extent in each block, following Chipman et al. (1998), to regularize tree complexity.
• Two proposal strategies are evaluated: the prior proposal (simple, fast) and the one-step optimal proposal (more accurate but slower), with performance trade-offs analyzed.
• Tree growth is guided by conditional likelihoods at each node, with resampling used to eliminate low-likelihood particles and concentrate on better-fitting structures.
• The method avoids explicit pruning by relying on the prior and likelihood to naturally limit tree depth and prevent overfitting.

Experimental results

Research questions

• RQ1 Can a top-down, particle-based SMC algorithm achieve comparable predictive accuracy to MCMC-based Bayesian decision tree inference?
• RQ2 Does the proposed SMC method offer a significant speed advantage over existing MCMC approaches while maintaining posterior approximation quality?
• RQ3 How do different proposal distributions (prior vs. optimal) affect the efficiency and accuracy of the SMC algorithm in Bayesian decision tree learning?
• RQ4 Can the SMC framework be made more efficient than MCMC by leveraging the greedy, hierarchical structure of decision trees?

Key findings

• The proposed SMC algorithm achieves predictive accuracy on par with the most popular MCMC method for Bayesian decision trees, as measured by test accuracy and log predictive probability.
• The SMC method runs more than an order of magnitude faster than the MCMC baseline, delivering a superior computation-accuracy trade-off.
• The one-step optimal proposal improves convergence per particle but is too computationally expensive overall, especially on datasets with few irrelevant features.
• Expanding multiple nodes simultaneously degrades performance, indicating that sequential expansion is more effective.
• More sophisticated node selection strategies did not improve performance, suggesting that simple heuristics suffice for effective particle propagation.
• The Bayesian SMC approach outperforms the classic CART algorithm in terms of log predictive probability, even when CART is tuned for accuracy and smoothed with Laplacian correction.