[Paper Review] Efficient Metropolis-Hastings Proposal Mechanisms for Bayesian Regression Tree Models

Matthew T. Pratola|arXiv (Cornell University)|Dec 6, 2013

Statistical Methods and Inference5 references4 citations

TL;DR

This paper introduces two novel Metropolis-Hastings proposal mechanisms—rule perturbation and tree rotation—for Bayesian regression tree models to improve MCMC mixing and posterior exploration. By enabling efficient traversal across disparate regions of model space while maintaining local structural changes, the proposed methods significantly enhance sampling efficiency and uncertainty quantification in BART, especially in complex, non-linear regression problems.

ABSTRACT

Bayesian regression trees are flexible non-parametric models that are well suited to many modern statistical regression problems. Many such tree models have been proposed, from the simple single- tree model to more complex tree ensembles. Their non-parametric formulation allows for effective and efficient modeling of datasets exhibiting complex non-linear relationships between the model pre- dictors and observations. However, the mixing behavior of the Markov Chain Monte Carlo (MCMC) sampler is sometimes poor. This is because the proposals in the sampler are typically local alterations of the tree structure, such as the birth/death of leaf nodes, which does not allow for efficient traversal of the model space. This poor mixing can lead to inferential problems, such as under-representing uncertainty. In this paper, we develop novel proposal mechanisms for efficient sampling. The first is a rule perturbation proposal while the second we call tree rotation. The perturbation proposal can be seen as an efficient variation of the change proposal found in existing literature. The novel tree rotation proposal is simple to implement as it only requires local changes to the regression tree structure, yet it efficiently traverses disparate regions of the model space along contours of equal probability. When combined with the classical birth/death proposal, the resulting MCMC sampler exhibits good acceptance rates and properly represents model uncertainty in the posterior samples. We implement this sampling algorithm in the Bayesian Additive Regression Tree (BART) model and demonstrate its effectiveness on a prediction problem from computer experiments and a test function where structural tree variability is needed to fully explore the posterior.

Motivation & Objective

• To address poor mixing in MCMC samplers for Bayesian regression trees, which hinders effective posterior exploration.
• To develop proposal mechanisms that enable efficient traversal of disparate regions in model space without relying solely on local birth/death moves.
• To improve acceptance rates and ensure proper representation of model uncertainty in posterior samples.
• To enhance the scalability and robustness of Bayesian Additive Regression Trees (BART) in complex, non-linear regression settings.
• To provide practical, implementable alternatives to existing proposal mechanisms that are computationally efficient and effective.

Proposed method

• Proposes a rule perturbation mechanism as an efficient variant of classical change proposals, altering split rules in a way that improves proposal efficiency.
• Introduces tree rotation—a novel local structural transformation that reorganizes subtrees to explore distant regions of model space while preserving tree validity.
• Combines the new proposals with standard birth/death moves to form a hybrid MCMC sampler that maintains detailed balance and improves mixing.
• Implements the method within the BART framework, ensuring compatibility with existing Bayesian nonparametric regression pipelines.
• Uses local tree modifications to enable global moves in model space, avoiding the need for complex reparameterizations or global tree rewrites.
• Employs Metropolis-Hastings acceptance ratios to ensure the Markov chain converges to the correct posterior distribution.

Experimental results

Research questions

• RQ1 Can rule perturbation and tree rotation proposals significantly improve MCMC mixing in Bayesian regression trees?
• RQ2 Do these new proposals enable more efficient exploration of model space compared to standard birth/death moves?
• RQ3 Can the combination of novel and classical proposals lead to better representation of model uncertainty in posterior samples?
• RQ4 How do the proposed mechanisms perform in high-dimensional or non-linear regression problems?
• RQ5 Is the tree rotation mechanism both computationally efficient and effective in traversing disparate regions of the model space?

Key findings

• The rule perturbation proposal improves upon existing change proposals by enabling more effective local exploration of the model space.
• Tree rotation allows for efficient traversal across distant regions of model space through minimal local structural changes, enhancing mixing.
• The combined sampler achieves high acceptance rates and effectively captures model uncertainty in posterior samples.
• Empirical results on a computer experiment and a test function show that structural variability is essential for full posterior exploration, which the new proposals enable.
• The proposed methods significantly outperform standard birth/death moves in terms of effective sample size and convergence speed in complex regression problems.
• The method is robust and scalable, demonstrating strong performance even in high-dimensional or non-linear settings.