[Paper Review] On Theory for BART

Veronika Ročková, Enakshi Saha|arXiv (Cornell University)|Oct 1, 2018

Bayesian Methods and Mixture Models35 references4 citations

TL;DR

This paper establishes theoretical optimality for the Bayesian Additive Regression Trees (BART) method by analyzing its exact prior through heterogeneous Galton-Watson branching processes. It introduces a modified prior that achieves optimal posterior concentration rates, proving BART achieves the minimax rate of posterior contraction under regularity conditions.

ABSTRACT

Ensemble learning is a statistical paradigm built on the premise that many weak learners can perform exceptionally well when deployed collectively. The BART method of Chipman et al. (2010) is a prominent example of Bayesian ensemble learning, where each learner is a tree. Due to its impressive performance, BART has received a lot of attention from practitioners. Despite its wide popularity, however, theoretical studies of BART have begun emerging only very recently. Laying the foundations for the theoretical analysis of Bayesian forests, Rockova and van der Pas (2017) showed optimal posterior concentration under conditionally uniform tree priors. These priors deviate from the actual priors implemented in BART. Here, we study the exact BART prior and propose a simple modification so that it also enjoys optimality properties. To this end, we dive into branching process theory. We obtain tail bounds for the distribution of total progeny under heterogeneous Galton-Watson (GW) processes exploiting their connection to random walks. We conclude with a result stating the optimal rate of posterior convergence for BART.

Motivation & Objective

• To address the lack of theoretical foundations for the exact BART prior, which differs from the priors used in prior optimal posterior concentration results.
• To close the gap between practical BART implementations and theoretical guarantees by analyzing the actual prior used in BART.
• To develop a modified prior that retains the practical appeal of BART while achieving optimal posterior concentration rates.
• To establish the optimal rate of posterior convergence for BART using tools from branching process theory.

Proposed method

• The paper models the BART prior as a heterogeneous Galton-Watson branching process to analyze the distribution of tree sizes and progeny.
• It derives tail bounds for the total progeny of heterogeneous Galton-Watson processes using connections to random walk theory.
• The analysis leverages generating functions and moment generating functions to characterize the stochastic behavior of tree growth under the BART prior.
• A modified prior is proposed that ensures the posterior concentration rate matches the minimax optimal rate.
• Theoretical results are derived by linking tree structure to branching process properties, particularly the extinction probability and population size distribution.
• The paper establishes posterior concentration rates by combining probabilistic bounds on tree size with regularity conditions on the regression function.

Experimental results

Research questions

• RQ1 Can the exact BART prior be shown to achieve optimal posterior concentration rates under standard smoothness assumptions?
• RQ2 How does the distribution of tree sizes under the actual BART prior compare to the conditionally uniform priors used in prior theoretical work?
• RQ3 What modifications to the BART prior are necessary to achieve optimal posterior concentration while preserving its practical properties?
• RQ4 What role do heterogeneous Galton-Watson processes play in characterizing the complexity of BART trees?
• RQ5 What is the optimal rate of posterior contraction for BART under the actual prior, and can it be achieved?

Key findings

• The paper establishes that the original BART prior does not achieve optimal posterior concentration due to its specific distributional properties on tree size.
• A modified BART prior is proposed that ensures optimal posterior concentration at the minimax rate under Hölder-smooth regression functions.
• Tail bounds for the total progeny of heterogeneous Galton-Watson processes are derived, enabling precise control over tree complexity.
• The optimal posterior concentration rate for BART is shown to be the minimax rate, confirming its adaptivity to unknown smoothness.
• The analysis confirms that the theoretical performance of BART can be rigorously justified when the prior is appropriately modified.