[Paper Review] Bayesian quantile additive regression trees

Bereket P. Kindo, Hao Wang+2 more|arXiv (Cornell University)|Jul 10, 2016

Statistical Methods and Inference4 references3 citations

TL;DR

This paper proposes Bayesian quantile additive regression trees (BayesQArt), a novel method that uses asymmetric Laplace likelihood with mixture representations for efficient posterior computation in quantile regression. It demonstrates strong predictive performance across simulation studies and real-world data, including non-linear relationships and high-dimensional classification, outperforming linear quantile regression and random forests in key benchmarks.

ABSTRACT

Ensemble of regression trees have become popular statistical tools for the estimation of conditional mean given a set of predictors. However, quantile regression trees and their ensembles have not yet garnered much attention despite the increasing popularity of the linear quantile regression model. This work proposes a Bayesian quantile additive regression trees model that shows very good predictive performance illustrated using simulation studies and real data applications. Further extension to tackle binary classification problems is also considered.

Motivation & Objective

• To address the lack of ensemble methods for quantile regression by proposing a Bayesian framework for additive regression trees.
• To enable robust estimation of conditional quantiles under skewed or heavy-tailed error distributions.
• To extend the method to binary classification problems with non-linear decision boundaries.
• To improve predictive accuracy in high-dimensional settings where p >> n.
• To provide a computationally efficient Bayesian approach using MCMC with conjugate priors and mixture representations.

Proposed method

• Employs an asymmetric Laplace distribution (ALD) likelihood to link quantile regression to Bayesian inference via latent variables.
• Uses a mixture representation of ALD involving exponential and normal variates to enable efficient Gibbs sampling.
• Applies reversible jump MCMC to jointly estimate tree structures and node parameters, including split rules and terminal node values.
• Introduces GROW and PRUNE moves to explore tree space efficiently, reducing computational cost.
• Adapts the model for binary classification by modeling the probability of class membership through quantile-specific tree ensembles.
• Uses predictive performance metrics like classification error rate and AUC to evaluate model accuracy.

Experimental results

Research questions

• RQ1 Can a Bayesian additive regression tree model effectively estimate conditional quantiles while maintaining computational tractability?
• RQ2 How does the proposed method compare to linear quantile regression and random forests in terms of predictive accuracy for non-linear and high-dimensional data?
• RQ3 Can the Bayesian quantile additive regression tree model handle extreme or skewed response distributions more robustly than mean-based methods?
• RQ4 Does the model maintain strong performance in high-dimensional classification settings with p >> n?
• RQ5 Can the mixture representation of the asymmetric Laplace distribution enable efficient MCMC sampling for tree-structured quantile regression?

Key findings

• In simulation studies with non-linear boundaries, BayesQArt achieved a test error rate of 0.2185 (SE = 0.0100) and AUC of 0.8297 (SE = 0.0088), outperforming random forests (error = 0.2600, AUC = 0.6363).
• For high-dimensional cancer classification (p ≈ 10,000, n = 200), BayesQArt achieved a test error rate of 0.1650 (SE = 0.0170) and AUC of 0.8712 (SE = 0.0165), outperforming random forests (error = 0.1800, AUC = 0.8184).
• The method demonstrated robustness in handling skewed responses and extreme observations, consistent with the theoretical advantages of quantile regression.
• The use of GROW and PRUNE moves reduced computational cost while maintaining model flexibility and predictive accuracy.
• BayesQArt showed superior performance over Bayesian linear quantile regression (BayesQR.AL), which had a high error rate of 0.5500 and AUC of 0.4684 in the simulation study.
• The model achieved an average computation time of 5.42 minutes on a standard personal computer, indicating practical scalability despite high-dimensional inputs.