[Paper Review] NGBoost: Natural Gradient Boosting for Probabilistic Prediction

Tony Duan, Anand Avati+5 more|arXiv (Cornell University)|Oct 8, 2019

Machine Learning in HealthcareComputer Science143 citations

TL;DR

NGBoost extends gradient boosting to probabilistic regression by jointly boosting multiple distribution parameters using natural gradients, yielding calibrated predictive distributions with flexible base learners and distributions.

ABSTRACT

We present Natural Gradient Boosting (NGBoost), an algorithm for generic probabilistic prediction via gradient boosting. Typical regression models return a point estimate, conditional on covariates, but probabilistic regression models output a full probability distribution over the outcome space, conditional on the covariates. This allows for predictive uncertainty estimation -- crucial in applications like healthcare and weather forecasting. NGBoost generalizes gradient boosting to probabilistic regression by treating the parameters of the conditional distribution as targets for a multiparameter boosting algorithm. Furthermore, we show how the Natural Gradient is required to correct the training dynamics of our multiparameter boosting approach. NGBoost can be used with any base learner, any family of distributions with continuous parameters, and any scoring rule. NGBoost matches or exceeds the performance of existing methods for probabilistic prediction while offering additional benefits in flexibility, scalability, and usability. An open-source implementation is available at github.com/stanfordmlgroup/ngboost.

Motivation & Objective

• Motivate probabilistic regression and the need for uncertainty quantification in regression tasks such as healthcare and weather forecasting.
• Propose NGBoost, a modular algorithm that learns distribution parameters as functions of covariates via multiparameter boosting and natural gradients.
• Demonstrate that NGBoost is flexible (base learners, distributions, scoring rules) and scalable, with competitive probabilistic predictions.
• Empirically compare NGBoost to existing probabilistic regression methods and analyze ablations to show the benefits of natural gradients and multiparameter boosting.

Proposed method

• Represent P(y|x) with a chosen parametric distribution P_theta, where theta are learned as functions of x.
• Use proper scoring rules S (e.g., log-likelihood L, CRPS) as the training objective across the dataset.
• Compute natural gradients tilde{∇}S by pre-multiplying ordinary gradients by the inverse of the Fisher information or corresponding S-divergence metric.
• At each boosting stage, fit base learners to predict the natural gradients for each parameter, then perform a line-search based update with stage-specific scalings ρ^(m) and a common learning rate η.
• Iterate for M boosting stages to obtain theta = theta^(0) - η sum_m ρ^(m) f^(m)(x), enabling simultaneous modeling of multiple distributional parameters (e.g., μ and log σ for Normal).
• Demonstrate parameterization invariance of natural gradients and discuss computational costs (linear in N and in number of parameters p; p^3 for gradient inversions) and practical mini-batching.

Experimental results

Research questions

• RQ1 Can NGBoost provide probabilistic predictions by jointly modeling multiple distributional parameters as functions of covariates?
• RQ2 Do natural gradients improve training dynamics and predictive uncertainty calibration compared to ordinary or second-order gradients in multiparameter boosting?
• RQ3 Is NGBoost competitive with state-of-the-art probabilistic regression methods across diverse datasets and distributions?
• RQ4 How do ablations (multiparameter boosting, natural gradient, homoscedastic variance) affect performance?

Key findings

• NGBoost delivers competitive negative log-likelihood (NLL) scores on UCI regression datasets versus MC dropout, Deep Ensembles, Concrete Dropout, Gaussian Processes, GAMLSS, and Distributional Forests.
• Ablation shows that using both multiparameter boosting and the natural gradient yields best NLL, outperforming setups with only one of these components.
• NGBoost provides competitive RMSE for point estimates even though optimization targets NLL with probabilistic forecasts.
• Natural-gradient-based learning achieves more balanced updates across examples and parameters, avoiding over- and under-fitting seen with ordinary gradients.
• The method scales with the number of distributional parameters and dataset size similarly to standard boosting, with modest additional costs.
• NGBoost remains flexible, allowing various base learners, distributions, and scoring rules, and demonstrates practical usability and scalability.