[Paper Review] Deep Bayesian regression models

Aliaksandr Hubin, Geir Storvik+1 more|arXiv (Cornell University)|Jun 6, 2018

Gaussian Processes and Bayesian Inference35 references4 citations

TL;DR

This paper introduces Deep Bayesian Regression Models (DBRM), a framework that combines deep learning's non-linear feature engineering with Bayesian inference to enable interpretable, uncertainty-quantified regression. By using MCMC-based algorithms for model selection and averaging, DBRM achieves superior predictive performance while quantifying uncertainty through credible intervals, outperforming state-of-the-art methods in benchmark examples.

ABSTRACT

Regression models are used for inference and prediction in a wide range of applications providing a powerful scientific tool for researchers and analysts from different fields. In many research fields the amount of available data as well as the number of potential explanatory variables is rapidly increasing. Variable selection and model averaging have become extremely important tools for improving inference and prediction. However, often linear models are not sufficient and the complex relationship between input variables and a response is better described by introducing non-linearities and complex functional interactions. Deep learning models have been extremely successful in terms of prediction although they are often difficult to specify and potentially suffer from overfitting. The aim of this paper is to bring the ideas of deep learning into a statistical framework which yields more parsimonious models and allows to quantify model uncertainty. To this end we introduce the class of deep Bayesian regression models (DBRM) consisting of a generalized linear model combined with a comprehensive non-linear feature space, where non-linear features are generated just like in deep learning but combined with variable selection in order to include only important features. DBRM can easily be extended to include latent Gaussian variables to model complex correlation structures between observations, which seems to be not easily possible with existing deep learning approaches. Two different algorithms based on MCMC are introduced to fit DBRM and to perform Bayesian inference. The predictive performance of these algorithms is compared with a large number of state of the art algorithms. Furthermore we illustrate how DBRM can be used for model inference in various applications.

Motivation & Objective

• To develop a statistically principled framework that integrates deep learning's non-linear feature extraction with Bayesian inference to improve model interpretability and uncertainty quantification.
• To address the limitations of deep learning—such as poor interpretability, overfitting, and lack of uncertainty measures—by embedding deep feature learning within a Bayesian regression context.
• To enable automatic, data-driven selection of relevant non-linear features through combinatorial optimization and variable selection, avoiding overfitting via informative priors.
• To extend the model to include latent Gaussian processes for capturing complex correlation structures between observations, a capability often missing in standard deep learning.
• To provide a scalable, computationally feasible approach to Bayesian deep learning that supports model averaging and marginal likelihood estimation for model comparison.

Proposed method

• Proposes a hierarchical Bayesian model where non-linear features are generated layer-by-layer using deep neural network-like transformations, but with Bayesian priors to regularize complexity.
• Employs Markov Chain Monte Carlo (MCMC) algorithms—specifically GMJMCMC and RGMJMCMC—for posterior inference, enabling exploration of model space and model averaging.
• Introduces a feature engineering strategy that combines non-linear transformations with variable selection, where only statistically significant features are retained via posterior model probabilities.
• Uses marginal likelihood estimation via bridge sampling and Laplace approximations to compare models and select optimal architectures.
• Supports the inclusion of latent Gaussian processes to model complex dependence structures in the data, extending the model beyond independent observations.
• Implements a memory-efficient, parallelized MCMC algorithm in the EMJMCMC R package, enabling scalable inference on moderate to large datasets.

Experimental results

Research questions

• RQ1 Can a Bayesian framework be effectively combined with deep feature engineering to produce models that are both highly predictive and interpretable?
• RQ2 How can model uncertainty be rigorously quantified in deep learning models using a fully Bayesian approach, rather than relying on approximations?
• RQ3 To what extent can automatic model selection and variable selection in deep feature spaces improve predictive performance compared to standard deep learning and classical regression?
• RQ4 Can latent Gaussian processes be effectively integrated into deep learning architectures to model complex correlation structures in the data?
• RQ5 What are the computational trade-offs between different strategies for optimizing feature weights in the deep feature generation process?

Key findings

• DBRM achieves superior predictive performance compared to a wide range of state-of-the-art algorithms, including deep neural networks and classical regression models, across multiple benchmark examples.
• The use of MCMC-based model averaging and marginal likelihood estimation enables well-calibrated uncertainty quantification, with credible intervals that are mathematically justified and interpretable.
• The proposed feature engineering strategy—based on non-linear transformations followed by variable selection—outperforms more complex optimization strategies in terms of computational efficiency and predictive accuracy.
• The inclusion of latent Gaussian variables allows the model to capture complex correlation structures in the data, a capability not easily attainable in standard deep learning frameworks.
• The EMJMCMC R package provides a scalable and flexible implementation that supports various prior specifications and marginal likelihood estimation techniques, enabling reproducible and extensible research.
• Despite the computational demands, the framework remains feasible for moderate to large datasets, with potential for future extension to sub-sampling MCMC techniques to scale to Big Data.