[Paper Review] Dirichlet Process Mixtures of Generalized Linear Models

Lauren A. Hannah, David M. Blei+1 more|arXiv (Cornell University)|Sep 28, 2009

Bayesian Methods and Mixture Models3 citations

TL;DR

This paper proposes Dirichlet Process Mixtures of Generalized Linear Models (DP-GLM), a Bayesian nonparametric regression method that combines the flexibility of Dirichlet process mixtures with the modeling power of GLMs to handle diverse response types, covariates, and heteroscedasticity. It establishes conditions for asymptotic unbiasedness of the mean function estimate and demonstrates superior performance across continuous, categorical, and count responses compared to CART, Bayesian trees, and Gaussian processes.

ABSTRACT

We propose Dirichlet Process mixtures of Generalized Linear Models (DP-GLM), a new method of nonparametric regression that accommodates continuous and categorical inputs, and responses that can be modeled by a generalized linear model. We prove conditions for the asymptotic unbiasedness of the DP-GLM regression mean function estimate. We also give examples for when those conditions hold, including models for compactly supported continuous distributions and a model with continuous covariates and categorical response. We empirically analyze the properties of the DP-GLM and why it provides better results than existing Dirichlet process mixture regression models. We evaluate DP-GLM on several data sets, comparing it to modern methods of nonparametric regression like CART, Bayesian trees and Gaussian processes. Compared to existing techniques, the DP-GLM provides a single model (and corresponding inference algorithms) that performs well in many regression settings.

Motivation & Objective

• To develop a general-purpose nonparametric regression method that handles continuous, categorical, and count responses with mixed covariate types.
• To model complex, non-linear response functions while capturing heteroscedasticity through local GLM components.
• To use a Bayesian nonparametric approach that allows the data to determine the number of mixture components automatically.
• To establish theoretical conditions for asymptotic unbiasedness of the DP-GLM mean function estimator.
• To empirically evaluate DP-GLM against modern nonparametric methods like CART, Bayesian trees, and Gaussian processes.

Proposed method

• Uses a Dirichlet process prior to nonparametrically mix a finite number of generalized linear models (GLMs), each with its own mean function and response distribution.
• Each component in the mixture is a GLM with a linear predictor, allowing modeling of exponential family responses including Gaussian, Bernoulli, and Poisson.
• The overall mean function is formed by marginalizing over the mixture component weights, enabling non-linear response surfaces through component uncertainty.
• Employs Gibbs sampling with collapsed and auxiliary variable techniques for posterior computation, including Hamiltonian dynamics for non-conjugate parameters.
• Applies conjugate and log-normal base measures for parameters, with hyperpriors on precision and component variances to improve fit and flexibility.
• Uses a data-driven number of components via the Chinese Restaurant Process, avoiding overfitting and enabling model complexity adaptation.

Experimental results

Research questions

• RQ1 Under what conditions is the DP-GLM mean function estimator asymptotically unbiased?
• RQ2 Can the DP-GLM model non-linear response functions while accommodating diverse response types and heteroscedasticity?
• RQ3 How does the DP-GLM compare in predictive performance to CART, Bayesian trees, and Gaussian processes across varied regression tasks?
• RQ4 What are the statistical properties of the posterior distribution under the DP-GLM, particularly in high-dimensional or non-exchangeable settings?
• RQ5 How do hyperparameter choices and base measure selection affect model fit and computational efficiency?

Key findings

• The DP-GLM achieves asymptotic unbiasedness of the mean function estimator under general conditions, including when the true mean function is continuous and the base measures have sufficient weak support.
• The model successfully captures heteroscedasticity by allowing each GLM component to have distinct variance and mean structure, outperforming conjugate base measure models in this regard.
• On the CMB, CCS, and Solar datasets, DP-GLM demonstrated strong predictive performance, with run times under 500 seconds in Matlab and under 10 seconds in optimized Java implementations.
• The model outperformed CART, Bayesian trees, and Gaussian processes in multiple regression settings, particularly in handling mixed covariate types and non-linear response patterns.
• The use of log-normal hyperpriors on variance parameters significantly improved response fit, especially in capturing heteroscedasticity, compared to conjugate priors.
• Posterior convergence was stable within 1,000 iterations, with log posterior probabilities stabilizing early, supporting efficient MCMC sampling.