[Paper Review] Infinite Mixtures of Multivariate Gaussian Processes

Shiliang Sun|arXiv (Cornell University)|Jul 26, 2013

Gaussian Processes and Bayesian Inference7 references13 citations

TL;DR

This paper proposes an infinite mixture of multivariate Gaussian processes (IMMGP) to model multimodal vector-valued functions and enable multitask learning, leveraging a Dirichlet process prior to infer an unknown, data-driven number of components. The model uses Markov chain Monte Carlo sampling for joint inference of latent variables and hyperparameters, achieving improved regression performance on synthetic data by capturing complex, non-Gaussian data structures while mitigating computational complexity.

ABSTRACT

This paper presents a new model called infinite mixtures of multivariate Gaussian processes, which can be used to learn vector-valued functions and applied to multitask learning. As an extension of the single multivariate Gaussian process, the mixture model has the advantages of modeling multimodal data and alleviating the computationally cubic complexity of the multivariate Gaussian process. A Dirichlet process prior is adopted to allow the (possibly infinite) number of mixture components to be automatically inferred from training data, and Markov chain Monte Carlo sampling techniques are used for parameter and latent variable inference. Preliminary experimental results on multivariate regression show the feasibility of the proposed model.

Motivation & Objective

• To address the limitations of standard multivariate Gaussian processes in modeling multimodal data and handling large-scale data due to cubic computational complexity.
• To extend the infinite mixture framework—successful in univariate Gaussian processes—to the multivariate setting for joint learning of related tasks.
• To enable automatic inference of the number of mixture components via a Dirichlet process prior, avoiding manual model selection.
• To develop a scalable inference and prediction framework using Markov chain Monte Carlo methods for complex, high-dimensional output spaces.

Proposed method

• Employs a Dirichlet process prior over an infinite number of multivariate Gaussian process components to allow nonparametric inference of component count.
• Models input space using a full-covariance Gaussian distribution per component, with conjugate priors on mean and precision matrices.
• Uses a Chinese restaurant process metaphor to compute the probability of assigning new data points to existing or new components.
• Applies MCMC sampling to jointly infer latent component assignments, hyperparameters, and model parameters through Gibbs and slice sampling.
• Performs prediction by marginalizing over component assignments and latent functions, using Monte Carlo approximation for predictive distributions.
• Incorporates non-informative priors for new components and uses Monte Carlo integration to approximate intractable predictive densities.

Experimental results

Research questions

• RQ1 Can an infinite mixture of multivariate Gaussian processes effectively model multimodal, vector-valued data that standard multivariate GPs fail to capture?
• RQ2 How can the number of mixture components be automatically inferred from data without manual tuning, using a nonparametric Bayesian approach?
• RQ3 Can the proposed model maintain computational feasibility while improving regression accuracy on multivariate outputs compared to standard multitask learning methods?
• RQ4 How does the inclusion of new components during prediction affect performance, and what is the trade-off between model complexity and accuracy?

Key findings

• The proposed IMMGP model achieves a test root mean squared error (RMSE) of 0.7963 on a synthetic multivariate regression dataset, significantly outperforming the MTLNN baseline (RMSE = 2.0659).
• Including the option to assign new components during prediction (IMMGP2) yields nearly identical performance to the model restricted to existing components (IMMGP1), indicating robustness and stability in component selection.
• The model successfully captures multimodal data structures through mixture components, overcoming the unimodal limitation of standard multivariate Gaussian processes.
• The use of Dirichlet process priors enables automatic inference of the number of components, eliminating the need for prior specification and reducing model selection burden.
• MCMC-based inference effectively explores the posterior over latent variables and hyperparameters, enabling accurate prediction despite high-dimensional and complex output spaces.
• The predictive mean is computed via Monte Carlo marginalization over component assignments, with analytical computation of expectations under Gaussian process priors.