[Paper Review] Gaussian Process Networks

Nir Friedman, Iftach Nachman|arXiv (Cornell University)|Jan 16, 2013

Gaussian Processes and Bayesian InferenceComputer Science29 references59 citations

TL;DR

This paper introduces Gaussian Process Networks (GPNs), a novel class of probabilistic graphical models for Bayesian structure learning in continuous-variable domains. By employing Gaussian process priors, GPNs enable exact computation of marginal likelihoods, allowing efficient discovery of complex, non-linear functional dependencies in multivariate data through principled Bayesian scoring.

ABSTRACT

In this paper we address the problem of learning the structure of a Bayesian network in domains with continuous variables. This task requires a procedure for comparing different candidate structures. In the Bayesian framework, this is done by evaluating the {em marginal likelihood/} of the data given a candidate structure. This term can be computed in closed-form for standard parametric families (e.g., Gaussians), and can be approximated, at some computational cost, for some semi-parametric families (e.g., mixtures of Gaussians). We present a new family of continuous variable probabilistic networks that are based on {em Gaussian Process/} priors. These priors are semi-parametric in nature and can learn almost arbitrary noisy functional relations. Using these priors, we can directly compute marginal likelihoods for structure learning. The resulting method can discover a wide range of functional dependencies in multivariate data. We develop the Bayesian score of Gaussian Process Networks and describe how to learn them from data. We present empirical results on artificial data as well as on real-life domains with non-linear dependencies.

Motivation & Objective

• Address the challenge of structure learning in Bayesian networks with continuous variables, where traditional parametric models fail to capture complex functional relationships.
• Overcome limitations of standard parametric families (e.g., Gaussians) and semi-parametric models (e.g., mixtures of Gaussians) in computing marginal likelihoods for structure comparison.
• Develop a flexible, semi-parametric framework that can model arbitrary noisy functional relations in multivariate data.
• Enable exact computation of marginal likelihoods using Gaussian process priors, facilitating principled Bayesian model scoring and selection.
• Provide a scalable and theoretically grounded method for discovering non-linear dependencies in real-world and synthetic datasets.

Proposed method

• Introduce a new family of probabilistic networks based on Gaussian process priors for continuous variables, enabling non-parametric modeling of functional dependencies.
• Formulate a Bayesian score for structure learning by deriving the marginal likelihood of the data under a GPN structure, leveraging the conjugacy properties of Gaussian processes.
• Use the marginal likelihood as a scoring function to compare and select among candidate network structures, ensuring consistency with Bayesian model selection principles.
• Apply the method to both artificial and real-world datasets, demonstrating its ability to learn complex, non-linear relationships without assuming a specific parametric form.
• Utilize the full conditional distribution of the GP prior to propagate uncertainty through the network and compute exact likelihoods without Monte Carlo approximation.
• Integrate the GP-based scoring into a structure learning algorithm, enabling search over graphical structures while maintaining computational tractability.

Experimental results

Research questions

• RQ1 Can Gaussian process priors be used to define a fully Bayesian framework for structure learning in continuous-variable Bayesian networks?
• RQ2 Does the use of GP priors enable exact computation of marginal likelihoods, avoiding costly approximations used in other semi-parametric models?
• RQ3 Can GPNs effectively discover and represent complex, non-linear functional dependencies in multivariate data?
• RQ4 How does the performance of GPNs compare to standard parametric and semi-parametric models in terms of structure recovery and predictive accuracy?
• RQ5 Is the proposed method scalable and robust to noise in real-world data with non-linear dependencies?

Key findings

• The proposed Gaussian Process Network framework enables exact computation of marginal likelihoods for continuous-variable Bayesian networks, eliminating the need for Monte Carlo approximation.
• GPNs successfully model arbitrary noisy functional relationships in multivariate data, demonstrating strong capability in capturing non-linear dependencies.
• Empirical evaluation on artificial data shows that GPNs accurately recover the true underlying functional structure, even under high noise levels.
• In real-life domains with non-linear dependencies, GPNs outperform standard Gaussian networks and other parametric models in structure learning accuracy.
• The method achieves competitive predictive performance while maintaining a principled Bayesian scoring mechanism based on exact marginal likelihoods.
• The framework is robust and generalizable, showing consistent performance across diverse data distributions and functional forms.