[Paper Review] Graph Based Gaussian Processes on Restricted Domains

David B. Dunson, Hau‐Tieng Wu+1 more|arXiv (Cornell University)|Oct 14, 2020

Gaussian Processes and Bayesian Inference36 references4 citations

TL;DR

This paper proposes Graph Laplacian-based Gaussian Processes (GL-GPs) for nonparametric regression on restricted, complex domains by learning a covariance structure that respects the intrinsic geometry of the input space. By approximating the heat kernel via finitely many eigenpairs of a graph Laplacian constructed from Euclidean distances, GL-GPs enable geometry-aware smoothing and Nyström-based extension to new data, achieving optimal posterior contraction rates in theory and improved prediction accuracy in practice, especially on non-linear manifolds like hand surfaces.

ABSTRACT

In nonparametric regression, it is common for the inputs to fall in a restricted subset of Euclidean space. Typical kernel-based methods that do not take into account the intrinsic geometry of the domain across which observations are collected may produce sub-optimal results. In this article, we focus on solving this problem in the context of Gaussian process (GP) models, proposing a new class of Graph Laplacian based GPs (GL-GPs), which learn a covariance that respects the geometry of the input domain. As the heat kernel is intractable computationally, we approximate the covariance using finitely-many eigenpairs of the Graph Laplacian (GL). The GL is constructed from a kernel which depends only on the Euclidean coordinates of the inputs. Hence, we can benefit from the full knowledge about the kernel to extend the covariance structure to newly arriving samples by a Nyström type extension. We provide substantial theoretical support for the GL-GP methodology, and illustrate performance gains in various applications.

Motivation & Objective

• To address suboptimal performance in standard GP models when inputs lie on a restricted, non-linear domain with complex geometry.
• To develop a GP framework that learns a covariance function respecting the intrinsic geometry of the input domain without requiring explicit manifold knowledge.
• To provide theoretical guarantees on posterior contraction rates for the proposed GL-GP model under fixed design.
• To enable scalable inference and out-of-sample prediction using Nyström-type extensions based on graph Laplacian eigenpairs.
• To demonstrate empirical superiority of GL-GPs over standard Euclidean-distance GP kernels on real and synthetic data with geometric structure.

Proposed method

• Construct a weighted graph from input points using a kernel (e.g., Gaussian) based on Euclidean distances to define neighborhood structure.
• Compute the graph Laplacian matrix from the adjacency matrix and degree matrix of the graph.
• Approximate the heat kernel covariance using the first $K$ eigenpairs of the graph Laplacian, replacing the intractable heat kernel.
• Define the GP prior using the graph Laplacian-based covariance $\mathsf{C}_{\epsilon,K}(x,y,\mathbf{T})$, where $\mathbf{T}$ is a random bandwidth parameter with a prior $g(t)$.
• Enable out-of-sample prediction via a Nyström-type extension that leverages the eigen-decomposition of the graph Laplacian.
• Use a fixed design setup to derive posterior contraction rates, proving the method achieves near-optimal convergence under regularity conditions.

Experimental results

Research questions

• RQ1 Can a GP model learn a covariance function that respects the intrinsic geometry of a restricted, possibly non-linear domain without prior knowledge of its manifold structure?
• RQ2 What is the theoretical posterior contraction rate of a GP prior based on graph Laplacian approximations of the heat kernel?
• RQ3 How does the performance of the proposed GL-GP compare to standard GP models using Euclidean distance kernels on domains with complex geometry?
• RQ4 Can the GL-GP framework support scalable inference and out-of-sample prediction using eigen-decomposition and Nyström extension?
• RQ5 What are the conditions under which the GL-GP achieves optimal or near-optimal convergence rates in nonparametric regression?

Key findings

• The GL-GP achieves a posterior contraction rate of $\gamma_n = \left(\frac{n}{\log n}\right)^{-\frac{d}{2q+2d}}$ under regularity conditions, approaching the minimax optimal rate for the given function class.
• Empirical results show significant improvement in prediction accuracy on a 3D hand surface with Raynaud’s disease data, where standard GP models incorrectly smooth across fingers due to Euclidean distance assumptions.
• The use of graph Laplacian eigenpairs provides a computationally feasible approximation to the intractable heat kernel, enabling geometry-aware smoothing.
• The Nyström-type extension allows efficient prediction on new inputs by leveraging the precomputed eigen-decomposition of the graph Laplacian.
• Theoretical analysis confirms that the GL-GP prior contracts to the true function at a rate that depends on the smoothness of the true function and the dimensionality of the input domain.
• The method is robust to unknown domain geometry and performs well even when the input domain is not a known manifold, extending beyond prior assumptions in the literature.