[Paper Review] A Joint introduction to Gaussian Processes and Relevance Vector Machines with Connections to Kalman filtering and other Kernel Smoothers

Luca Martino, Jesse Read|arXiv (Cornell University)|Sep 19, 2020

Gaussian Processes and Bayesian Inference53 references70 citations

TL;DR

This paper presents a unified probabilistic framework linking Gaussian Processes (GPs) and Relevance Vector Machines (RVMs) through Kernel Ridge Regression, highlighting their connections to Kalman filtering and kernel smoothers. It demonstrates that GP smoothing solutions are mathematically equivalent to Kalman smoothing under specific state-space models, offering a coherent view of regression, filtering, smoothing, and interpolation across Bayesian nonparametric and signal processing methods.

ABSTRACT

The expressive power of Bayesian kernel-based methods has led them to become an important tool across many different facets of artificial intelligence, and useful to a plethora of modern application domains, providing both power and interpretability via uncertainty analysis. This article introduces and discusses two methods which straddle the areas of probabilistic Bayesian schemes and kernel methods for regression: Gaussian Processes and Relevance Vector Machines. Our focus is on developing a common framework with which to view these methods, via intermediate methods a probabilistic version of the well-known kernel ridge regression, and drawing connections among them, via dual formulations, and discussion of their application in the context of major tasks: regression, smoothing, interpolation, and filtering. Overall, we provide understanding of the mathematical concepts behind these models, and we summarize and discuss in depth different interpretations and highlight the relationship to other methods, such as linear kernel smoothers, Kalman filtering and Fourier approximations. Throughout, we provide numerous figures to promote understanding, and we make numerous recommendations to practitioners. Benefits and drawbacks of the different techniques are highlighted. To our knowledge, this is the most in-depth study of its kind to date focused on these two methods, and will be relevant to theoretical understanding and practitioners throughout the domains of data-science, signal processing, machine learning, and artificial intelligence in general.

Motivation & Objective

• To establish a common theoretical and computational framework for Gaussian Processes (GPs) and Relevance Vector Machines (RVMs) in regression.
• To clarify the connections between GP and RVM methods via probabilistic kernel ridge regression (Bayesian Ridge Regression).
• To demonstrate the equivalence between Gaussian Process smoothing and Kalman smoothing under a discrete-time state-space model with a specific covariance structure.
• To unify concepts across machine learning, signal processing, and statistics by linking GPs, RVMs, kernel smoothers, and Kalman filtering.
• To provide practical guidance on uncertainty estimation, kernel selection, and functional prior/posterior sampling in both GP and RVM frameworks.

Proposed method

• Derives the RVM solution via Bayesian linear regression with automatic relevance determination (ARD) priors on coefficients.
• Introduces a probabilistic version of Kernel Ridge Regression (KRR) as a bridge between RVM and GP, referred to as 'quasi-GP'.
• Derives the GP solution using a zero-mean Gaussian process prior with a stationary covariance function ψ(t, t′) = γ^|t−t′|σ²_v / (1 − γ²).
• Establishes a dual formulation showing that RVM solutions can be interpreted as GP posteriors under a specific kernel structure.
• Demonstrates equivalence between GP smoothing and Kalman smoothing by mapping the GP covariance function to a state-space model with a first-order Markov process.
• Uses continuous-time linear stochastic differential equations to derive the spectral density and covariance function of the underlying GP process, linking it to the power spectral density of the state-space model.

Experimental results

Research questions

• RQ1 How can Gaussian Processes and Relevance Vector Machines be unified under a common probabilistic framework?
• RQ2 What is the precise mathematical relationship between GP smoothing and Kalman smoothing in discrete and continuous time?
• RQ3 How does the choice of basis functions in RVM compare to kernel selection in GP in terms of predictive variance and interpretability?
• RQ4 In what way do kernel ridge regression, kernel smoothers, and k-NN schemes relate to GP and RVM methods?
• RQ5 Can the state-space model representation of a GP be derived from a linear SDE, and how does this relate to the spectral density of the process?

Key findings

• The GP smoothing solution is mathematically equivalent to the Kalman smoothing solution when the GP covariance function corresponds to a first-order Markov process with a specific autocorrelation structure.
• The predictive variance in GP models exhibits a well-behaved, monotonic decay with distance from observed points, whereas RVMs can produce counterintuitive variance behavior due to basis function selection.
• RVMs offer greater flexibility in basis function design, enabling domain-specific modeling, while GPs are constrained by the need to define a valid kernel function.
• The power spectral density of a GP process derived from a linear SDE with constant coefficients matches the spectral representation of the Kalman filter's state transition dynamics.
• The backward recursion in Kalman smoothing exactly reproduces the posterior mean and variance of a GP under the same kernel and observation model.
• The paper establishes that the kernel function in a GP can be implicitly induced by the state transition matrix in a state-space model, providing a deep link between Bayesian nonparametric regression and classical filtering theory.