[Paper Review] Variational Fourier features for Gaussian processes

James Hensman, Nicolas Durrande+1 more|arXiv (Cornell University)|Nov 21, 2016

Gaussian Processes and Bayesian Inference71 references97 citations

TL;DR

This paper introduces Variational Fourier Features (VFF), a novel Gaussian process approximation that combines variational inference with spectral representations using regularly spaced Fourier frequencies. By projecting the GP onto a finite domain with a Reproducing Kernel Hilbert Space (RKHS) framework, VFF achieves O(NM) computational complexity—significantly faster than O(NM²) in standard sparse GP methods—while maintaining high accuracy and scalability, enabling inference on 4 million data points in minutes on a standard laptop.

ABSTRACT

This work brings together two powerful concepts in Gaussian processes: the variational approach to sparse approximation and the spectral representation of Gaussian processes. This gives rise to an approximation that inherits the benefits of the variational approach but with the representational power and computational scalability of spectral representations. The work hinges on a key result that there exist spectral features related to a finite domain of the Gaussian process which exhibit almost-independent covariances. We derive these expressions for Matern kernels in one dimension, and generalize to more dimensions using kernels with specific structures. Under the assumption of additive Gaussian noise, our method requires only a single pass through the dataset, making for very fast and accurate computation. We fit a model to 4 million training points in just a few minutes on a standard laptop. With non-conjugate likelihoods, our MCMC scheme reduces the cost of computation from O(NM2) (for a sparse Gaussian process) to O(NM) per iteration, where N is the number of data and M is the number of features.

Motivation & Objective

• Address the computational bottleneck in Gaussian process models, which scale cubically with data size N due to dense covariance matrix operations.
• Overcome limitations of existing sparse GP methods that rely on inducing points and suffer from high computational cost in non-conjugate likelihood settings.
• Develop a scalable, accurate GP approximation that leverages the representational power of spectral methods while maintaining the theoretical rigor of variational inference.
• Enable efficient inference for large-scale datasets (e.g., 4M points) with both conjugate and non-conjugate likelihoods using a single-pass algorithm.
• Construct Fourier features with almost-independent covariance structure to enable fast, scalable computation without sacrificing approximation quality.

Proposed method

• Use a finite domain windowing technique combined with RKHS projection to create valid, finite-variance inducing features from the spectral representation of Matérn kernels.
• Construct Fourier features using a regular grid of frequencies (harmonic basis functions) to ensure a structured, decomposable covariance matrix.
• Derive closed-form expressions for the Gram matrix between Fourier features in one dimension for Matérn-1/2, 3/2, and 5/2 kernels using RKHS inner products.
• Apply variational inference by minimizing the KL divergence between a variational posterior and the true GP posterior, using the Fourier features as a basis.
• Generalize the method to multidimensional inputs by exploiting additive and product kernel structures, preserving computational efficiency.
• Design an MCMC scheme that reduces the cost per iteration from O(NM²) to O(NM) for non-conjugate likelihoods, enabling scalable posterior inference.

Experimental results

Research questions

• RQ1 How can we construct a set of inducing features for Gaussian processes that are both computationally efficient and statistically valid, avoiding the infinite variance problem of standard Fourier features?
• RQ2 What is the structure of the covariance matrix between Fourier features derived from a finite domain and RKHS projection, and can it be decomposed efficiently?
• RQ3 Can we achieve O(NM) computational complexity in variational Gaussian process inference by using structured Fourier features instead of inducing points?
• RQ4 How does the proposed method scale to large datasets with non-conjugate likelihoods, and what is the computational cost per MCMC iteration?
• RQ5 Can the combination of spectral representation and variational inference yield a method that is both accurate and fast enough for real-world applications on massive datasets?

Key findings

• The proposed Variational Fourier Features (VFF) achieve O(NM) computational complexity per iteration, a significant improvement over the O(NM²) complexity of standard sparse GP methods.
• Using a regular grid of frequencies and RKHS projection, the method constructs Fourier features with an almost-independent covariance structure that enables fast matrix operations.
• VFF enables inference on 4 million training points in just a few minutes on a standard laptop, demonstrating exceptional scalability.
• For non-conjugate likelihoods, the MCMC scheme reduces the cost per iteration from O(NM²) to O(NM), making large-scale posterior inference tractable.
• The method achieves high accuracy with minimal approximation error, as shown in toy examples and empirical experiments, particularly for Matérn-3/2 and Matérn-5/2 kernels.
• Closed-form expressions for the Gram matrix between Fourier features are derived for all Matérn kernels, enabling exact and efficient computation of the variational objective.