[Paper Review] Bayesian quadrature and energy minimization for space-filling design

Luc Pronzato, Anatoly Zhigljavsky|arXiv (Cornell University)|Aug 31, 2018

Probabilistic and Robust Engineering Design18 references3 citations

TL;DR

This paper establishes a theoretical connection between Bayesian quadrature, optimal experimental design, and energy minimization for space-filling design by showing that integrally strictly positive definite kernels induce strictly convex energy functionals, enabling equivalence between discrepancy minimization and BLUE construction. It demonstrates that minimizing posterior variance in Bayesian integration—equivalent to kernel discrepancy minimization—provides an efficient, computationally tractable alternative to IMSPE for space-filling design.

ABSTRACT

A standard objective in computer experiments is to approximate the behaviour of an unknown function on a compact domain from a few evaluations inside the domain. When little is known about the function, space-filling design is advisable: typically, points of evaluation spread out across the available space are obtained by minimizing a geometrical (for instance, covering radius) or a discrepancy criterion measuring distance to uniformity. The paper investigates connections between design for integration (quadrature design), construction of the (continuous) BLUE for the location model, space-filling design, and minimization of energy (kernel discrepancy) for signed measures. Integrally strictly positive definite kernels define strictly convex energy functionals, with an equivalence between the notions of potential and directional derivative, showing the strong relation between discrepancy minimization and more traditional design of optimal experiments. In particular, kernel herding algorithms, which are special instances of vertex-direction methods used in optimal design, can be applied to the construction of point sequences with suitable space-filling properties.

Motivation & Objective

• To unify space-filling design, Bayesian quadrature, and optimal experimental design under a common framework based on kernel-based energy functionals.
• To show that minimizing posterior variance in Bayesian integration corresponds to minimizing kernel discrepancy and energy functionals, offering a practical alternative to IMSPE minimization.
• To demonstrate that kernel herding algorithms—special cases of vertex-direction methods—can generate high-quality space-filling point sequences.
• To establish theoretical equivalence between the construction of the continuous BLUE in a location model and optimal design for integration.
• To provide a principled, computationally efficient approach to designing point sets with strong space-filling properties for expensive simulation-based inference.

Proposed method

• Uses integrally strictly positive definite kernels to define strictly convex energy functionals, linking potential and directional derivatives.
• Shows that the posterior variance in Bayesian quadrature equals the minimum of a squared kernel discrepancy for signed measures with unit total mass.
• Establishes equivalence between Bayesian integration and the construction of the continuous Best Linear Unbiased Estimator (BLUE) under a modified correlation structure.
• Applies kernel herding algorithms—derived from vertex-direction methods in optimal design—to generate sequences with space-filling properties.
• Employs Karhunen-Loève expansions to reformulate the Gaussian process model as a Bayesian linear model, linking posterior variance minimization to c-optimal design.
• Utilizes approximate design theory and vertex-exchange algorithms to compute optimal design measures, which are then extracted into exact designs.

Experimental results

Research questions

• RQ1 How are Bayesian quadrature, energy minimization, and space-filling design connected through kernel-based functionals?
• RQ2 Can minimizing posterior variance in Bayesian integration serve as a practical alternative to minimizing IMSPE for space-filling design?
• RQ3 What is the relationship between kernel discrepancy minimization and the construction of the continuous BLUE in a location model with correlated errors?
• RQ4 How do kernel herding algorithms relate to vertex-direction methods in optimal design and what space-filling properties do they induce?
• RQ5 To what extent can the Karhunen-Loève expansion of a Gaussian process model be used to reduce Bayesian integration to a c-optimal design problem?

Key findings

• Integrally strictly positive definite kernels define strictly convex energy functionals, with equivalence between potential and directional derivative, revealing deep connections between discrepancy and optimal design.
• The posterior variance in Bayesian quadrature is mathematically equivalent to the minimum of a squared kernel discrepancy for signed measures with total mass one.
• Minimizing posterior variance in Bayesian integration is equivalent to minimizing an energy functional for a reduced kernel, providing a computationally tractable alternative to IMSPE minimization.
• Kernel herding algorithms, as special cases of vertex-direction methods, generate point sequences with strong space-filling properties and can be used as anytime designs.
• The optimal design measure ξ* minimizing {M_B,m⁻¹(ξ)}₁,₁ can be computed via convex optimization and vertex-exchange algorithms, with support points suitable for exact design extraction.
• In the tensor-product kernel case, eigenfunctions and eigenvalues can be approximated via one-dimensional quadrature, enabling efficient numerical construction of optimal designs and Sobol’ index estimation.