[Paper Review] Exploiting gradients and Hessians in Bayesian optimization and Bayesian quadrature

Anqi Wu, Mikio Aoi+1 more|arXiv (Cornell University)|Mar 31, 2017

Gaussian Processes and Bayesian Inference10 references22 citations

TL;DR

This paper proposes a novel framework for Bayesian optimization (BO) and Bayesian quadrature (BQ) that leverages function values, gradients, and Hessians within Gaussian process (GP) models to accelerate convergence and reduce uncertainty. By introducing input rescaling and spectral representation to address ill-conditioning from high-order derivatives, the method achieves faster hyperparameter learning and superior performance over standard GP-based BO and BQ, with empirical gains in both synthetic and real-world problems including marginal likelihood optimization.

ABSTRACT

An exciting branch of machine learning research focuses on methods for learning, optimizing, and integrating unknown functions that are difficult or costly to evaluate. A popular Bayesian approach to this problem uses a Gaussian process (GP) to construct a posterior distribution over the function of interest given a set of observed measurements, and selects new points to evaluate using the statistics of this posterior. Here we extend these methods to exploit derivative information from the unknown function. We describe methods for Bayesian optimization (BO) and Bayesian quadrature (BQ) in settings where first and second derivatives may be evaluated along with the function itself. We perform sampling-based inference in order to incorporate uncertainty over hyperparameters, and show that both hyperparameter and function uncertainty decrease much more rapidly when using derivative information. Moreover, we introduce techniques for overcoming ill-conditioning issues that have plagued earlier methods for gradient-enhanced Gaussian processes and kriging. We illustrate the efficacy of these methods using applications to real and simulated Bayesian optimization and quadrature problems, and show that exploting derivatives can provide substantial gains over standard methods.

Motivation & Objective

• To develop a scalable and numerically stable method for incorporating first and second derivatives into GP-based Bayesian optimization and quadrature.
• To address the long-standing issue of ill-conditioning in derivative-enhanced GPs, especially when Hessian information is included.
• To improve uncertainty quantification and convergence speed in BO and BQ by exploiting derivative information.
• To enable practical use of higher-order derivatives in Bayesian inference for expensive-to-evaluate functions.
• To demonstrate the method’s effectiveness on both synthetic and real-world problems, including marginal likelihood optimization.

Proposed method

• Introduces a spectral representation of the GP kernel that allows arbitrary accuracy in approximating the exact posterior distribution, even with derivative observations.
• Applies input rescaling to improve numerical conditioning of the covariance matrix when including gradients and Hessians.
• Uses sampling-based inference over hyperparameters to propagate uncertainty, enabling robust posterior updates with derivative data.
• Employs two active sampling strategies in BQ: minimizing posterior variance of the integral and maximizing uncertainty in the integrand-weighted function.
• Derives closed-form expressions for posterior mean and variance in the spectral domain, enabling efficient computation of uncertainty for active learning.
• Implements a dual-space derivation of derivative-enhanced GPs, generalizing prior work that relied on restrictive assumptions like monotonicity.

Experimental results

Research questions

• RQ1 Can incorporating gradients and Hessians into GP-based Bayesian optimization significantly improve convergence speed and uncertainty reduction compared to standard methods?
• RQ2 How can ill-conditioning in derivative-enhanced GPs—especially with Hessian observations—be effectively mitigated for practical use?
• RQ3 Does the use of derivative information lead to faster and more accurate marginal likelihood estimation in GP models?
• RQ4 Can spectral representation and input rescaling jointly provide a stable and accurate alternative to exact GP inference with derivatives?
• RQ5 How do different active sampling strategies in Bayesian quadrature perform when derivative information is available?

Key findings

• The inclusion of gradients and Hessians leads to significantly faster shrinkage of function uncertainty and hyperparameter uncertainty compared to standard GP-based methods.
• The proposed spectral representation of the GP kernel achieves arbitrarily high accuracy in approximating the exact posterior, even with high-order derivative data.
• Input rescaling effectively mitigates ill-conditioning issues that have historically prevented practical use of Hessian information in GP models.
• In Bayesian quadrature, the derivative-enhanced method (GPBQhess) converges to the true integral value faster than standard GPBQ, with lower average error.
• Active sampling using the uncertainty in the integrand-weighted function (Eq. 28) provides substantial computational savings while maintaining strong convergence.
• The method demonstrates superior performance in both unimodal and multimodal optimization tasks, as well as in real-world marginal likelihood optimization, outperforming baseline approaches in convergence speed and accuracy.