[Paper Review] Accelerating ABC methods using Gaussian processes

R. Wilkinson|arXiv (Cornell University)|Jan 7, 2014

Gaussian Processes and Bayesian Inference24 references66 citations

TL;DR

This paper proposes Gaussian process (GP)-accelerated approximate Bayesian computation (ABC) to reduce simulator evaluations by modeling the log-likelihood function with GPs, exploiting smoothness and continuity. It achieves up to 100× fewer simulations than standard ABC and enables the first approximation of the exact posterior in a population genetics model.

ABSTRACT

Approximate Bayesian computation (ABC) methods are used to approximate posterior distributions using simulation rather than likelihood calculations. We introduce Gaussian process (GP) accelerated ABC, which we show can significantly reduce the number of simulations required. As computational resource is usually the main determinant of accuracy in ABC, GP-accelerated methods can thus enable more accurate inference in some models. GP models of the unknown log-likelihood function are used to exploit continuity and smoothness, reducing the required computation. We use a sequence of models that increase in accuracy, using intermediate models to rule out regions of the parameter space as implausible. The methods will not be suitable for all problems, but when they can be used, can result in significant computational savings. For the Ricker model, we are able to achieve accurate approximations to the posterior distribution using a factor of 100 fewer simulator evaluations than comparable Monte Carlo approaches, and for a population genetics model we are able to approximate the exact posterior for the first time.

Motivation & Objective

• Address the high computational cost of ABC methods, especially when simulators are expensive and high accuracy (low tolerance) is required.
• Overcome the inefficiency of standard ABC algorithms that rely on random sampling and ignore known smoothness in the likelihood function.
• Develop a method that uses sequential, adaptive design and GP models to efficiently explore parameter space and reduce the number of simulator runs.
• Enable accurate posterior approximation in models where traditional ABC is computationally infeasible, such as complex simulators with high-dimensional outputs.

Proposed method

• Use Gaussian processes to model the unknown log-likelihood function, treating simulator outputs as noisy observations of the true likelihood.
• Apply a sequence of increasingly accurate GP models across multiple waves, using space-filling designs to select new evaluation points.
• Leverage the continuity and smoothness of the log-likelihood to predict plausible regions and rule out implausible parameter space areas early.
• Use a generalized ABC (GABC) framework with a smooth acceptance kernel to reduce variance in likelihood estimates and improve GP modeling.
• Apply sparse GP approximations to reduce training cost from O(N³) to O(M²N), where M ≪ N, enabling scalability to larger datasets.
• Use diagnostic checks and user supervision at each wave to ensure model reliability and avoid errors from poor GP fit or design choices.

Experimental results

Research questions

• RQ1 Can Gaussian processes be used to model the log-likelihood function in ABC to reduce the number of simulator evaluations?
• RQ2 How does the sequential, adaptive design of GP models compare to random sampling in ABC in terms of computational efficiency and accuracy?
• RQ3 Can GP-accelerated ABC approximate the exact posterior distribution in models where standard ABC fails due to computational cost?
• RQ4 What is the impact of smoothness in the likelihood function on the performance of GP-accelerated ABC?
• RQ5 How can computational costs of GP training be reduced without sacrificing accuracy in ABC inference?

Key findings

• For the Ricker model, GP-accelerated ABC achieved accurate posterior approximations using a factor of 100 fewer simulator evaluations than standard Monte Carlo ABC.
• In a population genetics model, the method enabled the first approximation of the exact posterior distribution, which was previously infeasible with standard ABC due to computational limits.
• The GP-ABC posterior was found to be more accurate than rejection ABC with local linear adjustment, as it better captures the trend toward the exact posterior as tolerance ε decreases.
• The method relies on smoothness in the log-likelihood; it is not universally applicable but provides substantial savings when the likelihood surface is sufficiently smooth.
• Sparse GP approximations reduced training cost from O(N³) to O(M²N), making the method scalable to larger problems while maintaining accuracy.
• Despite introducing an additional layer of approximation, the method enables higher accuracy in ABC by allowing smaller tolerance values than feasible with standard approaches.