[Paper Review] Exact ABC using Importance Sampling

Minh Ngoc Tran, Robert Kohn|arXiv (Cornell University)|Jan 1, 2015

Markov Chains and Monte Carlo Methods8 references4 citations

TL;DR

This paper proposes Exact ABC using Importance Sampling (EABC-IS2), a method that eliminates the systematic error inherent in standard Approximate Bayesian Computation (ABC) by constructing an unbiased estimator of the likelihood through debiasing techniques. By leveraging importance sampling with unbiased likelihood estimates, the method achieves almost sure convergence to the true posterior expectation without requiring a tolerance level, as demonstrated in Gaussian and Ising model examples with consistent convergence to ground-truth values as sample size increases.

ABSTRACT

Approximate Bayesian Computation (ABC) is a powerful method for carrying out Bayesian inference when the likelihood is computationally intractable. However, a drawback of ABC is that it is an approximate method that induces a systematic error because it is necessary to set a tolerance level to make the computation tractable. The issue of how to optimally set this tolerance level has been the subject of extensive research. This paper proposes an ABC algorithm based on importance sampling that estimates expectations with respect to the "exact" posterior distribution given the observed summary statistics. This overcomes the need to select the tolerance level. By "exact" we mean that there is no systematic error and the Monte Carlo error can be made arbitrarily small by increasing the number of importance samples. We provide a formal justification for the method and study its convergence properties. The method is illustrated in two applications and the empirical results suggest that the proposed ABC based estimators consistently converge to the true values as the number of importance samples increases. Our proposed approach can be applied more generally to any importance sampling problem where an unbiased estimate of the likelihood is required.

Motivation & Objective

• To address the systematic error introduced by the tolerance level ǫ > 0 in standard ABC methods.
• To develop a method that estimates expectations with respect to the exact posterior distribution p(θ|sobs) without relying on approximate likelihoods.
• To provide a formal justification and convergence analysis for an importance sampling estimator based on unbiased likelihood estimation.
• To demonstrate empirically that the proposed estimator converges to the true posterior mean as the number of importance samples increases.
• To generalize the approach to any importance sampling problem requiring an unbiased likelihood estimate.

Proposed method

• Uses the debiasing framework of Rhee and Glynn (2013) and McLeish (2012) to construct an unbiased estimator of the likelihood p(sobs|θ) by randomizing over a sequence of decreasing tolerance levels ǫk.
• Employs a sequence of pseudo-datasets s_i^k ∼ p(·|θ) to estimate the ABC likelihood pABC,ǫk(sobs|θ) via kernel density estimation.
• Applies importance sampling to estimate posterior expectations E[ϕ(θ)|sobs] using the unbiased likelihood estimator, ensuring the estimator converges almost surely to the true value.
• Introduces a random stopping time T with P(T ≥ k) > 0 to construct the unbiased estimator bλ = ζ₀ + Σ_{k=1}^T ̟k(ζk − ζk−1), where ̟k = 1/P(T ≥ k).
• Derives theoretical conditions under which the variance of the unbiased estimator is finite, ensuring convergence properties.
• Uses a proposal distribution gIS(θ) and weights bw(θ, u) to reweight samples in the importance sampling framework, ensuring consistency and asymptotic normality.

Experimental results

Research questions

• RQ1 Can an unbiased estimator of the intractable likelihood be constructed in ABC using randomization and debiasing techniques?
• RQ2 Does the proposed EABC-IS2 method eliminate the systematic error due to tolerance ǫ > 0 in ABC inference?
• RQ3 Can the importance sampling estimator based on unbiased likelihoods achieve almost sure convergence to the true posterior expectation?
• RQ4 How does the performance of EABC-IS2 compare to standard ABC and exact MCMC in terms of accuracy and convergence as sample size increases?
• RQ5 Under what conditions is the variance of the unbiased likelihood estimator finite, ensuring the convergence of the importance sampling estimator?

Key findings

• The EABC-IS2 estimator for the posterior mean of θ in the Ising model converged to 0.5099 (0.0001) with M = 200,000 samples, matching the exact MCMC estimate of 0.5099 obtained from 1,000,000 iterations.
• As the number of importance samples M increased, the EABC-IS2 estimates consistently approached the true posterior mean, demonstrating almost sure convergence without systematic bias.
• For the Ising model, ABC-PMMH with ǫ = 0.1, 1, and 10 produced biased estimates of 0.5100, 0.5108, and 0.5094 respectively, showing persistent systematic error even with large M.
• The theoretical analysis proves that the variance of the unbiased likelihood estimator is finite under regularity conditions on the kernel and likelihood, ensuring the convergence of the importance sampling estimator.
• The method achieves asymptotic normality of the estimator, with √M( Êπ(ϕ) − Eπ(ϕ) ) converging in distribution to N(0, σ²ϕ), confirming valid inference.
• The approach generalizes to any importance sampling problem where an unbiased likelihood estimate is required, extending beyond ABC to broader Bayesian computation contexts.