[Paper Review] Approximate Bayesian Computation via Classification

Yuexi Wang, Tetsuya Kaji+1 more|arXiv (Cornell University)|Nov 22, 2021

Markov Chains and Monte Carlo Methods52 references4 citations

TL;DR

This paper proposes a novel Approximate Bayesian Computation (ABC) framework that replaces traditional summary statistics with a classification-based Kullback-Leibler (KL) divergence estimator, enabling likelihood-free inference in simulator-based models. By training a classifier to distinguish real from simulated data, the method estimates the KL divergence and uses it as a discrepancy measure, achieving posterior concentration rates dependent on classifier estimation error and enabling asymptotic normality via an exponential kernel, outperforming standard ABC with summary statistics in simulation and real-world volatility estimation.

ABSTRACT

Approximate Bayesian Computation (ABC) enables statistical inference in simulator-based models whose likelihoods are difficult to calculate but easy to simulate from. ABC constructs a kernel-type approximation to the posterior distribution through an accept/reject mechanism which compares summary statistics of real and simulated data. To obviate the need for summary statistics, we directly compare empirical distributions with a Kullback-Leibler (KL) divergence estimator obtained via contrastive learning. In particular, we blend flexible machine learning classifiers within ABC to automate fake/real data comparisons. We consider the traditional accept/reject kernel as well as an exponential weighting scheme which does not require the ABC acceptance threshold. Our theoretical results show that the rate at which our ABC posterior distributions concentrate around the true parameter depends on the estimation error of the classifier. We derive limiting posterior shape results and find that, with a properly scaled exponential kernel, asymptotic normality holds. We demonstrate the usefulness of our approach on simulated examples as well as real data in the context of stock volatility estimation.

Motivation & Objective

• To address the challenge of selecting effective summary statistics in Approximate Bayesian Computation (ABC), which is critical for accurate inference in simulator-based models with intractable likelihoods.
• To develop a likelihood-free inference framework that bypasses manual summary statistic selection by directly comparing empirical distributions using a classifier-based KL divergence estimator.
• To improve posterior concentration and inference efficiency by replacing the ad hoc ABC acceptance threshold with a smooth exponential kernel based on classifier confidence.
• To provide theoretical justification for the method’s performance, showing posterior concentration rates depend on classifier estimation error and posterior contraction.
• To demonstrate the method’s practical utility through benchmark simulations and real-world application to stock volatility estimation using historical price data.

Proposed method

• The method uses a binary classifier to distinguish between real and simulated data, with the classifier’s output probability serving as an estimate of the KL divergence between the empirical distributions of real and simulated data.
• The classifier is trained on paired real and simulated datasets to learn a decision boundary that reflects statistical similarity, enabling a non-parametric, flexible discrepancy measure without summary statistics.
• The ABC posterior is constructed using a kernel-based acceptance mechanism: either an accept/reject kernel with a threshold or an exponential kernel that weights samples by the classifier’s confidence (i.e., the probability of being real data).
• The exponential kernel is defined as $ w( heta) = ext{exp}(- au imes ext{KL}_{ ext{est}}( heta)) $, where $ au $ is a scaling parameter, and $ ext{KL}_{ ext{est}} $ is the classifier-based KL estimator.
• Theoretical analysis shows that posterior concentration depends on the classifier’s estimation error $ ilde{ ho}_n $, and under proper scaling, the posterior achieves asymptotic normality.
• The method is connected to Generalized Bayesian Inference (GBI) and the $ c $-posterior framework, with the KL estimator acting as a loss function in a robust Bayesian update.

Experimental results

Research questions

• RQ1 Can a classifier trained to distinguish real from simulated data provide a reliable, summary-statistic-free discrepancy measure for ABC?
• RQ2 How does the performance of ABC based on a classifier-based KL estimator compare to traditional ABC with summary statistics in terms of posterior concentration and coverage?
• RQ3 What is the theoretical rate of posterior concentration for the proposed classification-based ABC method, and how does it depend on the classifier’s estimation error?
• RQ4 Can an exponential kernel based on classifier confidence eliminate the need for an arbitrary ABC acceptance threshold $ ilde{ ho}_n $, and does it yield better frequentist properties?
• RQ5 Does the proposed method achieve asymptotic normality of the posterior, and under what conditions on the classifier and model?

Key findings

• The proposed classification-based ABC method achieves tighter posterior distributions than ABC with summary statistics, as shown in the stock volatility estimation example, where posterior intervals were significantly narrower.
• The posterior concentration rate of the accept/reject ABC depends on both the ABC threshold $ ilde{ ho}_n $ and the classifier estimation error $ ilde{ ho}_n $, with faster rates when the classifier is more accurate.
• The exponential kernel version of ABC achieves asymptotic normality under proper scaling, with the posterior concentrating around the true parameter at a rate determined by the classifier’s estimation error and the actual posterior contraction rate.
• In the real data analysis of BA and PG stock prices, the method with exponential kernels produced posterior estimates of volatility ($ ar{oldsymbol{ heta}} $) that were consistent across methods, while the ABC with summary statistics showed wider and less precise intervals.
• The method is robust to model misspecification, as the accept/reject variant inherits robustness from the $ c $-posterior framework, and the exponential kernel is formally linked to Generalized Bayesian Inference with KL divergence as the loss.
• Theoretical results justify the method of Gutmann et al. (2018) using classification accuracy in ABC, showing that the classifier’s ability to separate real and fake data directly informs posterior inference quality.