[Paper Review] Neural Approximate Sufficient Statistics for Implicit Models

Yanzhi Chen, Dinghuai Zhang+3 more|ePrints Soton (University of Southampton)|Oct 20, 2020

Markov Chains and Monte Carlo Methods59 references21 citations

TL;DR

This paper proposes a deep learning approach to automatically learn near-sufficient statistics for likelihood-free inference in implicit generative models, using mutual information maximization via neural networks without density estimation. The method improves performance in both traditional ABC and modern neural likelihood methods by learning compact, informative representations that boost accuracy and robustness across diverse data types.

ABSTRACT

We consider the fundamental problem of how to automatically construct summary statistics for implicit generative models where the evaluation of the likelihood function is intractable, but sampling data from the model is possible. The idea is to frame the task of constructing sufficient statistics as learning mutual information maximizing representations of the data with the help of deep neural networks. The infomax learning procedure does not need to estimate any density or density ratio. We apply our approach to both traditional approximate Bayesian computation and recent neural likelihood methods, boosting their performance on a range of tasks.

Motivation & Objective

• To address the challenge of designing low-dimensional, informative summary statistics for likelihood-free inference in implicit models where the likelihood is intractable.
• To eliminate reliance on expert-designed or handcrafted statistics by automating the construction of sufficient statistics using deep learning.
• To improve the performance and robustness of both approximate Bayesian computation (ABC) and neural density estimation (NDE)-based likelihood-free inference methods.
• To establish a principled link between representation learning and likelihood-free inference through information-theoretic principles.

Proposed method

• The method frames sufficient statistic learning as an information-maximization problem, aiming to preserve maximal mutual information between data and parameters.
• It employs distribution-free mutual information estimators or their proxies (e.g., MINE, InfoNCE) to train a neural network to map raw data to compact, informative representations.
• The learned statistics are integrated into both SMC-ABC and SNL frameworks, forming SMC-ABC+ and SNL+, enabling joint learning of posterior and sufficient statistics.
• The approach avoids density or density ratio estimation, relying solely on simulated data and mutual information optimization.
• The architecture is trained end-to-end, allowing iterative refinement between posterior estimation and sufficient statistic learning.
• The method is evaluated on i.i.d., sequential, and graph-structured data, including Gaussian copula and Ornstein-Uhlenbeck processes.

Experimental results

Research questions

• RQ1 Can neural networks learn near-sufficient statistics that outperform handcrafted summary statistics in likelihood-free inference?
• RQ2 Does mutual information maximization via deep learning lead to more robust and accurate posterior approximations in implicit models?
• RQ3 Can the proposed method improve performance across both traditional ABC and modern neural likelihood methods?
• RQ4 How does the performance of the learned statistics compare to expert-designed statistics in high-dimensional or complex data settings?

Key findings

• The proposed neural approximate sufficient statistics significantly improve the accuracy and robustness of both SMC-ABC and SNL, reducing posterior estimation error.
• In the Gaussian copula model, the method achieved better performance than SNL and comparable results to SNPE, with lower variability in posterior estimates.
• For the Ornstein-Uhlenbeck process, the learned statistics matched or outperformed expert-designed summary statistics (mean, std, autocorrelation) in terms of JSD between true and learned posteriors.
• The JSD between true and learned posteriors was reduced to 0.009±0.002 for SNL+ and 0.004±0.001 for SNL+ with expert statistics, indicating high posterior fidelity.
• The method demonstrated consistent improvements across diverse data types, including i.i.d., time-series, and graph-structured data.
• The approach outperformed SRE and SNPE in the OU process experiment, with lower JSD and higher stability.