[Paper Review] Statistical Inference for Generative Models with Maximum Mean Discrepancy

François‐Xavier Briol, Alessandro Barp+2 more|arXiv (Cornell University)|Jun 13, 2019

Markov Chains and Monte Carlo Methods106 references35 citations

TL;DR

The paper develops minimum MMD estimators for intractable generative models, analyzes their consistency, asymptotic normality, and robustness, and introduces a natural-gradient-like optimization method for efficient inference.

ABSTRACT

While likelihood-based inference and its variants provide a statistically efficient and widely applicable approach to parametric inference, their application to models involving intractable likelihoods poses challenges. In this work, we study a class of minimum distance estimators for intractable generative models, that is, statistical models for which the likelihood is intractable, but simulation is cheap. The distance considered, maximum mean discrepancy (MMD), is defined through the embedding of probability measures into a reproducing kernel Hilbert space. We study the theoretical properties of these estimators, showing that they are consistent, asymptotically normal and robust to model misspecification. A main advantage of these estimators is the flexibility offered by the choice of kernel, which can be used to trade-off statistical efficiency and robustness. On the algorithmic side, we study the geometry induced by MMD on the parameter space and use this to introduce a novel natural gradient descent-like algorithm for efficient implementation of these estimators. We illustrate the relevance of our theoretical results on several classes of models including a discrete-time latent Markov process and two multivariate stochastic differential equation models.

Motivation & Objective

• Motivate inference for intractable generative models where likelihoods are unavailable or expensive to compute.
• Propose a minimum distance framework using Maximum Mean Discrepancy (MMD) to compare model and data distributions.
• Analyze statistical properties of minimum MMD estimators, including consistency, asymptotic normality, and robustness.
• Investigate how kernel choice and geometry affect generalisation performance and efficiency.
• Develop an efficient optimization algorithm based on information-geometric ideas (natural gradient) for MMD-based inference.

Proposed method

• Define MMD between model and data distributions via kernel mean embeddings in a reproducing kernel Hilbert space.
• Formulate minimum MMD estimators by minimizing MMD^2 between P_theta and the empirical data distribution Q^m.
• Derive a U-statistic gradient estimator for SGD to update theta without computing intractable integrals.
• Introduce a stochastic natural gradient descent algorithm using a kernel-induced Riemannian metric on the parameter space.
• Discuss a fully implicit discretization that links gradient flow to proximal-like updates for robustness.
• Relate minimum MMD estimators to kernel scoring rules and their corresponding divergences.

Experimental results

Research questions

• RQ1 How can one perform statistical inference for intractable generative models using a distance-based criterion instead of likelihood?
• RQ2 What are the theoretical properties (consistency, asymptotic normality, robustness) of minimum MMD estimators under M-closed and M-open settings?
• RQ3 How does the choice of kernel influence generalisation bounds, efficiency, and robustness of the estimators?
• RQ4 Can a natural gradient or information-geometric approach yield computationally efficient optimization for MMD-based inference?
• RQ5 What are practical implications and performance of minimum MMD estimators on models such as latent Markov processes and stochastic differential equations?

Key findings

• Minimum MMD estimators are consistent and asymptotically normal in the M-closed setting under suitable assumptions.
• Estimator robustness is established in the M-open setting, with qualitative and quantitative robustness results.
• Generalisation bounds for MMD estimators are dimension-robust, with rates of order m^{-1/2} (and n^{-1/2} in the n,m setting) and explicit kernel-dependent constants.
• The kernel choice, including Gaussian kernels and kernel mixtures, trades off efficiency and robustness, and median-lengthscale heuristics can mitigate dimensionality effects.
• A stochastic natural gradient descent algorithm, based on the information geometry of MMD, offers computational gains over standard SGD for these estimators.
• Applications to discrete-time latent Markov processes and multivariate SDEs illustrate the practical relevance of the theory.