[Paper Review] MMD-Bayes: Robust Bayesian Estimation via Maximum Mean Discrepancy

Badr-Eddine Chérief-Abdellatif, Pierre Alquier|arXiv (Cornell University)|Sep 29, 2019

Statistical Methods and Bayesian Inference22 citations

TL;DR

This paper proposes MMD-Bayes, a robust Bayesian estimation framework that replaces the likelihood with a pseudo-likelihood based on Maximum Mean Discrepancy (MMD) to ensure consistency and robustness under model misspecification. By embedding probability measures into a reproducing kernel Hilbert space and using a temperature-parameterized posterior, the method achieves minimax optimal convergence rates and maintains robustness even with contaminated data, while variational inference enables tractable computation via stochastic gradients.

ABSTRACT

In some misspecified settings, the posterior distribution in Bayesian statistics may lead to inconsistent estimates. To fix this issue, it has been suggested to replace the likelihood by a pseudo-likelihood, that is the exponential of a loss function enjoying suitable robustness properties. In this paper, we build a pseudo-likelihood based on the Maximum Mean Discrepancy, defined via an embedding of probability distributions into a reproducing kernel Hilbert space. We show that this MMD-Bayes posterior is consistent and robust to model misspecification. As the posterior obtained in this way might be intractable, we also prove that reasonable variational approximations of this posterior enjoy the same properties. We provide details on a stochastic gradient algorithm to compute these variational approximations. Numerical simulations indeed suggest that our estimator is more robust to misspecification than the ones based on the likelihood.

Motivation & Objective

• To address the inconsistency of standard Bayesian posteriors under model misspecification.
• To develop a robust alternative to the likelihood function in Bayesian inference using the Maximum Mean Discrepancy (MMD) metric.
• To ensure the resulting posterior maintains consistency and minimax optimal convergence rates even when the true data-generating distribution is outside the model.
• To provide a tractable inference method via variational approximation with theoretical guarantees.
• To demonstrate empirical robustness against outliers and data contamination through numerical simulations.

Proposed method

• Define a pseudo-posterior using the squared MMD distance between the model distribution and the empirical measure, weighted by a temperature parameter β.
• Construct the MMD-Bayes posterior as πβn ∝ exp(−β·D²k(Pθ, ˆPn)) π(dθ), where Dk is the MMD in a reproducing kernel Hilbert space.
• Introduce a robust prior mass condition based on the MMD distance to the closest model distribution θ∗, ensuring posterior concentration.
• Use variational inference with a Gaussian family to approximate the intractable MMD-Bayes posterior.
• Implement a projected stochastic gradient algorithm (PSGAVI) to optimize the variational parameters using stochastic gradients derived via reparameterization and log-derivative tricks.
• Derive stochastic gradient estimators for both mean and covariance parameters using Monte Carlo samples from the variational distribution.

Experimental results

Research questions

• RQ1 Can a Bayesian posterior based on MMD as a divergence measure achieve consistency and robustness under model misspecification?
• RQ2 Does the MMD-Bayes posterior converge at the minimax optimal rate n−1/2 under well-specified and misspecified models?
• RQ3 Can variational approximations of the MMD-Bayes posterior inherit the same theoretical guarantees as the exact posterior?
• RQ4 How does the MMD-Bayes estimator compare to classical estimators like MLE and median in the presence of outliers?
• RQ5 Can a stochastic gradient algorithm efficiently compute the variational approximation of the MMD-Bayes posterior?

Key findings

• The MMD-Bayes posterior achieves minimax optimal convergence rate n−1/2 under the robust prior mass condition, even when the true distribution is not in the model.
• Theoretical analysis shows that the posterior expectation of D²k(Pθ, P0) is bounded by 8 infθ D²k(Pθ, P0) + 16/n, confirming consistency and robustness.
• The posterior concentrates around the closest model distribution θ∗ at rate n−1/2 in probability, even when P0 ∉ {Pθ : θ ∈ Θ}.
• Variational approximations of the MMD-Bayes posterior inherit the same consistency and robustness properties as the exact posterior.
• Numerical simulations demonstrate that the MMD-Bayes estimator outperforms MLE and median estimators in terms of MSE under increasing contamination, especially in uniform location and Gaussian mean estimation with outliers.
• The method remains stable and accurate even with up to 20% of contaminated data, while MLE and method of moments estimators degrade significantly under the same conditions.