[Paper Review] Bayesian Posterior Perturbation Analysis with Integral Probability Metrics

Alfredo Garbuno-Iñigo, Tapio Helin+2 more|arXiv (Cornell University)|Mar 2, 2023

Gaussian Processes and Bayesian Inference4 citations

TL;DR

This paper introduces a novel framework for analyzing the stability of Bayesian posterior measures under perturbations in the likelihood potential and prior using integral probability metrics (IPMs), particularly tailored for non-linear and locally Lipschitz likelihoods. It establishes convergence rates for posterior approximations via deep neural network surrogates and data-driven priors, overcoming limitations of traditional divergences like total variation or Hellinger that require absolute continuity.

ABSTRACT

In recent years, Bayesian inference in large-scale inverse problems found in science, engineering and machine learning has gained significant attention. This paper examines the robustness of the Bayesian approach by analyzing the stability of posterior measures in relation to perturbations in the likelihood potential and the prior measure. We present new stability results using a family of integral probability metrics (divergences) akin to dual problems that arise in optimal transport. Our results stand out from previous works in three directions: (1) We construct new families of integral probability metrics that are adapted to the problem at hand; (2) These new metrics allow us to study both likelihood and prior perturbations in a convenient way; and (3) our analysis accommodates likelihood potentials that are only locally Lipschitz, making them applicable to a wide range of nonlinear inverse problems. Our theoretical findings are further reinforced through specific and novel examples where the approximation rates of posterior measures are obtained for different types of perturbations and provide a path towards the convergence analysis of recently adapted machine learning techniques for Bayesian inverse problems such as data-driven priors and neural network surrogates.

Motivation & Objective

• To address the lack of robust stability analysis for Bayesian posterior measures under prior perturbations, especially in non-equivalent or singular prior settings.
• To develop a flexible framework using integral probability metrics (IPMs) that accommodate both likelihood and prior perturbations without requiring absolute continuity.
• To extend convergence analysis to modern machine learning techniques such as deep neural network surrogates and data-driven priors in high-dimensional or functional Bayesian inverse problems.
• To provide theoretical guarantees for posterior approximation rates under locally Lipschitz likelihood potentials, which are common in nonlinear inverse problems.
• To overcome the limitations of existing divergences like total variation and Hellinger, which fail when prior measures become singular under perturbation.

Proposed method

• The authors define a family of IPMs adapted to the problem structure, using dual formulations inspired by optimal transport to measure posterior divergence.
• They introduce a general stability framework based on Assumption 1, which bounds the difference between perturbed and true likelihood potentials via a reference function h(u;y) and a distance-like function c.
• The method leverages holomorphic extension properties of likelihood potentials to derive exponential convergence rates for deep neural network approximations.
• It applies Theorem 3.2 to bound the IPM between the true posterior ν and the approximate posterior ν^N in terms of the approximation error of the likelihood potential.
• The framework handles non-equivalent priors by using IPMs such as the Wasserstein metric, which do not require absolute continuity between measures.
• The approach is validated through a concrete example involving ReLU deep neural networks approximating holomorphic likelihoods, yielding exponential convergence rates in network size.

Experimental results

Research questions

• RQ1 Can we establish convergence rates for Bayesian posterior measures under perturbations of the likelihood potential when the potential is only locally Lipschitz or non-smooth?
• RQ2 How can we analyze posterior stability under prior perturbations when the prior and its perturbation are mutually singular, such as in data-driven or pushforward priors?
• RQ3 Can we extend convergence analysis to deep neural network surrogates of the likelihood in Bayesian inverse problems, especially when the true likelihood is holomorphic?
• RQ4 What IPM-based divergences allow for stable posterior analysis without requiring absolute continuity between prior and perturbed prior?
• RQ5 How do we quantify the robustness of Bayesian inference when both the likelihood and prior are approximated, as in modern machine learning applications?

Key findings

• The paper establishes an exponential convergence rate of the form $ \mathcal{D}(\nu, \nu^N; c) \leq C\|c(\cdot,0)\|_{L^1(\mu)} \exp\left(-\kappa M^{\frac{1}{d+1}}\right) $, where $ M $ is the network size, for deep neural network approximations of holomorphic likelihood potentials.
• The framework enables stability analysis for data-driven priors and pushforward priors even when the prior and its perturbation are mutually singular, which is excluded by total variation or Hellinger metrics.
• The use of IPMs such as the Wasserstein metric allows for robust posterior perturbation analysis without requiring the perturbed prior to be absolutely continuous with respect to the original prior.
• The method applies to likelihood potentials that are only locally Lipschitz, broadening applicability to nonlinear and non-smooth inverse problems.
• The theoretical results are reinforced by a novel example showing that ReLU DNNs can approximate holomorphic likelihoods with exponential accuracy in network size, leading to corresponding posterior convergence rates.
• The framework provides a path toward convergence analysis of advanced machine learning techniques such as neural network surrogates and data-driven priors in Bayesian inverse problems.