[Paper Review] A Geometric View of Posterior Approximation

Tian Chen, Jeffrey Streets+1 more|arXiv (Cornell University)|Oct 3, 2015

Gaussian Processes and Bayesian Inference37 references3 citations

TL;DR

This paper introduces a geometric framework for posterior approximation in Bayesian inference using a distance measure derived from ambient Fisher geometry, offering a principled alternative to traditional variational methods. By treating the posterior approximation as a Riemannian optimization problem on a statistical manifold, the method enables more accurate and globally consistent approximations than standard KL-divergence-based approaches.

ABSTRACT

Although Bayesian methods are robust and principled, their application in practice could be limited since they typically rely on computationally intensive Markov Chain Monte Carlo algorithms for their implementation. One possible solution is to find a fast approximation of posterior distribution and use it for statistical inference. For commonly used approximation methods, such as Laplace and variational free energy, the objective is mainly defined in terms of computational convenience as opposed to a true distance measure between the target and approximating distributions. In this paper, we provide a geometric view of posterior approximation based on a valid distance measure derived from ambient Fisher geometry. Our proposed framework is easily generalizable and can inspire a new class of methods for approximate Bayesian inference.

Motivation & Objective

• To address the limitations of existing posterior approximation methods that rely on computationally convenient divergences rather than true geometric distances.
• To develop a principled, geometrically grounded framework for approximating posterior distributions in Bayesian inference.
• To generalize existing variational methods by embedding them within a Riemannian geometric structure based on Fisher information.
• To enable more accurate and globally consistent posterior approximations by minimizing a valid distance measure derived from ambient Fisher geometry.

Proposed method

• The method formulates posterior approximation as a Riemannian optimization problem on a statistical manifold, where the distance between distributions is measured via the Fisher information metric.
• It defines a valid distance measure derived from ambient Fisher geometry, replacing ad hoc divergences like KL-divergence with a geometrically consistent alternative.
• The approach uses orthonormal basis vectors derived from score functions of the approximating density to parameterize the tangent space at a given point on the manifold.
• It applies gradient-like updates on the manifold using inner products between basis vectors and the square root of the true posterior, ensuring geometric consistency.
• The optimization proceeds via iterative updates of parameters (e.g., mean and variance) using normalized directional components derived from the inner product with the posterior's square root.
• The framework generalizes beyond exponential families and allows for flexible approximation families by leveraging the intrinsic geometry of the statistical model.

Experimental results

Research questions

• RQ1 Can a geometric distance measure derived from Fisher information provide a more principled alternative to KL-divergence in posterior approximation?
• RQ2 How can the structure of the statistical manifold be exploited to improve the global accuracy of posterior approximations?
• RQ3 To what extent does the proposed geometric framework generalize beyond standard variational families like mean-field approximations?
• RQ4 What are the implications of using a Riemannian optimization framework for posterior approximation in terms of convergence and approximation quality?

Key findings

• The proposed geometric framework provides a valid, Riemannian distance measure between posterior and approximating distributions, derived from ambient Fisher geometry, which is more principled than traditional divergences.
• The method enables accurate posterior approximation by optimizing in the tangent space of the statistical manifold using orthonormal basis vectors derived from score functions.
• The framework generalizes existing variational methods by embedding them in a geometric context, allowing for better global approximation properties beyond local mode matching.
• The use of inner products between basis vectors and the square root of the true posterior ensures geometric consistency and stable parameter updates.
• The approach achieves improved approximation quality in high-dimensional settings by avoiding the asymmetry and bias inherent in KL-divergence minimization.