[Paper Review] Computational information geometry: theory and practice

Karim Anaya‐Izquierdo, Frank Critchley+2 more|arXiv (Cornell University)|Sep 10, 2012

Bayesian Modeling and Causal Inference25 references3 citations

TL;DR

This paper introduces computational information geometry, a unified framework that uses the simplex as a fundamental geometric object to enable numerical computation across diverse statistical models—especially those without fixed dimension or support. By embedding models in finite-dimensional spaces, it integrates information geometry, exponential families, and mixture geometry, enabling robust computation of likelihoods, curvature, and model selection metrics via explicit numerical algorithms.

ABSTRACT

This paper lays the foundations for a unified framework for numerically and computationally applying methods drawn from a range of currently distinct geometrical approaches to statistical modelling. In so doing, it extends information geometry from a manifold based approach to one where the simplex is the fundamental geometrical object, thereby allowing applications to models which do not have a fixed dimension or support. Finally, it starts to build a computational framework which will act as a proxy for the 'space of all distributions' that can be used, in particular, to investigate model selection and model uncertainty. A varied set of substantive running examples is used to illustrate theoretical and practical aspects of the discussion. Further developments are briefly indicated.

Motivation & Objective

• To unify disparate geometric approaches in statistics—information geometry, exponential families, and mixture geometry—into a single computational framework.
• To extend information geometry beyond fixed-dimensional manifolds to models with variable support and dimension using the simplex as a universal domain.
• To develop a computational proxy for the space of all probability distributions to study model selection and model uncertainty.
• To enable numerical implementation of geometric statistical methods through explicit computation in finite-dimensional simplex spaces.
• To demonstrate the framework’s utility through diverse, real-world statistical examples, including mixture models, logistic regression, and curved exponential families.

Proposed method

• Represent statistical models as subsets of the probability simplex $\Delta^k$, where $k+1$ is the number of categories, enabling finite-dimensional numerical treatment.
• Use the extended multinomial distribution to model distributions with variable support, allowing non-fixed dimensionality in the geometric framework.
• Apply differential geometry on the simplex to compute curvature, Fisher information, and higher-order moments numerically.
• Utilize linear projections $\Pi_L$ to preserve convexity and likelihood structure, enabling efficient computation of likelihood and score functions.
• Leverage the log-likelihood expansion and directional derivatives to compute nonparametric maximum likelihood estimates (NPMLE) in mixture models.
• Use total positivity of the kernel $K(x,y) = \exp(xy)$ to prove nonsingularity of key matrices, ensuring rank conditions for model identifiability.

Experimental results

Research questions

• RQ1 How can information geometry be extended to models without fixed dimension or support, such as those with variable support sets?
• RQ2 Can a single computational framework unify disparate geometric approaches in statistics, including exponential families, mixture models, and information geometry?
• RQ3 How can the simplex serve as a universal domain for numerically approximating the space of all probability distributions?
• RQ4 What numerical algorithms can be derived from geometric principles to compute likelihood, curvature, and model adequacy in complex models?
• RQ5 How can higher-order asymptotic methods and curvature-based dimension reduction be implemented computationally in non-exponential family models?

Key findings

• The simplex $\Delta^k$ provides a universal, finite-dimensional embedding space for all probability distributions, enabling numerical computation across models with variable dimension and support.
• The framework successfully unifies information geometry, extended exponential families, and Lindsay’s mixture geometry through a common algebraic and geometric foundation.
• Numerical computation of the log-likelihood, score functions, and higher-order derivatives (e.g., third derivatives) is feasible and explicitly derived using the extended multinomial model.
• The NPMLE in mixture models is shown to satisfy a first-order bound on log-likelihood change within $\epsilon$-neighborhoods, enabling stability analysis.
• The rank of the matrix $\widetilde{B}$, derived from exponential families on the simplex, is proven to be $k$, ensuring model identifiability and invertibility of key geometric quantities.
• The use of total positivity of the kernel $K(x,y) = \exp(xy)$ ensures that the matrix $B^*$ is nonsingular, which is critical for proving the rank condition in the geometric framework.