[Paper Review] Extension of information geometry for modelling non-statistical systems

Ben Anthonis|arXiv (Cornell University)|Jan 5, 2015

Neural Networks and Applications1 references3 citations

TL;DR

This paper extends information geometry by introducing a data set model formalism that uses divergences to construct differential geometric structures on statistical models, enabling the description of non-statistical systems. The key contribution is a geometric framework based on Hessian structures and affine connections derived from divergences, which successfully reproduces expected metrics in von Mises-Fisher submanifolds despite a non-Euclidean ambient space.

ABSTRACT

In this dissertation, an abstract formalism extending information geometry is introduced. This framework encompasses a broad range of modelling problems, including possible applications in machine learning and in the information theoretical foundations of quantum theory. Its purely geometrical foundations make no use of probability theory and very little assumptions about the data or the models are made. Starting only from a divergence function, a Riemannian geometrical structure consisting of a metric tensor and an affine connection is constructed and its properties are investigated. Also the relation to information geometry and in particular the geometry of exponential families of probability distributions is elucidated. It turns out this geometrical framework offers a straightforward way to determine whether or not a parametrised family of distributions can be written in exponential form. Apart from the main theoretical chapter, the dissertation also contains a chapter of examples illustrating the application of the formalism and its geometric properties, a brief introduction to differential geometry and a historical overview of the development of information geometry.

Motivation & Objective

• To develop a general geometric framework for modeling data sets that are not necessarily statistical, extending traditional information geometry.
• To address the limitation of classical information geometry in handling non-exponential family models and non-stationary systems.
• To provide a unified mathematical structure for fitting procedures in diverse domains, including quantum physics and machine learning.
• To explore whether natural affine connections and curvature can be systematically defined in this extended formalism.
• To demonstrate the formalism's applicability through concrete examples in statistical physics and quantum information theory.

Proposed method

• Proposes a data set model formalism based on a divergence function, generalizing the concept of distance between probability distributions.
• Uses the divergence to define a Hessian structure that induces a Riemannian metric on the model manifold.
• Introduces a flat affine connection derived from the divergence, enabling consistent geometric operations on the model space.
• Applies the formalism to submanifolds of von Mises-Fisher distributions to test geometric consistency.
• Employs differential geometry tools such as metric tensors and connections to analyze the intrinsic curvature and structure of the model manifold.
• Demonstrates that the derived geometric structures match those expected from Euclidean embeddings, even when the ambient space lacks a Euclidean metric.

Experimental results

Research questions

• RQ1 Can a geometric framework be constructed for non-statistical systems using divergence-based models?
• RQ2 How can a consistent affine connection and Riemannian metric be derived from a divergence function in non-exponential family models?
• RQ3 Do the geometric structures of submanifolds of von Mises-Fisher distributions match those expected in Euclidean embeddings, despite the non-Euclidean ambient space?
• RQ4 Is there a natural one-parameter family of affine connections in this formalism, analogous to the α-connections in classical information geometry?
• RQ5 Can this formalism unify diverse modeling techniques in machine learning and quantum information theory?

Key findings

• The data set model formalism successfully constructs a Riemannian metric and affine connection from a divergence function, enabling geometric analysis of non-statistical models.
• For submanifolds of von Mises-Fisher distributions, the derived metric and connection exactly match those expected from Euclidean embeddings, even though the ambient space is not Euclidean.
• The formalism naturally accommodates models outside exponential families, such as the non-interacting boson system and linear regression, demonstrating broad applicability.
• The Hessian structure derived from the divergence provides a consistent and mathematically robust way to describe the geometry of the model manifold.
• The approach shows potential for unifying disparate modeling techniques in machine learning and quantum information, particularly through its compatibility with positive-operator valued measures (POVMs).
• The results suggest that the formalism may enable the detection of statistical models that are submanifolds of exponential families via geometric criteria.