[Paper Review] On the probabilistic rationale of I-divergence and J-divergence minimization

Marian Grendár, Grendar, Marian|ArXiv.org|Aug 4, 2000

Multi-Criteria Decision Making3 citations

TL;DR

This paper establishes a probabilistic foundation for I-divergence (Kullback-Leibler divergence) and J-divergence (Jeffreys divergence) minimization in statistical inference. It demonstrates that minimizing these divergences corresponds to maximum likelihood estimation under specific stochastic models, providing a unified probabilistic rationale for these widely used information-theoretic criteria in non-parametric and exponential family modeling.

ABSTRACT

A probabilistic rationale for I-divergence minimization (relative entropy maximization), non-parametric likelihood maximization and J-divergence minimization (Jeffres' entropy maximization) criteria is provided.

Motivation & Objective

• To provide a probabilistic justification for I-divergence minimization, commonly used in maximum entropy and relative entropy maximization.
• To establish a stochastic basis for J-divergence minimization, linked to Jeffreys' entropy maximization principle.
• To unify the interpretation of I-divergence and J-divergence minimization within a common probabilistic framework.
• To connect these divergence minimization criteria with non-parametric likelihood maximization and exponential family models.
• To clarify the statistical underpinnings of these information-theoretic criteria in terms of stochastic models and likelihood principles.

Proposed method

• Derives the probabilistic interpretation of I-divergence minimization as equivalent to maximizing the likelihood under a specific stochastic model.
• Shows that J-divergence minimization corresponds to maximizing a symmetric likelihood criterion, aligning with Jeffreys' entropy maximization.
• Uses exponential family models and sufficient statistics to formalize the connection between divergence minimization and likelihood maximization.
• Applies large deviation principles and asymptotic stochastic reasoning to justify the optimality of divergence minimization in estimation.
• Demonstrates that minimizing I-divergence corresponds to finding the maximum likelihood estimate in a non-parametric setting.
• Establishes that J-divergence minimization arises naturally from a symmetric likelihood formulation, enhancing robustness to model misspecification.

Experimental results

Research questions

• RQ1 What is the underlying probabilistic model that justifies I-divergence minimization as a statistical inference principle?
• RQ2 How does J-divergence minimization relate to likelihood-based estimation under symmetric stochastic assumptions?
• RQ3 Can both I-divergence and J-divergence minimization be unified under a single probabilistic framework?
• RQ4 What is the connection between divergence minimization and non-parametric likelihood maximization?
• RQ5 How do these divergence criteria emerge from stochastic models and large sample behavior?

Key findings

• I-divergence minimization is probabilistically justified as equivalent to maximum likelihood estimation in a non-parametric exponential family model.
• J-divergence minimization corresponds to maximizing a symmetric likelihood criterion, providing a foundation for Jeffreys' entropy maximization.
• The paper establishes that both I-divergence and J-divergence minimization arise naturally from stochastic models with specific sufficient statistics.
• The asymptotic behavior of these criteria is shown to align with large deviation principles, supporting their use in consistent estimation.
• The unified framework reveals that divergence minimization is not merely a heuristic but a statistically grounded inference method.
• The results demonstrate that minimizing I-divergence and J-divergence leads to consistent estimators under regularity conditions, with strong connections to exponential families.