[Paper Review] On Unified Generalizations of Relative Jensen--Shannon and Arithmetic--Geometric Divergence Measures, and Their Properties

Pranesh Kumar, Inder J. Taneja|ArXiv.org|Jan 19, 2005

Mathematical Inequalities and Applications16 references3 citations

TL;DR

This paper introduces a unified one-parameter generalization of non-symmetric divergence measures, including relative Jensen-Shannon and arithmetic-geometric divergences, within the framework of Csiszár's f-divergence. By applying bounds based on the ratio of probability mass functions, the authors derive tight inequalities for these generalized divergences, with explicit error bounds involving chi-squared divergence and variance terms, offering a comprehensive analytical tool for information-theoretic applications.

ABSTRACT

In this paper we shall consider one parametric generalization of some non-symmetric divergence measures. The extit{non-symmetric divergence measures} are such as: Kullback-Leibler extit{relative information}, $χ^2-$ extit{divergence}, extit{relative J -- divergence}, extit{relative Jensen -- Shannon divergence} and extit{relative Arithmetic -- Geometric divergence}. All the generalizations considered can be written as particular cases of Csiszár's extit{f-divergence}. By putting some conditions on the probability distribution, the aim here is to develop bounds on these measures and their parametric generalizations.

Motivation & Objective

• To develop a unified parametric generalization of non-symmetric divergence measures, including relative Jensen-Shannon and arithmetic-geometric divergences.
• To establish tight upper bounds for these generalized divergences under constraints on the ratio of probability masses.
• To unify and extend existing divergence measures such as Kullback-Leibler, chi-squared, and J-divergence into a single parametric family.
• To provide a framework that enables tighter error estimation and comparison across diverse divergence measures.

Proposed method

• The paper generalizes known divergences via a one-parameter family, defined as $\Omega_s(P||Q) = [s(s-1)]^{-1} \left[ \sum p_i \left( \frac{p_i + q_i}{2p_i} \right)^s - 1 \right] $ for $ s \neq 0,1 $, with limits at $ s=0,1 $.
• It leverages Csiszár's f-divergence framework to unify relative information, JS, and AG divergences under a single parametric form.
• The method applies Taylor expansion and integral representation techniques to derive bounds on the deviation of $ \Omega_s $ from known divergences.
• It introduces bounds involving $ \chi^2(P||Q) $, $ |\chi|^3(P||Q) $, and a variance-like term $ V(P||Q) $, with constants derived from the ratio bounds $ r \leq \frac{p_i}{q_i} \leq R $.
• The bounds are derived using the function $ L_{-1}^{-1} $ and $ L_{-2}^{-2} $, which are inverse functions of logarithmic means.
• Specific inequalities are proven for $ s = -1, 0, 1 $, yielding explicit bounds for $ \Delta(P||Q) $, $ F(P||Q) $, and $ G(P||Q) $.

Experimental results

Research questions

• RQ1 How can relative Jensen-Shannon and arithmetic-geometric divergences be unified under a single parametric framework?
• RQ2 What are the tightest possible bounds for the generalized divergence $ \Omega_s(P||Q) $ in terms of standard divergence measures?
• RQ3 How do the bounds on $ \Omega_s $ behave as $ s $ approaches 0, 1, and -1, and what do they reveal about the structure of the divergences?
• RQ4 Can the unified parametric form yield tighter error estimates than existing bounds for individual divergences?
• RQ5 What role do the ratio bounds $ r \leq p_i/q_i \leq R $ play in deriving uniform bounds across the family of divergences?

Key findings

• For $ s = -1 $, the generalized divergence $ \Omega_{-1}(P||Q) $ yields a bound on $ \Delta(P||Q) $, with error terms bounded by $ \min\left\{ \left[ \frac{1}{(r+1)^3} - \frac{1}{(R+1)^3} \right] \chi^2(P||Q), \frac{1}{(r+1)^4} |\chi|^3(P||Q), \frac{2(R-r)(R+r+2)}{(r+1)^2(R+1)^2} V(P||Q) \right\} $.
• For $ s = 0 $, the bound on $ F(P||Q) $ is $ \left| F(P||Q) - \sum (q_i - p_i) \ln\left( \frac{p_i - q_i}{p_i + 3q_i} \right) + \frac{1}{2} \sum \frac{(p_i - q_i)^2}{p_i + 3q_i} \right| \leq \min\left\{ \frac{1}{8} \left[ \frac{1}{r(r+1)^2} - \frac{1}{R(R+1)^2} \right] \chi^2(P||Q), \frac{3r+1}{24r^2(r+1)^3} |\chi|^3(P||Q), \frac{R-r}{2rR} \left[ L_{-1}^{-1} - L_{-2}^{-2} \right] V(P||Q) \right\} $.
• For $ s = 1 $, the bound on $ G(P||Q) $ is $ \left| G(P||Q) - \frac{1}{2} \Delta(P||Q) - \frac{1}{2} \sum (p_i - q_i) \ln\left( \frac{p_i - q_i}{p_i + 3q_i} \right) \right| \leq \min\left\{ \frac{1}{16} \left[ \frac{1}{r^2(r+1)} - \frac{1}{R^2(R+1)} \right] \chi^2(P||Q), \frac{3r+2}{48r^3(r+1)^2} |\chi|^3(P||Q), \frac{R-r}{4rR} \left[ 1 - L_{-1}^{-1} \right] V(P||Q) \right\} $.
• The bounds are uniformly tight across the parametric family, with explicit dependence on $ r $ and $ R $, the lower and upper bounds of $ p_i/q_i $.
• The derived bounds are sharper than existing ones, as demonstrated by the inclusion of higher-order terms like $ |\chi|^3(P||Q) $ and variance-like components.
• The unified parametric form allows for the recovery of known divergences (e.g., $ \Phi_1 = K(P||Q) $, $ \Phi_2 = \frac{1}{2} \chi^2(P||Q) $) as special cases, validating the generalization.