[Paper Review] Generalized Symmetric Divergence Measures and Metric Spaces

G. A. T. F. da Costa, Inder J. Taneja|arXiv (Cornell University)|May 13, 2011

Statistical Mechanics and Entropy7 references3 citations

TL;DR

This paper establishes that the square root of two generalized symmetric divergence measures—Jensen-Shannon and Arithmetic-Geometric divergences—induce metric space structures on the space of discrete probability distributions. Using analytical proofs based on convexity and monotonicity, it demonstrates that the square root of these divergences satisfies the triangle inequality and thus forms a metric, extending known results for specific divergences like Hellinger and triangular discrimination.

ABSTRACT

Recently, Taneja studied two one parameter generalizations of J-divergence, Jensen-Shannon divergence and Arithmetic-Geometric divergence. These two generalizations in particular contain measures like: Hellinger discrimination, symmetric chi-square divergence, and triangular discrimination. These measures are well known in the literature of Statistics and Information theory. In this paper our aim is to prove metric space properties for square root of these two symmetric generalized divergence measures.

Motivation & Objective

• To establish metric space properties for the square root of two generalized symmetric divergence measures: the Arithmetic-Geometric (AG) and Jensen-Shannon (JS) divergences.
• To extend known results on specific divergences (e.g., Hellinger, triangular discrimination) to a broader class of divergences parameterized by a real parameter s.
• To unify and generalize existing inequalities among symmetric divergences by proving metric properties under the square root transformation.
• To provide a theoretical foundation for using these divergences in applications requiring metric structure, such as clustering and information geometry.

Proposed method

• Define two generalized symmetric divergence measures: $\xi_s(P||Q)$ (AG-type) and $\zeta_s(P||Q)$ (J-divergence type), parameterized by $s \in \mathbb{R}$.
• Prove that $\sqrt{\xi_s(P||Q)}$ and $\sqrt{\zeta_s(P||Q)}$ satisfy the four metric axioms, with focus on the triangle inequality.
• Use substitution $t = p/r$ and $\beta = p/q$ to analyze the derivative of the sum of square roots, showing it has a unique minimum at $p = q = r$.
• Apply continuity arguments for $s = 0$ and $s = 1$, where closed-form expressions are used.
• Use Taylor expansion and second-order derivatives to derive asymptotic approximations of the divergences near $P \to Q$.
• Relate the divergences to Csiszár $f$-divergences and apply known results on second-order expansions to derive asymptotic equivalence to $\chi^2$-divergence.

Experimental results

Research questions

• RQ1 Does the square root of the generalized symmetric Arithmetic-Geometric divergence $\xi_s(P||Q)$ satisfy the triangle inequality and thus define a metric?
• RQ2 Does the square root of the generalized symmetric J-divergence $\zeta_s(P||Q)$ form a metric space for all $s \in \mathbb{R}$?
• RQ3 How do the generalized divergences relate to known divergences (e.g., Hellinger, triangular discrimination) in the limit as $P \to Q$?
• RQ4 Can the asymptotic behavior of these divergences be characterized using second-order Taylor expansion around $x=1$?
• RQ5 What is the relationship between the second derivative of the generating function and the asymptotic coefficient of the $\chi^2$-divergence?

Key findings

• The square root of $\xi_s(P||Q)$, the generalized AG-divergence, forms a metric on $\Gamma_n$ for all $s \in \mathbb{R}$, including $s=0,1$.
• The square root of $\zeta_s(P||Q)$, the generalized J-divergence, also forms a metric on $\Gamma_n$ for all $s \in \mathbb{R}$, proven via derivative analysis and continuity.
• For $P \to Q$, $\zeta_s(P||Q) \approx \frac{1}{8} \chi^2(P||Q)$, showing asymptotic equivalence to $\frac{1}{8}$ times the symmetric $\chi^2$-divergence.
• For $P \to Q$, $\xi_s(P||Q) \approx \chi^2(P||Q)$, indicating the AG-type divergence asymptotically matches the standard $\chi^2$-divergence.
• The second derivative of the generating function $\psi_s(x)$ at $x=1$ is $\psi_s''(1) = \frac{1}{4}$, which underpins the asymptotic approximation of $\zeta_s$.
• The second derivative of $\phi_s(x)$ at $x=1$ is $\phi_s''(1) = 2$, which leads to the asymptotic equivalence $\xi_s(P||Q) \approx \chi^2(P||Q)$.