[Paper Review] A generalization of the Jensen divergence: The chord gap divergence

Frank Nielsen|arXiv (Cornell University)|Sep 29, 2017

Advanced Statistical Methods and Models22 references3 citations

TL;DR

This paper introduces the chord gap divergence, a triparametric generalization of the Jensen divergence that extends skew Jensen and Bhattacharyya divergences, enabling improved modeling of statistical dissimilarities in exponential families. It derives a Taylor-Lagrange remainder form for the divergence and establishes the competitiveness of k-means++ clustering under this new metric via convergence analysis.

ABSTRACT

We introduce a novel family of distances, called the chord gap divergences, that generalizes the Jensen divergences (also called the Burbea-Rao distances), and study its properties. It follows a generalization of the celebrated statistical Bhattacharyya distance that is frequently met in applications. We report an iterative concave-convex procedure for computing centroids, and analyze the performance of the $k$-means++ clustering with respect to that new dissimilarity measure by introducing the Taylor-Lagrange remainder form of the skew Jensen divergences.

Motivation & Objective

• To generalize the Jensen divergence into a triparametric family of divergences that subsumes skew Jensen and Bhattacharyya distances.
• To provide a unified framework for statistical dissimilarity measures in exponential families using convex function inequalities.
• To enable better performance in clustering and information fusion by introducing a flexible, tunable divergence metric.
• To analyze the convergence and performance of k-means++ under the new divergence using Taylor-Lagrange remainder forms.
• To support reproducible research by releasing public Java source code for the chord gap divergence.

Proposed method

• Proposes the chord gap divergence as a generalization of skew Jensen divergences using three parameters: α, β, and γ.
• Derives the Taylor-Lagrange remainder form of the chord gap divergence using second-order Taylor expansions with exact remainder terms.
• Establishes that the chord gap divergence reduces to Bregman divergences in limit cases (α→0 or α→1).
• Introduces an iterative concave-convex procedure for computing centroids under the chord gap divergence.
• Uses the chord gap divergence to analyze k-means++ performance, showing it is Oρ(log k)-competitive under the divergence.
• Demonstrates that the chord gap divergence generalizes the skew Bhattacharyya distance for distributions in the same exponential family.

Experimental results

Research questions

• RQ1 How can the Jensen divergence be generalized to a triparametric family that includes skew Jensen and Bhattacharyya divergences as special cases?
• RQ2 What is the analytical structure of the chord gap divergence in terms of second-order Taylor expansions with exact remainders?
• RQ3 How does the chord gap divergence relate to Bregman divergences and Kullback-Leibler divergence in limit cases?
• RQ4 Can k-means++ be proven competitive under the chord gap divergence using remainder-based error bounds?
• RQ5 What is the role of the Hessian matrix in characterizing the curvature of the chord gap divergence?

Key findings

• The chord gap divergence generalizes the skew Jensen divergence and includes the Bhattacharyya distance as a special case when distributions belong to the same exponential family.
• The Taylor-Lagrange remainder form of the chord gap divergence is expressed as (p−q)⊤H_F^{α,β,γ}(p:q)(p−q), where the Hessian matrix term depends on intermediate points in the convex hull.
• The chord gap divergence centroid can be computed via an iterative concave-convex procedure, ensuring convergence for strictly convex functions.
• The k-means++ algorithm is shown to be Oρ(log k)-competitive for the chord gap divergence, where ρ is a condition number based on the Hessian's spectral properties.
• The skew Jensen divergence admits a Taylor-Lagrange remainder form: J_F^{(α)}(p:q) = (α(1−α)/2)(p−q)⊤(α∇²F(ξ₁)+(1−α)∇²F(ξ₂))(p−q).
• In the limit α→0 or α→1, the chord gap divergence converges to the Bregman divergence B_F(q:p) or B_F(p:q), respectively, confirming consistency with known divergences.