[Paper Review] Generalizing Jensen and Bregman divergences with comparative convexity and the statistical Bhattacharyya distances with comparable means

Frank Nielsen, Richard Nock|arXiv (Cornell University)|Feb 16, 2017

Advanced Statistical Methods and Models24 references3 citations

TL;DR

This paper generalizes Jensen and Bregman divergences using comparative convexity based on abstract means, particularly quasi-arithmetic means, and introduces a novel generalization of Bhattacharyya distances using comparable means. The key contribution is a closed-form expression for generalized Bregman divergences under quasi-arithmetic means, enabling efficient computation and application in machine learning and statistics.

ABSTRACT

Comparative convexity is a generalization of convexity relying on abstract notions of means. We define the Jensen divergence and the Jensen diversity from the viewpoint of comparative convexity, and show how to obtain the generalized Bregman divergences as limit cases of skewed Jensen divergences. In particular, we report explicit formula of these generalized Bregman divergences when considering quasi-arithmetic means. Finally, we introduce a generalization of the Bhattacharyya statistical distances based on comparative means using relative convexity.

Motivation & Objective

• To extend classical Jensen and Bregman divergences beyond standard convexity by introducing a framework based on comparative convexity with abstract means.
• To derive a closed-form expression for generalized Bregman divergences when using quasi-arithmetic means, enabling practical computation.
• To generalize the Bhattacharyya distance using comparable means, particularly for statistical distributions with non-trivial structure.
• To establish conditions under which the generalized Bhattacharyya distance yields homogeneous divergences and closed-form expressions.
• To demonstrate the utility of these generalized divergences in machine learning and information theory, especially for error-bound optimization and statistical inference.

Proposed method

• Define generalized Jensen divergences via comparative convexity using two abstract means M (domain) and N (codomain), generalizing standard midpoint convexity.
• Derive generalized Bregman divergences as the limit of skewed Jensen divergences, establishing a link between skewness and divergence behavior.
• Provide a closed-form formula for generalized Bregman divergences when M and N are quasi-arithmetic means, using inverse functions of generating functions.
• Introduce a generalized Bhattacharyya distance using comparable means, defined as the logarithmic ratio of means applied to probability densities or mass functions.
• Establish conditions under which the generalized Bhattacharyya distance becomes a homogeneous divergence, particularly when using homogeneous means.
• Apply the framework to specific families such as Cauchy and multivariate t-distributions, showing closed-form expressions via deformed exponential families or power means.

Experimental results

Research questions

• RQ1 How can Jensen and Bregman divergences be generalized using comparative convexity with abstract means?
• RQ2 What is the closed-form expression for generalized Bregman divergences when the means are quasi-arithmetic?
• RQ3 Can the Bhattacharyya distance be generalized to comparable means, and under what conditions does it yield closed-form expressions?
• RQ4 How do the generalized divergences behave under homogeneity and what implications does this have for statistical learning?
• RQ5 What is the relationship between the generalized Bhattacharyya distance and Chernoff information in terms of error probability bounds?

Key findings

• The generalized Bregman divergence under quasi-arithmetic means admits a closed-form expression, enabling efficient computation and integration into machine learning pipelines.
• The generalized Bregman divergences are conformal to standard Bregman divergences when applied to an embedded representation of the input space via the generating functions.
• The generalized Bhattacharyya distance using comparable means yields a symmetric, non-negative divergence that can be computed in closed form for discrete distributions and certain continuous families.
• For the Cauchy distribution family, the generalized Bhattacharyya distance is not expressible in closed form using standard (G,A) means, but can be tailored via deformed exponential families.
• When homogeneous comparative means are used, the generalized Bhattacharyya distance becomes a homogeneous divergence, preserving scale invariance.
• The framework allows for learning not only separable convex generators but also the underlying functions defining the means, enabling flexible, data-driven divergence design.