[Paper Review] Trivergence of Probability Distributions, at glance

Juan‐Manuel Torres‐Moreno|arXiv (Cornell University)|Jun 20, 2015

Statistical Mechanics and Entropy1 references3 citations

TL;DR

This paper introduces the concept of trivergence of probability distributions (TPD), a novel measure to quantify similarity among triplets of probability distributions using divergence metrics like Kullback-Leibler and Jensen-Shannon. It proposes two computational approaches—product of divergences and compound divergence function—enabling applications in text mining and information retrieval with smoothed probability estimation for unseen events.

ABSTRACT

In this paper we introduce the intuitive notion of trivergence of probability distributions (TPD). This notion allow us to calculate the similarity among triplets of objects. For this computation, we can use the well known measures of probability divergences like Kullback-Leibler and Jensen-Shannon. Divergence measures may be used in Information Retrieval tasks as Automatic Text Summarization, Text Classification, among many others.

Motivation & Objective

• To formalize a new measure of similarity among three probability distributions, termed trivergence, extending traditional pairwise divergence measures.
• To address the limitation of pairwise divergences in capturing complex relationships among three objects in data analysis and information retrieval.
• To develop a smoothed trivergence framework that handles unseen events using elementary probability smoothing techniques.
• To provide two computationally viable methods—product of divergences and compound divergence function—for practical implementation of trivergence.
• To demonstrate applicability in text-related tasks such as automatic text summarization and classification using divergence-based similarity.

Proposed method

• Proposes a trivergence measure τ as a product of three pairwise divergences: τ^π(p||q||r) = D(p||q) · D(q||r) · D(p||r), with ordering by distribution cardinality.
• Introduces a compound divergence function τ^c(p||q||r) = D[p || D(q||r)/|q∪r|], treating the divergence between q and r as a probability distribution for comparison with p.
• Applies elementary smoothing to handle zero-probability events: for unseen items, assign probability 1/|T| where |T| = |p| + |q| + |r|.
• Uses Kullback-Leibler (KL) and Jensen-Shannon (JS) divergences as base divergence measures, with JS divergence preferred due to its symmetry and metric properties.
• Derives explicit formulas for JS-based trivergence, including case-wise decomposition over the 7-set partition of p, q, and r (e.g., p∩q∩r, p\(q∪r), etc.).
• Employs logarithmic scoring with base-2 logs and handles edge cases (e.g., D(q||r) = 0) via defined limit behaviors in the compound function.

Experimental results

Research questions

• RQ1 How can a symmetric and meaningful measure of similarity be defined for three probability distributions, beyond pairwise comparisons?
• RQ2 What are the mathematical and computational properties of a trivergence measure based on existing divergence functions?
• RQ3 How can smoothing techniques be effectively integrated into trivergence computation to handle unseen events in empirical distributions?
• RQ4 Which formulation—product of divergences or compound divergence—better preserves interpretability and metric consistency in practical applications?
• RQ5 Can trivergence be effectively applied to real-world NLP tasks such as text summarization and classification using standard divergence metrics?

Key findings

• The trivergence measure τ^π(p||q||r) = D(p||q) · D(q||r) · D(p||r) provides a product-based similarity score that increases with pairwise dissimilarity, with ordering by cardinality to ensure consistency.
• The compound divergence formulation τ^c(p||q||r) = D[ p || D(q||r)/|q∪r| ] treats the JS divergence between q and r as a normalized probability distribution, enabling hierarchical comparison with p.
• The use of smoothing with probability 1/|T| for unseen items ensures numerical stability and avoids infinite divergence values in KL-based computations.
• Explicit analytical expressions are derived for D_JS(q||r) and τ^c_JS(p||q||r), decomposed over the 7 disjoint regions of the Venn diagram of p, q, and r.
• The Jensen-Shannon trivergence is shown to be well-defined and computable even when some distributions are disjoint, due to the smoothing and normalization in the compound function.
• The framework supports practical deployment in information retrieval and NLP tasks, with trivergence serving as a higher-order similarity measure beyond pairwise divergence.