[Paper Review] Bounds on $f$-Divergences and Related Distances

Igal Sason|arXiv (Cornell University)|Mar 27, 2014

Wireless Communication Security Techniques3 citations

TL;DR

This paper derives tight bounds for symmetric f-divergences—such as total variation, Hellinger, and Pearson chi-squared divergence—in terms of the total variation distance. It proves these bounds are achievable using 2- or 3-element probability distributions and applies them to improve lossless source coding bounds and channel-code detection.

ABSTRACT

Tight bounds for several symmetric divergence measures are derived in terms of the total variation distance. It is shown that each of these bounds is attained by a pair of 2 or 3-element probability distributions. An application of these bounds for lossless source coding is provided, refining and improving a certain bound by Csiszar. Another application of these bounds has been recently introduced by Yardi. et al. for channel-code detection.

Motivation & Objective

• To derive tight, explicit bounds for symmetric f-divergences using the total variation distance as a reference metric.
• To identify the minimal probability spaces (2- or 3-element distributions) that achieve these bounds.
• To refine existing bounds in information theory, particularly Csiszar’s bound in lossless source coding.
• To enable improved performance analysis in applications such as channel-code detection.

Proposed method

• Derives analytical bounds for symmetric f-divergences by exploiting the convexity and symmetry properties of f-divergence functions.
• Uses variational optimization over probability distributions with 2 or 3 elements to identify extremal configurations that achieve the tightest bounds.
• Expresses bounds in terms of total variation distance, establishing a direct functional relationship between f-divergences and this fundamental metric.
• Applies the derived bounds to refine Csiszar’s bound in lossless source coding via information-theoretic inequalities.
• Validates the tightness of bounds by constructing explicit 2- or 3-point distributions that achieve equality in the derived inequalities.
• Leverages known duality and minimax principles in f-divergence theory to establish optimality of the derived bounds.

Experimental results

Research questions

• RQ1 What is the tightest possible upper bound for a symmetric f-divergence in terms of total variation distance?
• RQ2 Which probability distributions achieve the tightest bounds among all possible distributions?
• RQ3 How can these bounds be applied to improve existing performance limits in lossless source coding?
• RQ4 Can these bounds be used to strengthen detection criteria in channel-code analysis?

Key findings

• Tight bounds for symmetric f-divergences are derived in terms of total variation distance, with explicit functional forms.
• The bounds are shown to be tight and are achieved by specific 2- or 3-element probability distributions.
• The derived bounds improve upon Csiszar’s bound in lossless source coding, providing a tighter characterization of coding efficiency.
• The results are applicable to channel-code detection, as demonstrated by Yardi et al., enhancing detection performance bounds.
• The analysis reveals that the extremal distributions for these bounds are sparse, involving only two or three outcomes.
• The method establishes a general framework for relating multiple f-divergences to a single reference metric (total variation).