[Paper Review] Local Pinsker inequalities via Stein's discrete density approach

Christophe Ley, Yvik Swan|arXiv (Cornell University)|Nov 15, 2012

Random Matrices and Applications30 references4 citations

TL;DR

This paper introduces generalized discrete Fisher information distances, denoted $\mathcal{J}(X,Y)$, for discrete distributions using a novel Stein operator framework. It proves that these distances dominate the square of the total variation distance, establishing local Pinsker-type inequalities with improved constants compared to existing results.

ABSTRACT

Pinsker's inequality states that the relative entropy $d_{\mathrm{KL}}(X, Y)$ between two random variables $X$ and $Y$ dominates the square of the total variation distance $d_{\mathrm{TV}}(X,Y)$ between $X$ and $Y$. In this paper we introduce generalized Fisher information distances $\mathcal{J}(X, Y)$ between discrete distributions $X$ and $Y$ and prove that these also dominate the square of the total variation distance. To this end we introduce a general discrete Stein operator for which we prove a useful covariance identity. We illustrate our approach with several examples. Whenever competitor inequalities are available in the literature, the constants in ours are at least as good, and, in several cases, better.

Motivation & Objective

• To develop a general framework for local Pinsker inequalities in discrete probability using Stein's method.
• To define a generalized discrete Fisher information distance $\mathcal{J}(X,Y)$ that quantifies discrepancy between discrete distributions $X$ and $Y$.
• To establish a covariance identity for a new discrete Stein operator that enables the derivation of information-theoretic inequalities.
• To improve upon existing bounds in the literature by providing tighter constants in local Pinsker inequalities for discrete distributions.
• To unify and generalize existing approaches to discrete information distances, including those based on Poisson and binomial targets.

Proposed method

• Introduces a general discrete Stein operator tailored for arbitrary target distributions $p$, defined via a forward and backward operator on functions in $\mathcal{F}^\eta(p)$.
• Derives a key covariance identity linking the Stein operator to the generalized Fisher information distance $\mathcal{J}(X,Y) = \mathbb{E}_q[(r(p,q)(Y))^2]$, where $r(p,q)$ is a score-like functional.
• Applies the framework to specific distributions: Poisson, binomial, negative hypergeometric, and Gibbs measures, deriving explicit forms of the Stein operators.
• Establishes that $\mathcal{J}(X,Y) \geq 0$ with equality iff $X \stackrel{\mathcal{L}}{=} Y$, ensuring the functional acts as a proper discrepancy measure.
• Uses the covariance identity to prove $d_{\mathrm{TV}}(X,Y) \leq \kappa \sqrt{\mathcal{J}(X,Y)}$, a local Pinsker inequality, with $\kappa$ depending only on the target distribution $p$.
• Validates the framework through examples, showing that the derived constants in the inequality are at least as good as, and often better than, those in prior literature.

Experimental results

Research questions

• RQ1 Can a general discrete Stein operator be constructed to yield a Fisher-type information distance that dominates total variation for arbitrary discrete distributions?
• RQ2 Does the resulting generalized Fisher information distance $\mathcal{J}(X,Y)$ satisfy a local Pinsker inequality with a universal constant $\kappa$ depending only on the target distribution?
• RQ3 How do the constants in the derived local Pinsker inequalities compare to those in existing literature for Poisson, binomial, and other discrete targets?
• RQ4 Can the framework be systematically applied to exponential families and other discrete exponential families using a unified operator construction?
• RQ5 What is the role of the covariance identity in connecting the Stein operator to information-theoretic divergences in the discrete setting?

Key findings

• The generalized Fisher information distance $\mathcal{J}(X,Y)$ is defined via a new discrete Stein operator and satisfies $\mathcal{J}(X,Y) \geq 0$ with equality if and only if $X \stackrel{\mathcal{L}}{=} Y$.
• The paper proves a local Pinsker inequality: $d_{\mathrm{TV}}(X,Y) \leq \kappa \sqrt{\mathcal{J}(X,Y)}$, where $\kappa$ depends only on the target distribution $p$.
• For the Poisson target, the derived constant in the inequality is at least as good as, and in several cases better than, those from existing approaches using $\mathcal{K}(Po(\lambda),Y)$ or $\mathcal{J}(Po(\lambda),Y)$.
• The framework is applied to multiple discrete distributions, including binomial, negative hypergeometric, and Gibbs measures, yielding explicit forms of the Stein operators and corresponding information distances.
• The covariance identity derived for the discrete Stein operator enables the derivation of the local Pinsker inequality in a unified and general way.
• The method improves upon known bounds in the literature, particularly in terms of the multiplicative constant $\kappa$, by leveraging a more refined score function construction.