[Paper Review] Weighted information and entropy rates

Yuri Suhov, Izabella Stuhl|arXiv (Cornell University)|Dec 29, 2016

Statistical Mechanics and Entropy14 references3 citations

TL;DR

This paper introduces weighted entropy and information rates for stochastic processes using additive and multiplicative weight functions (WFs), establishing asymptotic scaling laws: $\frac{1}{n^2}H^{\rm w}_{\phi_n}$ for additive WFs and $\frac{1}{n}\log H^{\rm w}_{\phi_n}$ for multiplicative WFs. It derives primary and secondary rates in ergodic and Markov processes, generalizing the Shannon–McMillan–Breiman theorem for weighted entropies in Gaussian processes with explicit formulas for entropy rates under quadratic and exponential WFs.

ABSTRACT

The weighted entropy $H^{ m w}_ϕ(X)=H^{ m w}_ϕ(f)$ of a random variable $X$ with values $x$ and a probability-mass/density function $f$ is defined as the mean value ${\mathbb E} I^{ m w}_ϕ(X)$ of the weighted information $I^{ m w}_ϕ(x)=-ϕ(x)\log\,f(x)$. Here $x\mapstoϕ(x)\in{\mathbb R}$ is a given weight function (WF) indicating a 'value' of outcome $x$. For an $n$-component random vector ${\mathbf{X}}_0^{n-1}=(X_0,\ldots ,X_{n-1})$ produced by a random process ${\mathbf{X}}=(X_i,i\in{\mathbb Z})$, the weighted information $I^{ m w}_{ϕ_n}({\mathbf x}_0^{n-1})$ and weighted entropy $H^{ m w}_{ϕ_n}({\mathbf{X}}_0^{n-1})$ are defined similarly, with an WF $ϕ_n({\mathbf x}_0^{n-1})$. Two types of WFs $ϕ_n$ are considered, based on additive and a multiplicative forms ($ϕ_n({\mathbf x}_0^{n-1})=\sum\limits_{i=0}^{n-1}φ (x_i)$ and $ϕ_n({\mathbf x}_0^{n-1})=\prod\limits_{i=0}^{n-1}φ (x_i)$, respectively). The focus is upon ${\it rates}$ of the weighted entropy and information, regarded as parameters related to ${\mathbf{X}}$. We show that, in the context of ergodicity, a natural scale for an asymptotically additive/multiplicative WF is $\frac{1}{n^2}H^{ m w}_{ϕ_n}({\mathbf{X}}_0^{n-1})$ and $\frac{1}{n}\log\;H^{ m w}_{ϕ_n}({\mathbf{X}}_0^{n-1})$, respectively. This gives rise to ${\it primary}$ ${\it rates}$. The next-order terms can also be identified, leading to ${\it secondary}$ ${\it rates}$. We also consider emerging generalisations of the Shannon-McMillan-Breiman theorem.

Motivation & Objective

• To extend the concept of entropy rate to weighted entropy by incorporating value-dependent weight functions (WFs) that reflect the significance of outcomes.
• To define and analyze weighted information and entropy rates for random processes, particularly focusing on ergodic, Markov, and Gaussian processes.
• To identify appropriate asymptotic scaling for weighted entropy rates—$\frac{1}{n^2}$ for additive WFs and $\frac{1}{n}\log$ for multiplicative WFs—leading to primary and secondary rate parameters.
• To generalize the Shannon–McMillan–Breiman theorem for weighted entropies in the context of stationary processes.
• To derive explicit expressions for weighted entropy rates in Gaussian processes under quadratic and exponential weight functions.

Proposed method

• Define weighted information as $I^{\rm w}_{\phi}(x) = -\phi(x)\log f(x)$, where $\phi(x)$ is a weight function reflecting outcome value, and weighted entropy as $H^{\rm w}_{\phi}(X) = \mathbb{E}[I^{\rm w}_{\phi}(X)]$.
• Extend the definition to vector-valued processes $\mathbf{X}_{0}^{n-1}$, with joint weighted entropy $H^{\rm w}_{\phi_n}(f_n) = \mathbb{E}[I^{\rm w}_{\phi_n}(\mathbf{X}_{0}^{n-1})]$ for weight functions $\phi_n(\mathbf{x}_{0}^{n-1})$.
• Consider two classes of weight functions: additive ($\phi_n = \sum_{i=0}^{n-1} \varphi(x_i)$) and multiplicative ($\phi_n = \prod_{i=0}^{n-1} \varphi(x_i)$), with $\varphi \geq 0$ in the multiplicative case.
• Establish asymptotic scaling: $\frac{1}{n^2}H^{\rm w}_{\phi_n}$ for additive WFs and $\frac{1}{n}\log H^{\rm w}_{\phi_n}$ for multiplicative WFs to define primary rates.
• Derive secondary rate terms by analyzing next-order asymptotics in the entropy rate expansion.
• Apply the framework to Gaussian processes with quadratic and exponential WFs, deriving explicit formulas for $H^{\rm w}_{\phi_n}(f_n)$ using moment generating functions and matrix algebra.

Experimental results

Research questions

• RQ1 What is the correct asymptotic scaling for weighted entropy rates when using additive and multiplicative weight functions in stationary processes?
• RQ2 How do primary and secondary rates emerge in the asymptotic expansion of weighted entropy for ergodic and Markov processes?
• RQ3 Can the Shannon–McMillan–Breiman theorem be generalized to weighted entropies under additive and multiplicative WFs?
• RQ4 What are the explicit expressions for weighted entropy rates in Gaussian processes with quadratic and exponential weight functions?
• RQ5 How do the weighted entropy rates relate to standard entropy rates when the weight function is non-constant?

Key findings

• For additive weight functions $\phi_n = \sum_{i=0}^{n-1} \varphi(x_i)$, the primary weighted entropy rate is defined as $\lim_{n \to \infty} \frac{1}{n^2} H^{\rm w}_{\phi_n}(\mathbf{X}_{0}^{n-1})$, with secondary rates emerging from next-order terms.
• For multiplicative weight functions $\phi_n = \prod_{i=0}^{n-1} \varphi(x_i)$, the primary rate is $\lim_{n \to \infty} \frac{1}{n} \log H^{\rm w}_{\phi_n}(\mathbf{X}_{0}^{n-1})$, with a corresponding secondary rate term.
• In the case of a Gaussian process with $\phi_n(\mathbf{x}_{0,n-1}) = \exp(\mathbf{x}^T_{0,n-1} \mathbf{C}_n^{-1} \mathbf{t}_{0,n-1})$, the weighted entropy rate satisfies $H^{\rm w}_{\phi_n}(f_n) = H(f_n) \exp(\frac{1}{2} \mathbf{t}^T_{0,n-1} \mathbf{C}_n^{-1} \mathbf{t}_{0,n-1})$, linking weighted and standard entropy.
• For a Gaussian process with $\phi_n(\mathbf{x}_{0,n-1}) = \exp(\frac{1}{2} \mathbf{x}^T_{0,n-1} \mathbf{A}_n \mathbf{x}_{0,n-1})$, the weighted entropy is $H^{\rm w}_{\phi_n}(f_n) = \frac{H(f_n) + \mathrm{tr}[(\mathbf{I}_n - \mathbf{A}_n \mathbf{C}_n)^{-1}] \log e}{2 [\det(\mathbf{I}_n - \mathbf{C}_n \mathbf{A}_n)]^{1/2}}$, providing an explicit formula.
• The ratio $\frac{H(f_n)}{a(n)} \to \alpha$ if and only if $\frac{H^{\rm w}_{\phi_n}(f_n)}{a(n)} \exp(-\frac{1}{2} \mathbf{t}^T_{0,n-1} \mathbf{C}_n^{-1} \mathbf{t}_{0,n-1}) \to \alpha$, showing equivalence between standard and weighted entropy rate convergence under specific WFs.
• For $\mathbf{t} = \mathbf{0}$, the weighted entropy rate $H^{\rm w}_{\phi_n}(f_n)$ is proportional to $H(f_n)$, with correction terms involving trace and determinant of $\mathbf{I}_n - \mathbf{C}_n \mathbf{A}_n$, enabling precise asymptotic analysis.