[Paper Review] Additive Entropies of Partitions

Adam Paszkiewicz, Tomasz Sobieszek|arXiv (Cornell University)|Feb 21, 2012

Statistical Mechanics and Entropy15 references3 citations

TL;DR

This paper characterizes additive partition entropies on nonatomic probability spaces by proving that any such entropy functional, continuous on finite partitions, decomposes uniquely into a Shannon-type entropy term and a measure-theoretic correction term. The key result establishes a canonical representation involving an absolutely continuous set function and a variance-like term, generalizing classical entropy axioms to partition-based information measures.

ABSTRACT

We provide, under minimal continuity assumptions, a description of extsl{additive partition entropies}. They are real functions $I$ on the set of finite partitions that are additive on stochastically independent partitions in a given probability space.

Motivation & Objective

• To characterize additive partition entropies—real functions on finite partitions that are additive over stochastically independent partitions—under continuity and structural assumptions.
• To extend classical entropy axioms (e.g., Shannon’s) from sequences of probabilities to partition-based information measures in a probability space framework.
• To identify the precise functional form of additive partition entropies that are uniformly continuous with respect to the L2 metric on partition measures.
• To provide a complete decomposition of such entropies into a base entropy term and a correction term involving a signed measure absolutely continuous with respect to the probability measure.
• To establish a canonical representation that unifies known entropies (e.g., Shannon, Hartley, Rényi) within a broader class of additive partition entropies.

Proposed method

• Define additive partition entropies as real-valued functions I on finite partitions of a nonatomic probability space (Ω, Σ, P), satisfying I(𝒜 ⊗ 𝒟) = I(𝒜) + I(𝒟) for independent partitions 𝒜 and 𝒟.
• Introduce the space 𝔸₂ of finite partitions equipped with the L2 metric ρ₂, and assume uniform continuity of I with respect to this metric.
• Use the Darboux property of nonatomic probability spaces to construct approximating sequences of partitions with arbitrarily small atom measures.
• Represent the entropy functional I as a sum of two components: Lₘ(𝒜) = ∫ H(𝒜) d𝑚, where 𝑚 is a countably additive signed measure absolutely continuous w.r.t. P, and βV(𝒜), where V is a variance-like functional.
• Establish uniqueness of the decomposition under the normalization 𝑚(Ω) = 0, ensuring the decomposition is canonical.
• Leverage known results on additive entropies depending only on atom measures (e.g., linear combinations of Shannon and uniform entropy) to derive the general form.

Experimental results

Research questions

• RQ1 What is the complete characterization of additive partition entropies that are continuous with respect to the L2 metric on partition measures?
• RQ2 How can additive partition entropies be decomposed into a base entropy term and a correction term involving a signed measure?
• RQ3 Which classical entropies (e.g., Shannon, Hartley, Rényi) arise as special cases within this generalized class of additive partition entropies?
• RQ4 What is the role of the Darboux property in nonatomic probability spaces for constructing continuous additive entropies?
• RQ5 Under what conditions is the decomposition of an additive partition entropy into Lₘ and βV unique?

Key findings

• Any additive partition entropy I that is continuous on the space of finite partitions (specifically, uniformly continuous w.r.t. ρ₂) admits a unique decomposition as I = Lₘ + βV, where Lₘ is an integral of the entropy function H(𝒜) with respect to a countably additive signed measure 𝑚 absolutely continuous w.r.t. P.
• The decomposition is unique when normalized by the condition 𝑚(Ω) = 0, ensuring no redundancy in the representation.
• The functional V(𝒜) = ∫[H(𝒜) − ∫H(𝒜) dP]² dP is a variance-like term that captures deviation from the expected entropy, and βV(𝒜) accounts for the part of entropy independent of atom measures.
• When I depends only on the measures of atoms, it must be a linear combination αLₚ + βV, where Lₚ is the Shannon entropy and V is the variance term.
• The result generalizes classical axiomatic characterizations of entropy (e.g., Fadeev’s or Shannon’s) to the setting of partitions in a probability space, extending them to a broader class of functionals.
• The paper establishes that the class of additive partition entropies is fully characterized by the pair (𝑚, β), with 𝑚 ∈ 𝒮(P) and β ∈ ℝ, under continuity and additivity constraints.