[Paper Review] Information as Distinctions: New Foundations for Information Theory

David Ellerman|arXiv (Cornell University)|Jan 23, 2013

Bayesian Modeling and Causal Inference28 references3 citations

TL;DR

This paper introduces a new foundation for information theory based on partition logic, where information is defined as distinctions between elements in a partition. It derives logical entropy as the normalized count of distinctions (dits), and shows how Shannon entropy emerges as a bit-based refinement of this notion via the dit-bit connection, unifying key concepts like mutual information and divergence through a dual logic framework.

ABSTRACT

The logical basis for information theory is the newly developed logic of partitions that is dual to the usual Boolean logic of subsets. The key concept is a "distinction" of a partition, an ordered pair of elements in distinct blocks of the partition. The logical concept of entropy based on partition logic is the normalized counting measure of the set of distinctions of a partition on a finite set--just as the usual logical notion of probability based on the Boolean logic of subsets is the normalized counting measure of the subsets (events). Thus logical entropy is a measure on the set of ordered pairs, and all the compound notions of entropy (join entropy, conditional entropy, and mutual information) arise in the usual way from the measure (e.g., the inclusion-exclusion principle)--just like the corresponding notions of probability. The usual Shannon entropy of a partition is developed by replacing the normalized count of distinctions (dits) by the average number of binary partitions (bits) necessary to make all the distinctions of the partition.

Motivation & Objective

• To establish a new foundation for information theory based on the duality between subset logic and partition logic.
• To define information as distinctions (dits) in a partition, formalizing logical entropy as the normalized count of such distinctions.
• To show how Shannon entropy arises naturally from logical entropy by replacing dit-counts with average bit requirements.
• To unify key information-theoretic concepts—mutual information, conditional entropy, divergence—under a single logical framework.
• To demonstrate that standard information-theoretic relationships (e.g., Venn diagram representations) emerge naturally from the logical structure of distinctions.

Proposed method

• Uses partition logic as the dual to Boolean subset logic, where partitions represent distinctions between elements.
• Defines logical entropy h(p) as the sum of p_i(1 - p_i) over outcomes, representing the probability of a random pair being in different blocks.
• Applies the dit-bit connection: logical entropy counts distinctions (dits), while Shannon entropy measures the average number of binary partitions (bits) needed to make them.
• Derives conditional entropy, mutual information, and divergence in both logical and Shannon forms using inclusion-exclusion and duality principles.
• Expresses logical divergence d(p||q) as half the sum of squared differences, which maps via the dit-bit connection to the symmetrized Kullback-Leibler divergence.
• Uses probability distributions and joint distributions to compare logical and Shannon entropies, showing identical structural relationships.

Experimental results

Research questions

• RQ1 How can information be fundamentally redefined not as uncertainty but as distinctions between elements?
• RQ2 What is the role of partition logic in providing a dual foundation to subset logic for information theory?
• RQ3 How does logical entropy, based on counting distinctions, relate to Shannon entropy, based on bit requirements?
• RQ4 Can standard information-theoretic measures like mutual information and conditional entropy be derived uniformly from a single logical framework?
• RQ5 What is the precise mathematical connection between the normalized count of distinctions (dits) and the average number of bits (bits) required to make them?

Key findings

• Logical entropy is defined as h(p) = Σ p_i(1 - p_i), representing the probability that a randomly selected pair of elements is distinguished by the partition.
• The dit-bit connection maps logical entropy (count of distinctions) to Shannon entropy (average number of bits), showing Shannon entropy as a refined, bit-based version of logical entropy.
• Mutual information in both frameworks satisfies the same structural identity: m(x,y) = h(x) + h(y) - h(x,y), mirroring the Venn diagram representation.
• Conditional logical entropy h(x|y) is defined as h(x,y) - h(y), analogous to H(x|y) = H(x,y) - H(y), preserving the same algebraic structure.
• Logical divergence d(p||q) = ½Σ(p_i - q_i)² maps via the dit-bit connection to the symmetrized Kullback-Leibler divergence D_s(p||q) = ½[D(p||q) + D(q||p)].
• The paper establishes that all standard information-theoretic identities (e.g., non-negativity of divergence, independence conditions) hold in both logical and Shannon forms, with identical structural relationships.