[Paper Review] Divergence, Entropy, Information: An Opinionated Introduction to Information Theory

Philip S. Chodrow|arXiv (Cornell University)|Aug 24, 2017

Computability, Logic, AI Algorithms9 references3 citations

TL;DR

This paper presents an intuitive, concept-first introduction to information theory centered on the Kullback-Leibler divergence as the foundational concept, deriving entropy and mutual information from it. It emphasizes that information-theoretic quantities make intuitive ideas about learning, prediction, and uncertainty precise, with key results like the Data Processing Inequality illustrating how information degrades through processing.

ABSTRACT

Information theory is a mathematical theory of learning with deep connections with topics as diverse as artificial intelligence, statistical physics, and biological evolution. Many primers on information theory paint a broad picture with relatively little mathematical sophistication, while many others develop specific application areas in detail. In contrast, these informal notes aim to outline some elements of the information-theoretic "way of thinking," by cutting a rapid and interesting path through some of the theory's foundational concepts and results. They are aimed at practicing systems scientists who are interested in exploring potential connections between information theory and their own fields. The main mathematical prerequisite for the notes is comfort with elementary probability, including sample spaces, conditioning, and expectations. We take the Kullback-Leibler divergence as our most basic concept, and then proceed to develop the entropy and mutual information. We discuss some of the main results, including the Chernoff bounds as a characterization of the divergence; Gibbs' Theorem; and the Data Processing Inequality. A recurring theme is that the definitions of information theory support natural theorems that sound ``obvious'' when translated into English. More pithily, ``information theory makes common sense precise.'' Since the focus of the notes is not primarily on technical details, proofs are provided only where the relevant techniques are illustrative of broader themes. Otherwise, proofs and intriguing tangents are referenced in liberally-sprinkled footnotes. The notes close with a highly nonexhaustive list of references to resources and other perspectives on the field.

Motivation & Objective

• To reframe information theory by starting with the Kullback-Leibler divergence rather than entropy, to avoid issues with differential entropy in continuous settings.
• To demonstrate how information-theoretic concepts provide a representation-independent, fundamental measure of uncertainty and learning potential.
• To show that core results like the Data Processing Inequality formalize intuitive ideas—such as information shrinking through processing—making them mathematically rigorous.
• To connect information theory to broader domains like statistics, physics, and biology by highlighting its role in quantifying learnability and predictability.
• To guide systems scientists in applying information-theoretic thinking to their own fields by emphasizing conceptual clarity over technical derivation.

Proposed method

• Uses the Kullback-Leibler divergence as the primary object of study, treating it as the fundamental measure of difference between probability distributions.
• Derives entropy and mutual information from the divergence, showing their natural emergence rather than treating them as axiomatic starting points.
• Applies the divergence to characterize the Chernoff bounds, linking it to large deviations and concentration of measure.
• Employs Gibbs’ inequality to establish the non-negativity of divergence and to prove the Data Processing Inequality.
• Uses conditional entropy and the chain rule of mutual information to prove that information cannot increase through data processing.
• Leverages coordinate-invariance of divergence (unlike differential entropy) to justify its foundational role in both discrete and continuous probability spaces.

Experimental results

Research questions

• RQ1 Why is the Kullback-Leibler divergence a more suitable foundational concept than entropy in information theory?
• RQ2 How does the representation-invariance of divergence resolve the conceptual problems of differential entropy in continuous distributions?
• RQ3 In what ways does the Data Processing Inequality formalize the intuitive idea that information degrades through processing?
• RQ4 How can information-theoretic concepts like mutual information and entropy be used to quantify learning and predictability in complex systems?
• RQ5 What are the deep connections between information theory, statistical inference, and the second law of thermodynamics?

Key findings

• The Kullback-Leibler divergence is well-defined and non-negative for both discrete and continuous distributions, unlike differential entropy, which can be negative and is not invariant under smooth reparameterizations.
• The Data Processing Inequality proves that mutual information cannot increase when one variable is processed through a Markov chain, formalizing the idea that information shrinks over time.
• The inequality $ I(X,Z) \leq I(X,Y) $ holds when $ Z = g(Y) $, showing that processing reduces information, with equality only if the transformation is sufficient for full information recovery.
• The divergence-based approach allows for a unified treatment of discrete and continuous systems, supporting a more robust and general theory of information.
• Information-theoretic quantities such as entropy and mutual information provide a fundamental, representation-independent measure of uncertainty and dependence.
• The paper establishes a conceptual parallel between the Data Processing Inequality and the Second Law of Thermodynamics, both describing a natural tendency toward loss of order or information over time.