[Paper Review] Information Theory and Statistical Physics - Lecture Notes

Neri Merhav|arXiv (Cornell University)|Jun 8, 2010

Wireless Communication Security Techniques38 references3 citations

TL;DR

This lecture notes document explores the deep analogies between information theory and statistical physics, emphasizing shared concepts like entropy, partition functions, and large deviations. It presents key tools—such as the replica method, Gibbs sampling, and Monte Carlo techniques—from statistical physics to analyze information-theoretic problems, particularly in random coding and channel capacity, with a focus on the Random Energy Model (REM) and its implications for coding theory and phase transitions.

ABSTRACT

This document consists of lecture notes for a graduate course, which focuses on the relations between Information Theory and Statistical Physics. The course is aimed at EE graduate students in the area of Communications and Information Theory, as well as to graduate students in Physics who have basic background in Information Theory. Strong emphasis is given to the analogy and parallelism between Information Theory and Statistical Physics, as well as to the insights, the analysis tools and techniques that can be borrowed from Statistical Physics and `imported' to certain problem areas in Information Theory. This is a research trend that has been very active in the last few decades, and the hope is that by exposing the student to the meeting points between these two disciplines, we will enhance his/her background and perspective to carry out research in the field. A short outline of the course is as follows: Introduction; Elementary Statistical Physics and its Relation to Information Theory; Analysis Tools in Statistical Physics; Systems of Interacting Particles and Phase Transitions; The Random Energy Model (REM) and Random Channel Coding; Additional Topics (optional).

Motivation & Objective

• To establish a conceptual and analytical bridge between information theory and statistical physics for graduate students in electrical engineering and physics.
• To demonstrate how techniques from statistical physics—such as partition functions, phase transitions, and Monte Carlo sampling—can be applied to solve problems in information theory.
• To explore the Random Energy Model (REM) as a framework for analyzing random coding exponents and channel capacity.
• To investigate the role of entropy, free energy, and large deviations in both fields, particularly through the lens of the maximum entropy principle and Gibbs distributions.
• To provide a foundation for advanced research in areas such as joint source-channel coding, spin glass models, and information measures in complex systems.

Proposed method

• Uses the partition function $ Z(\beta) = \sum_{\mathbf{s}} e^{-\beta \mathcal{E}(\mathbf{s})} $ as a central tool to connect statistical mechanics and information theory.
• Applies the Laplace and saddle point methods to approximate partition functions and derive asymptotic results in information-theoretic settings.
• Employs the replica method to analyze quenched averages in disordered systems, particularly in the context of random coding and the REM.
• Utilizes Markov chain Monte Carlo (MCMC) methods, including the Metropolis and heat bath (Glauber) algorithms, to sample from Boltzmann–Gibbs distributions.
• Implements the Metropolis algorithm via acceptance probability $ A_{rs} = \min\left(1, e^{-\beta(E_s - E_r)}\right) $ to ensure detailed balance and convergence to equilibrium.
• Generalizes the heat bath algorithm to sample from arbitrary distributions on $ \mathcal{X}^n $ by updating one variable at a time using conditional probabilities.

Experimental results

Research questions

• RQ1 How can the formalism of statistical physics, particularly the partition function and free energy, be used to analyze information-theoretic quantities like channel capacity and coding exponents?
• RQ2 What is the role of the Random Energy Model (REM) in modeling random code ensembles and understanding the behavior of error exponents in communication systems?
• RQ3 How do phase transitions in spin glass models relate to information-theoretic limits such as the rate-distortion function or joint source-channel coding?
• RQ4 In what ways can Monte Carlo methods like Metropolis and heat bath sampling be adapted to simulate information measures and their dynamics?
• RQ5 How does the detailed balance condition ensure convergence to the Gibbs-Boltzmann distribution in MCMC simulations of physical and information systems?

Key findings

• The partition function $ Z(\beta) $ serves as a unifying object linking statistical mechanics and information theory, with the free energy derived from $ \mathcal{F} = -\frac{1}{\beta} \log Z(\beta) $.
• The Metropolis algorithm ensures detailed balance via $ \frac{A_{rs}}{A_{sr}} = e^{-\beta(E_s - E_r)} $, enabling efficient sampling from the Gibbs distribution without computing $ Z(\beta) $.
• The heat bath algorithm generalizes to any discrete configuration space $ \mathcal{X}^n $, updating one variable at a time using $ P(X^i = x | \mathbf{x}^{\sim i}) $, ensuring convergence to the target distribution.
• The replica method enables the analysis of quenched averages in disordered systems, such as those arising in random coding, and is instrumental in deriving random coding exponents.
• The Random Energy Model (REM) provides a solvable framework for studying phase transitions in coding theory, with a phase transition at $ \beta_c = \sqrt{2 \log 2} $, and relates to the performance of random codes.
• The maximum entropy principle, rooted in the second law of thermodynamics, justifies the use of Gaussian and other distributions in signal and information processing under moment constraints.