[论文解读] Concentration of Measure Inequalities in Information Theory, Communications and Coding (Second Edition)

Maxim Raginsky, Igal Sason|arXiv (Cornell University)|Dec 19, 2012

Wireless Communication Security Techniques参考文献 4被引用 14

一句话总结

此第二版全面综述了现代测度集中不等式，强调其在信息论中的基础及其在通信与编码中的应用。它将鞅方法与熵方法——特别是通过对数索博列夫不等式和运输成本不等式——统一为推导精确集中界限的框架，并提出关于编码理论、强逆定理以及好码经验分布的新结果。

ABSTRACT

During the last two decades, concentration inequalities have been the subject of exciting developments in various areas, including convex geometry, functional analysis, statistical physics, high-dimensional statistics, pure and applied probability theory, information theory, theoretical computer science, and learning theory. This monograph focuses on some of the key modern mathematical tools that are used for the derivation of concentration inequalities, on their links to information theory, and on their various applications to communications and coding. In addition to being a survey, this monograph also includes various new recent results derived by the authors. The first part of the monograph introduces classical concentration inequalities for martingales, as well as some recent refinements and extensions. The power and versatility of the martingale approach is exemplified in the context of codes defined on graphs and iterative decoding algorithms, as well as codes for wireless communication. The second part of the monograph introduces the entropy method, an information-theoretic technique for deriving concentration inequalities. The basic ingredients of the entropy method are discussed first in the context of logarithmic Sobolev inequalities, which underlie the so-called functional approach to concentration of measure, and then from a complementary information-theoretic viewpoint based on transportation-cost inequalities and probability in metric spaces. Some representative results on concentration for dependent random variables are briefly summarized, with emphasis on their connections to the entropy method. Finally, we discuss several applications of the entropy method to problems in communications and coding, including strong converses, empirical distributions of good channel codes, and an information-theoretic converse for concentration of measure.

研究动机与目标

• 提供对测度集中不等式的统一、深入的处理，重点关注其信息论基础。
• 弥合经典概率工具与编码理论和通信系统中现代应用之间的鸿沟。
• 呈现关于依赖随机变量集中性的新结果及其在编码与信息论中的影响。
• 通过强逆定理与信道码经验分布的应用，展示熵方法的威力。
• 利用信息论工具（如运输成本不等式与对数索博列夫不等式）扩展并改进现有测度集中不等式。

提出的方法

• 应用鞅方法推导依赖随机变量的集中不等式，尤其适用于图模型与迭代译码算法。
• 通过对数索博列夫不等式引入熵方法，作为测度集中性的函数式方法。
• 利用运输成本不等式与度量空间概率重新诠释集中性，以连接信息论中的分歧度量。
• 使用相对熵与费希尔信息量化高维与依赖设定下的偏差界限。
• 结合信息论工具与概率不等式，推导信息论中的强逆定理。
• 应用熵方法分析好信道码的经验分布，表明其围绕典型行为集中。

实验结果

研究问题

• RQ1 如何系统性地应用熵方法，在高维与依赖设定下推导集中不等式？
• RQ2 对数索博列夫不等式与信息论语境下测度集中性之间有何联系？
• RQ3 运输成本不等式与度量空间概率如何增强对编码与通信系统中集中性的分析？
• RQ4 熵方法在何种方式下可导出信息论中的强逆定理？
• RQ5 好信道码的经验分布如何围绕典型分布集中，这对码设计意味着什么？

主要发现

• 基于对数索博列夫不等式的熵方法，为推导精确集中不等式提供了强大且灵活的框架。
• 通过熵方法推导的集中不等式可导出信息论中的强逆定理，确立通信速率的根本极限。
• 好信道码的经验分布紧密集中在典型分布附近，验证了在编码定理中使用典型序列的合理性。
• 鞅方法有效捕捉了图码与迭代译码中的集中性，使无线通信系统中的性能分析成为可能。
• 运输成本不等式在信息论分歧度量与度量空间中集中性之间建立了稳健的联系。
• 关于依赖随机变量的新结果表明，熵方法可超越独立设定，使结构化编码与信号处理中的应用成为可能。