[Paper Review] Empirical distribution of good channel codes with non-vanishing error probability (extended version)

Yury Polyanskiy, Sergio Verdú|arXiv (Cornell University)|Aug 31, 2013

Wireless Communication Security Techniques24 references3 citations

TL;DR

This paper establishes that for good channel codes with non-vanishing error probability, the empirical output distribution closely approximates the capacity-achieving output distribution in a strong sense. Using concentration of measure and isoperimetry, it shows that regular functions of channel outputs are essentially non-random and independent of the specific code, and that quadratic and higher-order moments of codewords in AWGN codes converge to those of i.i.d. Gaussian codewords.

ABSTRACT

This paper studies several properties of channel codes that approach the fundamental limits of a given (discrete or Gaussian) memoryless channel with a non-vanishing probability of error. The output distribution induced by an $ε$-capacity-achieving code is shown to be close in a strong sense to the capacity achieving output distribution. Relying on the concentration of measure (isoperimetry) property enjoyed by the latter, it is shown that regular (Lipschitz) functions of channel outputs can be precisely estimated and turn out to be essentially non-random and independent of the actual code. It is also shown that the output distribution of a good code and the capacity achieving one cannot be distinguished with exponential reliability. The random process produced at the output of the channel is shown to satisfy the asymptotic equipartition property. Using related methods it is shown that quadratic forms and sums of $q$-th powers when evaluated at codewords of good AWGN codes approach the values obtained from a randomly generated Gaussian codeword.

Motivation & Objective

• To characterize the empirical output distribution of good channel codes when the error probability does not vanish.
• To extend prior results on codebook distribution convergence (which required vanishing error) to the non-vanishing error regime.
• To show that regular functions of channel outputs are tightly concentrated and effectively independent of the code design.
• To establish that the output distribution of a good code cannot be distinguished from the capacity-achieving distribution with exponential reliability.
• To demonstrate that quadratic forms and q-th power sums over codewords of good AWGN codes converge to those of i.i.d. Gaussian codewords.

Proposed method

• Leverages the concentration of measure (isoperimetry) property of the capacity-achieving output distribution to analyze the behavior of regular (Lipschitz) functions of channel outputs.
• Uses relative entropy bounds to quantify the distance between the output distribution of a good code and the capacity-achieving distribution.
• Applies the Gaussian isoperimetric inequality to derive tail bounds on the ℓ4 and ℓ2 norms of channel outputs under the capacity-achieving measure.
• Employs the β-divergence and its lower bounds to compare the output distribution of a code with the capacity-achieving distribution, using the fact that good codes achieve rates within O(√n) of capacity.
• Derives lower bounds on the expectation of ℓ4 norms of codewords via a modified version of Proposition 11, incorporating a deviation term from the typical set.
• Uses the triangle inequality and Minkowski sum properties to combine concentration bounds on different norms, leading to exponential tail bounds on the output distribution.

Experimental results

Research questions

• RQ1 How does the output distribution of a good channel code with non-vanishing error probability compare to the capacity-achieving output distribution?
• RQ2 Can regular functions of channel outputs be approximated independently of the specific code when the error probability is bounded away from zero?
• RQ3 To what extent can the output distribution of a good code be distinguished from the capacity-achieving distribution with exponential reliability?
• RQ4 Do quadratic and higher-order moments of codewords in AWGN codes converge to those of i.i.d. Gaussian codewords under the same rate and error regime?
• RQ5 What is the relationship between the ℓ4 norm of codewords and the power constraint in codes achieving rate within O(√n) of capacity?

Key findings

• The normalized relative entropy between the output distribution of an ϵ-capacity-achieving code and the capacity-achieving output distribution is upper bounded by the difference between capacity and code rate.
• The output distribution of a good code and the capacity-achieving distribution cannot be distinguished with exponential reliability, as shown by the β-divergence bound decaying as exp(−b₁δ²√n).
• The distribution of regular functions of channel outputs is tightly concentrated around their expectation under the capacity-achieving measure, implying they are essentially non-random and code-independent.
• For AWGN codes achieving rate within O(√n) of capacity, the expectation of the sum of fourth powers of outputs converges to that of i.i.d. Gaussian codewords, with a deviation term of order n^{1/4}.
• The ℓ4 norm of the output vector is tightly concentrated around 6^{1/4}√(1+P)n^{1/4}, with tail probability bounded by exp(−b₁δ²√n), demonstrating strong concentration.
• The method fails to bound ℓ6 norms due to the inability to control ℓ4 norms of codewords under O(√n)-rate codes, as shown by counterexamples.