[Paper Review] Prefix Codes for Power Laws with Countable Support

Michael B. Baer|arXiv (Cornell University)|Nov 15, 2006

Algorithms and Data Compression25 references4 citations

TL;DR

This paper introduces a family of prefix codes optimized for power-law distributions with countable support, particularly improving compression efficiency for distributions like the zeta distribution (ζ(2)) and the Gauss-Kuzmin distribution. By leveraging recursive structure and adaptive coding based on binary representations and complete binary codes, the proposed codes achieve near-optimal performance, with rigorous entropy-based bounds and demonstrated improvements over existing methods like Golomb and Rice codes.

ABSTRACT

In prefix coding over an infinite alphabet, methods that consider specific distributions generally consider those that decline more quickly than a power law (e.g., Golomb coding). Particular power-law distributions, however, model many random variables encountered in practice. For such random variables, compression performance is judged via estimates of expected bits per input symbol. This correspondence introduces a family of prefix codes with an eye towards near-optimal coding of known distributions. Compression performance is precisely estimated for well-known probability distributions using these codes and using previously known prefix codes. One application of these near-optimal codes is an improved representation of rational numbers.

Motivation & Objective

• To address the lack of efficient prefix coding schemes for infinite-alphabet sources with power-law distributions, which decline slowly (e.g., $p(i) \sim i^{-\alpha}$, $\alpha > 1$).
• To develop a family of prefix codes that achieve near-optimal compression performance for known power-law distributions, especially those arising in practical contexts like continued fractions and word frequency.
• To provide precise analytical estimates of expected codeword length for these codes, using entropy-based bounds and recursive code construction.
• To improve the representation of rational numbers via continued fractions by using more efficient coding of the integer coefficients in the expansion.
• To extend the applicability of prefix coding beyond fast-decaying distributions (e.g., geometric) to slowly decaying power laws, which are common in real-world data.

Proposed method

• Proposes a recursive code construction based on splitting the integer range into blocks of size $2^{g_i}$, where $g_i = \lfloor \lg i \rfloor$, and using a complete binary code for the offset within each block.
• Defines a code family $c_\omega(i)$ that uses a unary prefix of length $g_i$ followed by a binary suffix $b(i - 2^{g_i}, m_i)$, with $m_i = (2^{g_i} - (-1)^{g_i})/3$.
• Introduces a normalization-based bound using entropy to estimate the expected codeword length, with $\sum p(i)n(i) \geq H_x + (y_x + \lg(1 - \sigma_x))(1 - \sigma_x)$, where $\sigma_x = \sum_{i=1}^{x-1} p(i)$.
• Applies a top-down recursive algorithm (similar to Shannon-Fano) that groups symbols into blocks of power-of-two sizes to minimize imbalance, using the condition $|S(k_1^h, P) - 0.5|$ as the splitting criterion.
• Uses the Kraft inequality and monotonicity constraints to ensure prefix code validity and optimal structure, assuming monotonic length distribution $n(i) \geq n(i+1)$.
• Employs a parameterized code family $\text{Code } k$ with parameters $\alpha, \beta, \mu, \tau, \upsilon$ to model various code types (e.g., $\gamma$, $\omega$, $\text{EG}_k$), enabling systematic comparison.

Experimental results

Research questions

• RQ1 How can prefix codes be designed to achieve near-optimal compression for infinite-alphabet sources with power-law distributions, particularly those with $p(i) \sim i^{-\alpha}$, $\alpha > 1$?
• RQ2 What is the precise expected codeword length for these new codes across well-known distributions like the zeta distribution ($\zeta(2)$) and the Gauss-Kuzmin distribution?
• RQ3 Can the proposed codes outperform established codes such as Golomb, Rice, and Elias codes in terms of expected bits per symbol for power-law-distributed data?
• RQ4 How can entropy-based bounds be used to rigorously estimate the performance of recursive, adaptive prefix codes over infinite alphabets?
• RQ5 To what extent can these codes improve the compression of rational numbers represented via continued fractions, given that the coefficients follow a Gauss-Kuzmin distribution?

Key findings

• The proposed $\omega$-code achieves a lower expected codeword length than existing codes for the Gauss-Kuzmin distribution, which models continued fraction coefficients of rational numbers.
• For the zeta distribution with parameter $\zeta(2)$, the new codes significantly reduce expected codeword length compared to standard Golomb and Rice codes.
• The entropy-based bound $\sum p(i)n(i) \geq H_x + (y_x + \lg(1 - \sigma_x))(1 - \sigma_x)$ provides a tight and rigorous estimate of expected code length, enabling precise performance evaluation.
• The recursive code construction using blocks of size $2^{g_i}$ and complete binary codes for offsets achieves near-optimality across multiple power-law distributions, with performance close to the theoretical entropy limit.
• The algorithm for selecting block sizes based on minimizing $|S(k_1^h, P) - 0.5|$ results in more balanced groupings than previous methods, improving compression efficiency.
• The method enables efficient on-the-fly code generation and performance estimation, making it suitable for practical applications such as arithmetic coding and rational number compression.