[Paper Review] Source Coding, Large Deviations, and Approximate Pattern Matching

Amir Dembo, Ioannis Kontoyiannis|ArXiv.org|Mar 2, 2001

Algorithms and Data Compression79 references6 citations

TL;DR

This paper develops a lossy version of the Asymptotic Equipartition Property (AEP) using large deviations theory to unify and generalize rate-distortion theory and pattern-matching algorithms for lossy data compression. It establishes a generalized AEP for i.i.d. and stationary processes, proves second-order coding theorems, and characterizes waiting times, match-lengths, and sphere-covering exponents, providing a rigorous foundation for analyzing Lempel-Ziv-type lossy compression schemes and mismatched codebooks with precise asymptotic behavior.

ABSTRACT

We present a development of parts of rate-distortion theory and pattern- matching algorithms for lossy data compression, centered around a lossy version of the Asymptotic Equipartition Property (AEP). This treatment closely parallels the corresponding development in lossless compression, a point of view that was advanced in an important paper of Wyner and Ziv in 1989. In the lossless case we review how the AEP underlies the analysis of the Lempel-Ziv algorithm by viewing it as a random code and reducing it to the idealized Shannon code. This also provides information about the redundancy of the Lempel-Ziv algorithm and about the asymptotic behavior of several relevant quantities. In the lossy case we give various versions of the statement of the generalized AEP and we outline the general methodology of its proof via large deviations. Its relationship with Barron's generalized AEP is also discussed. The lossy AEP is applied to: (i) prove strengthened versions of Shannon's source coding theorem and universal coding theorems; (ii) characterize the performance of mismatched codebooks; (iii) analyze the performance of pattern- matching algorithms for lossy compression; (iv) determine the first order asymptotics of waiting times (with distortion) between stationary processes; (v) characterize the best achievable rate of weighted codebooks as an optimal sphere-covering exponent. We then present a refinement to the lossy AEP and use it to: (i) prove second order coding theorems; (ii) characterize which sources are easier to compress; (iii) determine the second order asymptotics of waiting times; (iv) determine the precise asymptotic behavior of longest match-lengths. Extensions to random fields are also given.

Motivation & Objective

• To extend the Asymptotic Equipartition Property (AEP) to lossy compression by formulating a generalized AEP under large deviations.
• To provide a theoretical foundation for analyzing lossy pattern-matching algorithms, including Lempel-Ziv schemes, via random coding and typical set arguments.
• To derive second-order asymptotics for lossy source coding, waiting times, and match-lengths, refining first-order results.
• To characterize the performance of mismatched codebooks and weighted codebooks in terms of optimal sphere-covering exponents.
• To extend the framework to random fields, establishing first- and second-order results for spatial processes.

Proposed method

• Formulates a generalized AEP for lossy compression using large deviations, proving convergence of log-probability densities to entropy distortion rate.
• Applies the generalized AEP to random codes and mismatched codebooks, deriving bounds on error probability and redundancy.
• Uses Borel-Cantelli lemma and variance bounds on indicator variables to analyze waiting times and match-lengths in stationary processes.
• Derives second-order coding theorems by refining the generalized AEP with higher-order asymptotic expansions.
• Introduces a sphere-covering exponent formulation to characterize optimal codebook design under weighted distortion.
• Extends results to random fields using spatial mixing conditions and lattice-based sampling to generalize waiting time and match-length analysis.

Experimental results

Research questions

• RQ1 How can the Asymptotic Equipartition Property be generalized to lossy compression under distortion constraints?
• RQ2 What is the second-order asymptotic behavior of lossy source coding rates, and how does it refine Shannon's direct and converse theorems?
• RQ3 How do waiting times between stationary processes scale with distortion, and what is their connection to pattern-matching algorithms?
• RQ4 What is the precise asymptotic behavior of longest match-lengths in lossy compression schemes like Lempel-Ziv?
• RQ5 How do mismatched codebooks affect compression performance, and what is the optimal sphere-covering exponent for weighted codebooks?

Key findings

• The generalized AEP holds for i.i.d. and stationary processes, with convergence in probability of the log-probability density to the rate-distortion function.
• Second-order lossy source coding theorems are established, showing that redundancy scales as $\Theta(\sqrt{n})$ for i.i.d. sources.
• Waiting times between stationary processes with distortion $D$ satisfy $\log W_n \sim \log n$ almost surely, with precise asymptotic bounds derived via Borel-Cantelli.
• Longest match-lengths between stationary processes scale as $W_n \sim n^{1/d}$ in $d$-dimensional random fields, with tight bounds on the logarithmic growth.
• The optimal sphere-covering exponent for weighted codebooks is characterized as the infimum of a variational problem over distortion measures.
• For mismatched codebooks, the paper derives a converse bound on achievable rate that depends on the divergence between true and mismatched distributions.