[Paper Review] Introduction to Arithmetic Coding -- Theory and Practice

Amir Said|arXiv (Cornell University)|Feb 2, 2023

Algorithms and Data Compression50 references107 citations

TL;DR

This chapter introduces arithmetic coding, covering its theory, optimality, and practical implementation aspects including fixed-precision arithmetic, adaptive coding, and complexity considerations.

ABSTRACT

This introduction to arithmetic coding is divided in two parts. The first explains how and why arithmetic coding works. We start presenting it in very general terms, so that its simplicity is not lost under layers of implementation details. Next, we show some of its basic properties, which are later used in the computational techniques required for a practical implementation. In the second part, we cover the practical implementation aspects, including arithmetic operations with low precision, the subdivision of coding and modeling, and the realization of adaptive encoders. We also analyze the arithmetic coding computational complexity, and techniques to reduce it.

Motivation & Objective

• Explain why arithmetic coding is effective for entropy coding and its applicability to diverse data sources.
• Present the basic arithmetic coding model with intervals and code values, highlighting optimality and properties.
• Show practical implementation aspects, including fixed-precision arithmetic, adaptive coding, and synchronization requirements.
• Demonstrate how to separate coding from source modeling and analyze computational complexity.

Proposed method

• Define data source, probabilities, and cumulative distributions to establish the interval-based encoding framework.
• Describe the encoding process as nested interval updates Phi_k(S) with base b_k and length l_k according to p(s_k) and c(s_k).
• Describe the decoding process using normalized code values to recover symbol sequence via c(s) and p(s) based on observed v.
• Discuss fixed-precision arithmetic and implementation considerations such as carries, renormalization, and output efficiency.
• Explain adaptive coding by updating probability distributions and maintaining synchronization between encoder and decoder (and the importance of exact same rules).
• Analyze arithmetic coding complexity including interval renormalization, symbol search, and cumulative distribution estimation.

Experimental results

Research questions

• RQ1 How can arithmetic coding achieve optimal compression for i.i.d. sources and near-optimal performance for adaptive or non-i.i.d. sources?
• RQ2 What are the practical considerations and techniques to implement arithmetic coding with fixed precision and low complexity while preserving correctness?
• RQ3 How does the separation of coding and source modeling affect system design and adaptability in real-world compression applications?
• RQ4 What is the impact of using dynamic (time-varying) sources on encoder/decoder synchronization and correctness?

Key findings

• Arithmetic coding achieves asymptotically optimal compression, with the average bits per symbol approaching the source entropy H(Ω).
• Code values are mapped to the interval [0,1) with a uniform distribution property when coding is optimal, enabling efficient representation and decoding.
• There is a clear separation between coding and source modeling, enabling independent development of probability models and the coding engine.
• Encoder/decoder synchronization is essential for correctness, especially under adaptive and dynamic modeling, and must share identical rules and precision.
• Interval rescaling and fixed-precision strategies enable practical implementations with acceptable complexity and accuracy.