[Paper Review] Relative Information Loss - An Introduction

Bernhard C. Geiger, Gernot Kubin|arXiv (Cornell University)|Mar 30, 2012

Mathematical Analysis and Transform Methods14 references3 citations

TL;DR

This paper introduces relative information loss as a normalized measure to quantify information degradation in deterministic input-output systems, especially where absolute information loss is infinite. It establishes a link to Rényi’s information dimension and proves that quantizers incur 100% relative information loss, while deriving bounds on reconstruction error probability using the relative loss metric.

ABSTRACT

We introduce a relative variant of information loss to characterize the behavior of deterministic input-output systems. We show that the relative loss is closely related to Renyi's information dimension. We provide an upper bound for continuous input random variables and an exact result for a class of functions (comprising quantizers) with infinite absolute information loss. A connection between relative information loss and reconstruction error is investigated.

Motivation & Objective

• To address the limitation of absolute information loss being infinite for continuous inputs and deterministic mappings such as quantizers.
• To propose a normalized, relative measure of information loss that remains meaningful even when absolute loss diverges.
• To establish theoretical connections between relative information loss and Rényi’s information dimension.
• To derive upper bounds on reconstruction error probability using the relative loss metric.
• To demonstrate the applicability of the framework through examples like quantizers and center clippers.

Proposed method

• Defining relative information loss as the limit of the ratio of conditional entropy to marginal entropy over increasingly fine quantizations: $ l(\mathbf{X} \to \mathbf{Y}) = \lim_{n \to \infty} \frac{H(\hat{\mathbf{X}}_n | \mathbf{Y})}{H(\hat{\mathbf{X}}_n)} $, where $ \hat{\mathbf{X}}_n = \lfloor n\mathbf{X} \rfloor / n $.
• Using Rényi’s information dimension to characterize the scaling behavior of entropy in continuous distributions, linking it to the relative loss limit.
• Proving that for systems with infinite absolute information loss, such as quantizers, the relative loss converges to 1, indicating 100% information destruction.
• Deriving an upper bound on the probability of reconstruction error using the relative information loss: $ P_e \leq l(\mathbf{X} \to \mathbf{Y}) $, with equality in certain cases.
• Applying the framework to specific systems like scalar and vector quantizers and center clippers to validate theoretical results.

Experimental results

Research questions

• RQ1 How can information loss be meaningfully quantified in deterministic systems where absolute information loss is infinite?
• RQ2 What is the relationship between relative information loss and Rényi’s information dimension?
• RQ3 Can a bound on reconstruction error probability be derived using relative information loss?
• RQ4 To what extent does the relative loss capture the intrinsic information degradation of a system, especially in nonlinear or discontinuous mappings?
• RQ5 How does the relative loss behave in practical systems like quantizers and center clippers?

Key findings

• The relative information loss for quantizers is exactly 1, indicating that they destroy 100% of the available information, regardless of the number of quantization bins or design criterion.
• The relative information loss is mathematically linked to Rényi’s information dimension, providing a theoretical foundation for its interpretation.
• For continuous input random variables, an upper bound on the relative information loss is derived, which depends on the dimensionality and the structure of the input distribution.
• The probability of reconstruction error is bounded above by the relative information loss, and this bound is tight for systems like the center clipper where the function is bijective outside a measurable set.
• In the case of the center clipper, the relative information loss equals the probability mass of the clipping interval, $ P_X([-c,c]) $, and the reconstruction error probability matches this value, confirming the tightness of the bound.