[Paper Review] On the Capacity of Constrained Systems

Georg Böcherer, Valdemar C. da Rocha+1 more|ArXiv.org|Nov 5, 2009

Computability, Logic, AI Algorithms6 references3 citations

TL;DR

This paper generalizes the capacity of constrained systems beyond regular languages by using generating functions to define combinatorial capacity as the abscissa of convergence, proving that the maximum entropy rate of any input process is upper-bounded by this value. The framework enables a new method to establish equality between combinatorial capacity and maximum entropy rate by constructing a maxentropic input process, illustrated for the (j,k) run-length constraint with a closed-form capacity formula.

ABSTRACT

In the first chapter of Shannon's "A Mathematical Theory of Communication," it is shown that the maximum entropy rate of an input process of a constrained system is limited by the combinatorial capacity of the system. Shannon considers systems where the constraints define regular languages and uses results from matrix theory in his derivations. In this work, the regularity constraint is dropped. Using generating functions, it is shown that the maximum entropy rate of an input process is upper-bounded by the combinatorial capacity in general. The presented results also allow for a new approach to systems with regular constraints. As an example, the results are applied to binary sequences that fulfill the (j,k) run-length constraint and by using the proposed framework, a simple formula for the combinatorial capacity is given and a maxentropic input process is defined.

Motivation & Objective

• To extend the theory of constrained system capacity beyond regular languages and the 'not too dense' weight assumption.
• To define a generalized combinatorial capacity using generating functions that reduces to the classical definition under prior assumptions.
• To establish a new method for proving equality between combinatorial capacity and maximum entropy rate by constructing a maxentropic input process.
• To provide a simple, closed-form expression for the combinatorial capacity of the (j,k) run-length constraint.

Proposed method

• Represent constrained systems via generating functions with symbol weights as real positive values, using exponential generating functions with parameter s.
• Define the combinatorial capacity as the abscissa of convergence of the generating function, generalizing prior definitions.
• Apply known results from complex analysis to show that the abscissa of convergence equals the growth rate of constrained sequences.
• Construct an input process with i.i.d. symbols over a finite set of weighted blocks, assigning probabilities proportional to e^(-w(y)R) where R is the combinatorial capacity.
• Use the entropy rate formula involving limsup of H(Y₁,…,Yₗ)/E[w(cat(Y₁,…,Yₗ))], showing it equals R when the distribution is properly normalized.
• Leverage the upper bound on entropy rate (Theorem 2) to prove that a process achieving capacity R is maxentropic.

Experimental results

Research questions

• RQ1 Can the combinatorial capacity of a constrained system be defined and computed without assuming regularity or a 'not too dense' weight set?
• RQ2 Is the maximum entropy rate of input processes always upper-bounded by the abscissa of convergence of the system's generating function?
• RQ3 Can a maxentropic input process be explicitly constructed for systems where the entropy rate equals the combinatorial capacity?
• RQ4 Does the new framework allow for simpler derivations of capacity and input process design, especially for regular constraints like (j,k)?
• RQ5 Can the generating function approach replace matrix-theoretic methods in capacity analysis?

Key findings

• The combinatorial capacity of a constrained system is equal to the abscissa of convergence of its generating function, even when the weight set is dense or the constraints are non-regular.
• The maximum entropy rate of any input process is upper-bounded by the abscissa of convergence of the generating function, which is the generalized combinatorial capacity.
• For the (j,k) run-length constraint, a simple closed-form formula for the combinatorial capacity is derived using the new framework.
• A maxentropic input process is explicitly constructed for the (j,k) constraint by assigning i.i.d. symbols with probabilities e^(-w(y)R), where R is the capacity.
• The entropy rate of this constructed process equals the combinatorial capacity, proving it is maxentropic.
• The framework provides a new, general method to prove equality between maximum entropy rate and combinatorial capacity by construction, bypassing complex matrix-theoretic derivations.