[Paper Review] Greedy-Merge Degrading has Optimal Power-Law

Assaf Kartowsky, Ido Tal|arXiv (Cornell University)|Mar 15, 2017

Error Correcting Code Techniques18 references3 citations

TL;DR

This paper establishes that the greedy-merge algorithm for channel degrading achieves optimal power-law scaling in mutual information loss, with the reduction bounded within a constant factor of the information-theoretic lower bound. It further shows that the same power-law applies to channel upgrading, proving that certain channels are inherently hard to upgrade, and presents an efficient algorithm for optimal binary-input channel upgrading with matching theoretical guarantees.

ABSTRACT

Consider a channel with a given input alphabet size and a given input distribution. Our aim is to degrade or upgrade it to a channel with at most L output letters. The paper contains four main results. The first result, from which the paper title is derived, deals with the so called "greedy-merge" algorithm. We derive an upper bound on the reduction in mutual information between input and output. This upper bound is within a constant factor of an algorithm-independent lower bound. Thus, we establish that greedy-merge is optimal in the power-law sense. The other main results deal with upgrading. The second result shows that a certain sequence of channels that was previously shown to be "hard" for degrading, displays the same hardness in the context of upgrading. That is, suppose we are given such a channel and a corresponding input distribution. If we upgrade (degrade) to a new channel with L output letters, we incur an increase (decrease) in mutual information between input and output. We show that a previously derived bound on the decrease in mutual information for the degrading case is also a lower bound on the increase for the upgrading case. The third result is an efficient algorithm for optimal upgrading, in the binary-input case. That is, we are given a channel and an input distribution. We must find an upgraded channel with L output letters, for which the increase in mutual information is minimal. We give a simple characterization of such a channel, which implies an efficient algorithm. The fourth result is an analog of the first result for the upgrading case, when the input is binary. That is, we first present a sub-optimal algorithm for the setting considered in the third result. By analyzing the algorithm, we show that the increase incurred in mutual information is within a constant factor of the lower bound derived in the second result.

Motivation & Objective

• To analyze the performance of the greedy-merge algorithm for channel degrading and prove its optimality in the power-law sense.
• To demonstrate that the same sequence of channels that are hard to degrade are also hard to upgrade, establishing symmetric information-theoretic limits.
• To develop an efficient algorithm for optimal channel upgrading in the binary-input case, minimizing mutual information increase.
• To extend the power-law analysis from degrading to upgrading, showing that a sub-optimal algorithm still achieves performance within a constant factor of the theoretical lower bound.

Proposed method

• Proposes the greedy-merge algorithm for degrading a channel with a given input distribution to one with at most $L$ output letters, minimizing mutual information loss.
• Derives an upper bound on the mutual information reduction using the greedy-merge algorithm, showing it scales as $O(L^{-2/(| cal{X}|-1)})$.
• Proves that this bound is tight up to a constant factor by showing a matching lower bound of $ Omega(L^{-2/(| cal{X}|-1)})$ for a specific sequence of channels.
• Analyzes the upgrading problem analogously, showing that the same power-law lower bound applies to the increase in mutual information.
• Introduces a characterization of the optimal upgraded channel for binary-input channels, enabling an efficient algorithm to compute it.
• Applies the same analytical framework to the upgrading case, proving that a sub-optimal greedy-split-like algorithm still achieves performance within a constant factor of the optimal bound.

Experimental results

Research questions

• RQ1 Is the greedy-merge algorithm optimal in terms of mutual information loss scaling with $L$?
• RQ2 Do channels that are hard to degrade also exhibit the same hardness in upgrading?
• RQ3 Can an optimal upgrading algorithm be constructed efficiently for binary-input channels?
• RQ4 Does a sub-optimal upgrading algorithm still achieve near-optimal performance in terms of mutual information increase?
• RQ5 What is the fundamental power-law scaling of the minimal achievable mutual information change in both degrading and upgrading?

Key findings

• The greedy-merge algorithm achieves a mutual information reduction of $O(L^{-2/(| cal{X}|-1)})$, which is optimal up to a constant factor.
• A lower bound of $ Omega(L^{-2/(| cal{X}|-1)})$ on the minimal mutual information loss exists for a specific sequence of channels, proving the power-law is tight.
• The same sequence of channels that is hard to degrade is also hard to upgrade, with $ Delta I_{ uparrow}^{ ast} = Omega(L^{-2/(| cal{X}|-1)})$.
• An efficient algorithm is presented for optimal upgrading of binary-input channels, based on a precise characterization of the optimal output distribution.
• For the upgrading case, a sub-optimal algorithm is shown to incur an increase in mutual information within a constant factor of the theoretical lower bound.
• The analysis confirms that the power-law exponent $-2/(| cal{X}|-1)$ is fundamental for both degrading and upgrading, regardless of input alphabet size $| cal{X}|$.