[Paper Review] Optimal Document Exchange and New Codes for Small Number of Insertions and Deletions.

Bernhard Haeupler|arXiv (Cornell University)|Apr 10, 2018

Algorithms and Data Compression13 citations

TL;DR

This paper presents a communication-optimal document exchange protocol that computes an O(k log(n/k))-bit summary for any n-bit file, enabling reconstruction from a file with up to k edit operations (insertions/deletions). It achieves information-theoretically optimal summary size for all k, resolves an open question by Orlitsky, and yields near-optimal error-correcting codes with O(k log²(n/k)) redundancy for k insertions/deletions.

ABSTRACT

This paper gives a communication-optimal document exchange protocol and an efficient near optimal derandomization. This also implies drastically improved error correcting codes for small number of adversarial insertions and deletions. For any $n$ and $k < n$ our randomized hashing scheme takes any $n$-bit file $F$ and computes a $O(k \log \frac{n}{k})$-bit summary from which one can reconstruct $F$ given a related file $F'$ with edit distance $ED(F,F') \leq k$. The size of our summary is information-theoretically order optimal for all values of $k$, positively answering a question of Orlitsky. It also is the first non-trivial solution when a small constant fraction of symbols have been edited, producing an optimal summary of size $O(H(\delta)n)$ for $k=\delta n$. This concludes a long series of better-and-better protocols which produce larger summaries for sub-linear values of $k$. In particular, the recent break-through of [Belazzougi, Zhang; STOC'16] assumes that $k < n^\epsilon$ and produces a summary of size $O(k\log^2 k + k\log n)$. We also give an efficient derandomization with near optimal summary size $O(k \log^2 \frac{n}{k})$ improving, for every $k$, over a deterministic $O(k^2 + k \log^2 n)$ document exchange scheme by Belazzougi. This directly leads to near optimal systematic error correcting codes which efficiently can recover from any $k$ insertions and deletions while having $\Theta(k \log^2 \frac{n}{k})$ bits of redundancy. For the setting of $k=n\epsilon$ this $O(\epsilon \log^2 \frac{1}{\epsilon} \cdot n)$-bit redundancy is near optimal and a quadratic improvement over the binary codes of Guruswami and Li and Haeupler, Shahrasbi and Vitercik which have redundancy $\Theta\left(\sqrt{\epsilon} \log^{O(1)} \frac{1}{\epsilon} \cdot n ight)$.

Motivation & Objective

• To design a communication-optimal document exchange protocol for files with up to k edit operations.
• To resolve Orlitsky's open question on the information-theoretic optimality of summary size for small k.
• To develop an efficient derandomization of the randomized hashing scheme with near-optimal summary size.
• To construct near-optimal systematic error-correcting codes resilient to k insertions and deletions.
• To improve upon prior deterministic schemes with redundancy O(k² + k log² n) by achieving O(k log²(n/k)) redundancy.

Proposed method

• Uses a randomized hashing scheme to generate a summary of size O(k log(n/k)) from an n-bit file F.
• Employs a novel derandomization technique to achieve near-optimal summary size O(k log²(n/k)) without relying on randomness.
• Leverages properties of edit distance and combinatorial hashing to ensure reconstruction from any file F' with ED(F, F') ≤ k.
• Applies the document exchange protocol to construct systematic error-correcting codes with redundancy O(k log²(n/k)).
• Uses the structure of edit distance and information-theoretic bounds to prove optimality of the summary size.
• Introduces a new framework for handling small constant fractions of edits (k = δn) with O(H(δ)n) summary size.

Experimental results

Research questions

• RQ1 Can a document exchange protocol achieve information-theoretically optimal summary size for all values of k, including sub-linear and constant fractions of n?
• RQ2 Is there a derandomization of the randomized hashing scheme that maintains near-optimal summary size?
• RQ3 Can the document exchange protocol be used to construct near-optimal error-correcting codes for insertions and deletions?
• RQ4 How does the redundancy of the resulting codes compare to prior deterministic constructions for k = δn?
• RQ5 What is the minimal redundancy required to correct k insertions and deletions, and can it be achieved efficiently?

Key findings

• The proposed protocol achieves an O(k log(n/k))-bit summary size, which is information-theoretically optimal for all k < n.
• For k = δn with constant δ, the summary size is O(H(δ)n), matching the information-theoretic lower bound.
• The derandomized scheme achieves a summary size of O(k log²(n/k)), improving over the prior deterministic O(k² + k log² n) scheme.
• The resulting error-correcting codes have O(k log²(n/k)) redundancy, which is near-optimal for k = δn and represents a quadratic improvement over prior binary codes.
• The protocol resolves Orlitsky’s open question on the optimality of summary size for small edit distances.
• The scheme is the first non-trivial solution for constant fractions of edits (k = δn) with optimal summary size.