[Paper Review] Combinatorial information distance

Joel Ratsaby|arXiv (Cornell University)|May 14, 2009

Algorithms and Data Compression16 references3 citations

TL;DR

This paper introduces a novel combinatorial information distance $ d(A,B) = \max\{\delta(A,B), \delta(B,A)\} $, where $ \delta(A,B) = \log_2(t(|B \cap \overline{A}| \cdot |A|)) $, to measure dissimilarity between binary strings of different lengths. The distance is based on set-theoretic information theory and satisfies the triangle inequality under certain conditions, offering a universal, domain-independent measure rooted in Kolmogorov complexity and Lempel-Ziv principles.

ABSTRACT

Let $|A|$ denote the cardinality of a finite set $A$. For any real number $x$ define $t(x)=x$ if $x\geq1$ and 1 otherwise. For any finite sets $A,B$ let $δ(A,B)$ $=$ $\log_{2}(t(|B\cap\bar{A}||A|))$. We define {This appears as Technical Report # arXiv:0905.2386v4. A shorter version appears in the {Proc. of Mini-Conference on Applied Theoretical Computer Science (MATCOS-10)}, Slovenia, Oct. 13-14, 2010.} a new cobinatorial distance $d(A,B)$ $=$ $\max\{δ(A,B),δ(B,A)\} $ which may be applied to measure the distance between binary strings of different lengths. The distance is based on a classical combinatorial notion of information introduced by Kolmogorov.

Motivation & Objective

• To develop a universal distance function between binary strings that does not require domain-specific knowledge or prior assumptions about the data.
• To extend the notion of information complexity from strings to sets of substrings, leveraging combinatorial entropy and set projections.
• To define a metric that is invariant to string length differences and captures structural dissimilarity through set-theoretic information measures.
• To establish theoretical properties such as the triangle inequality for the proposed distance under non-containment conditions.
• To provide a foundation for applications in pattern recognition, bioinformatics, and information retrieval using a minimal, universal framework.

Proposed method

• Define $ \delta(A,B) = \log_2(t(|B \cap \overline{A}| \cdot |A|)) $, where $ t(x) = \max(x,1) $, to measure information distance between finite sets $ A $ and $ B $.
• Construct the full distance as $ d(A,B) = \max\{\delta(A,B), \delta(B,A)\} $, ensuring symmetry and robustness to set size imbalances.
• Model a binary string $ x $ as a set $ M(x) \subseteq \mathbb{Y} $ of substrings (e.g., k-words or n-grams), using mappings $ M $ that eliminate redundancy.
• Apply the distance $ d(M(x), M(x')) $ to compare strings via their set representations, enabling comparison of strings of unequal length.
• Use combinatorial entropy and projection-based information measures to define information content and conditional information between sets.
• Prove that $ d $ satisfies the triangle inequality when no set is strictly contained in another, establishing it as a semi-metric on the space of strings.

Experimental results

Research questions

• RQ1 Can a universal, domain-independent distance be defined between binary strings based solely on combinatorial information theory?
• RQ2 How can information content and dissimilarity be measured between sets of substrings derived from strings of different lengths?
• RQ3 Does the proposed distance function satisfy fundamental metric properties such as symmetry and the triangle inequality?
• RQ4 What is the relationship between this combinatorial distance and existing complexity measures like Lempel-Ziv complexity?
• RQ5 Can the distance be normalized while preserving its metric and information-theoretic properties?

Key findings

• The proposed distance $ d(A,B) = \max\{\delta(A,B), \delta(B,A)\} $ is symmetric and satisfies the triangle inequality when no set is strictly contained in another.
• The function $ \delta(A,B) $ is defined using the cardinality of the intersection of $ B $ with the complement of $ A $, scaled by $ |A| $, and capped at 1 via the $ t(x) $ function.
• The distance is invariant to string length differences and captures structural dissimilarity through set-theoretic information content.
• The information set-distance $ d_M(x,x') = d(M(x), M(x')) $ is a semi-metric on the space of finite binary strings, provided the sets $ M(x), M(x'), M(x'') $ are not strictly nested.
• The method generalizes Lempel-Ziv complexity by treating strings as sets of substrings and measuring information distance via set differences.
• The approach provides a universal framework for string comparison without requiring prior knowledge of the data domain, suitable for applications in bioinformatics and information retrieval.