[Paper Review] On an Application of Relative Entropy

Dmitry V. Khmelev, William J. Teahan|arXiv (Cornell University)|May 24, 2002

Machine Learning in Bioinformatics2 references3 citations

TL;DR

This paper argues that n-th order Markov chain models outperform Lempel-Ziv (LZ) compression-based methods in authorship attribution and linguistic analysis. Using empirical comparisons on 82 authors, it demonstrates that Markov chains achieve 69 correct classifications out of 82 test texts—surpassing LZ-based methods like gzip (50 correct) and showing superior speed and reliability.

ABSTRACT

We describe general approach to classification of character sequences (texts, DNA) using relative entropy estimated by off-the-shelf compression and Markov Chains and find them precise enough. We also notice that the method for estimating relative entropy described in the paper cond-mat/0108530 "Language Trees..." by D. Benedetto et al. was considered earlier and was found to be easily surpassed by the simple and computationally effective first order Markov Chain approach.

Motivation & Objective

• To challenge the claim that Lempel-Ziv compression is superior for linguistic analysis and authorship attribution.
• To demonstrate that Markov chain models provide more accurate and efficient classification than LZ-based methods in natural language processing.
• To expose flaws in the relative entropy and distance definitions used in prior work, including negative values and reliance on a-priori alphabet knowledge.
• To show that LZ-based methods are ill-suited for DNA analysis due to lack of biological substitution matrix integration.
• To advocate for Markov chains as the preferred method in entropy-based linguistic classification due to robustness and performance.

Proposed method

• The study compares the classification accuracy of 15 compression programs and a first-order Markov chain model on 82 authors’ texts (≥10⁵ characters each).
• The Markov chain model estimates n-th order conditional probabilities of characters based on preceding n characters, modeling text structure via statistical dependencies.
• Relative entropy is computed using standard definitions from Shannon (1948) and Yaglom & Yaglom (1983), ensuring non-negative values and statistical consistency.
• The authors re-implement and test the language tree method from Benedetto et al. (2002), identifying flaws such as negative entropy values and alphabet bias.
• Performance is evaluated via exact author identification rate across held-out test texts, with speed measured in processing time per document.
• The method avoids reliance on LZ compression by using probabilistic modeling of character transitions, which better captures linguistic structure.

Experimental results

Research questions

• RQ1 Does the Lempel-Ziv compression approach outperform Markov chain models in authorship attribution tasks?
• RQ2 Can the relative entropy measure used in prior work yield negative values, undermining its validity as a distance metric?
• RQ3 To what extent does the inclusion of Russian in language trees affect clustering results, and does this indicate reliance on a-priori alphabet knowledge?
• RQ4 Why is Lempel-Ziv compression less effective in DNA sequence analysis despite early attempts?
• RQ5 How do the classification speed and accuracy of LZ-based methods compare to Markov chain models in practical implementations?

Key findings

• The first-order Markov chain model correctly identified the author in 69 out of 82 test texts, significantly outperforming gzip (50 correct) and other compression tools.
• The LZ-based method from Benedetto et al. (2002) produced negative relative entropy values in practice, contradicting the theoretical requirement for non-negative entropy.
• Including Russian in the language tree disrupted expected clustering with other Slavic languages, indicating that the method relies on a-priori alphabet knowledge (e.g., Unicode), which undermines claims of being alphabet-agnostic.
• The Markov chain approach is faster in practice than the LZ-based method, contradicting claims of superior 'potentiality' for LZ.
• Compression-based methods like gzip are outperformed by both other compression tools and the Markov chain model, indicating that LZ is not optimal for authorship attribution.
• The use of substitution matrices (e.g., PAM250, BLOSUM62) in biological sequence analysis is incompatible with LZ, which limits its utility in DNA analysis.