[Paper Review] Entropy Rate Estimation for Markov Chains with Large State Space

Yanjun Han, Jiantao Jiao+4 more|arXiv (Cornell University)|Feb 22, 2018

Machine Learning and Algorithms40 references3 citations

TL;DR

This paper establishes the optimal sample complexity for estimating the entropy rate of a stationary reversible Markov chain with S states, showing that consistent estimation is possible when n ≫ S²/log S under mild mixing conditions, and impossible below n ≲ S²/log S under weak dependency. The optimal estimation accuracy is Θ(S²/(n log S)), significantly improving upon the Ω(S²) requirement of empirical estimators even in the memoryless case.

ABSTRACT

Estimating the entropy based on data is one of the prototypical problems in distribution property testing and estimation. For estimating the Shannon entropy of a distribution on $S$ elements with independent samples, [Paninski2004] showed that the sample complexity is sublinear in $S$, and [Valiant--Valiant2011] showed that consistent estimation of Shannon entropy is possible if and only if the sample size $n$ far exceeds $\frac{S}{\log S}$. In this paper we consider the problem of estimating the entropy rate of a stationary reversible Markov chain with $S$ states from a sample path of $n$ observations. We show that: (1) As long as the Markov chain mixes not too slowly, i.e., the relaxation time is at most $O(\frac{S}{\ln^3 S})$, consistent estimation is achievable when $n \gg \frac{S^2}{\log S}$. (2) As long as the Markov chain has some slight dependency, i.e., the relaxation time is at least $1+Ω(\frac{\ln^2 S}{\sqrt{S}})$, consistent estimation is impossible when $n \lesssim \frac{S^2}{\log S}$. Under both assumptions, the optimal estimation accuracy is shown to be $Θ(\frac{S^2}{n \log S})$. In comparison, the empirical entropy rate requires at least $Ω(S^2)$ samples to be consistent, even when the Markov chain is memoryless. In addition to synthetic experiments, we also apply the estimators that achieve the optimal sample complexity to estimate the entropy rate of the English language in the Penn Treebank and the Google One Billion Words corpora, which provides a natural benchmark for language modeling and relates it directly to the widely used perplexity measure.

Motivation & Objective

• To determine the optimal sample complexity for consistent entropy rate estimation in large-state Markov chains.
• To characterize the trade-off between mixing time (relaxation time) and sample size for reliable estimation.
• To close the gap between theoretical limits and practical estimators in high-dimensional, dependent data settings.
• To apply the proposed estimators to real-world language corpora and benchmark language modeling performance via perplexity.

Proposed method

• The authors analyze the entropy rate estimation problem under the framework of stationary reversible Markov chains with S states.
• They derive non-asymptotic bounds on the sample complexity using spectral gap analysis and concentration inequalities.
• The method involves constructing a random transition matrix model with controlled relaxation time and applying Weyl's inequality to bound the spectral gap.
• A key component is the use of empirical conditional distributions and Kullback–Leibler divergence to relate estimation error to empirical likelihood.
• The analysis leverages results from random matrix theory, particularly spectral norm concentration of Wigner-type matrices.
• Theoretical bounds are validated through synthetic experiments and applied to real language data from Penn Treebank and Google One Billion Words.

Experimental results

Research questions

• RQ1 What is the minimal sample size required to consistently estimate the entropy rate of a large-state Markov chain?
• RQ2 How does the relaxation time of the Markov chain affect the feasibility of consistent entropy rate estimation?
• RQ3 Can the optimal estimation accuracy be characterized in terms of S and n under different mixing conditions?
• RQ4 How does the proposed estimator compare to the empirical entropy rate estimator in terms of sample efficiency?
• RQ5 To what extent can the theoretical bounds be applied to real-world language modeling tasks?

Key findings

• Consistent entropy rate estimation is achievable when n ≫ S²/log S, provided the relaxation time is O(S / log³ S).
• Consistent estimation is impossible when n ≲ S²/log S if the relaxation time is at least 1 + Ω(ln² S / √S).
• The optimal estimation accuracy is Θ(S² / (n log S)), which matches the minimax rate under the given conditions.
• The empirical entropy rate estimator requires Ω(S²) samples for consistency, even in the memoryless case.
• The proposed estimator achieves optimal sample complexity and was successfully applied to estimate the entropy rate of English language in large-scale corpora, providing a benchmark for perplexity.