[Paper Review] On Learning Markov Chains

Hao Yi, Alon Orlitsky+1 more|arXiv (Cornell University)|Jan 1, 2018

Machine Learning and Algorithms13 citations

TL;DR

This paper resolves long-standing open problems in learning Markov chains by establishing tight min-max risk bounds for estimating k-state Markov chains from n sequential samples. It provides near-optimal bounds for predicting the next state using KL-divergence and solves the estimation problem under smooth f-divergences when transition probabilities are bounded away from zero.

ABSTRACT

The problem of estimating an unknown discrete distribution from its samples is a fundamental tenet of statistical learning. Over the past decade, it attracted significant research effort and has been solved for a variety of divergence measures. Surprisingly, an equally important problem, estimating an unknown Markov chain from its samples, is still far from understood. We consider two problems related to the min-max risk (expected loss) of estimating an unknown k-state Markov chain from its n sequential samples: predicting the conditional distribution of the next sample with respect to the KL-divergence, and estimating the transition matrix with respect to a natural loss induced by KL or a more general f-divergence measure. For the first measure, we determine the min-max prediction risk to within a linear factor in the alphabet size, showing it is \Omega(k\log\log n/n) and O(k^2\log\log n/n). For the second, if the transition probabilities can be arbitrarily small, then only trivial uniform risk upper bounds can be derived. We therefore consider transition probabilities that are bounded away from zero, and resolve the problem for essentially all sufficiently smooth f-divergences, including KL-, L_2-, Chi-squared, Hellinger, and Alpha-divergences.

Motivation & Objective

• To address the fundamental problem of estimating an unknown k-state Markov chain from n sequential samples, which remains poorly understood despite progress in distribution estimation.
• To determine the min-max prediction risk for predicting the next sample's conditional distribution using KL-divergence.
• To resolve the estimation of the transition matrix under general f-divergence measures, particularly when transition probabilities are bounded away from zero.
• To provide tight upper and lower bounds for the risk in both prediction and estimation settings, closing gaps in prior work.

Proposed method

• The authors analyze the min-max risk using information-theoretic techniques, focusing on the worst-case expected loss over all k-state Markov chains.
• They derive lower bounds via a construction of adversarial Markov chains that exploit the structure of the KL-divergence and the sample complexity.
• For upper bounds, they design a novel estimation strategy based on empirical frequency estimation with careful concentration bounds, adapted to the sequential sampling model.
• They extend the analysis to general f-divergences by leveraging the smoothness properties of the divergence, enabling results for KL, L2, Chi-squared, Hellinger, and Alpha-divergences.
• The analysis incorporates logarithmic and iterated logarithmic factors to refine the dependence on the sample size n and alphabet size k.
• They prove that when transition probabilities are bounded away from zero, non-trivial risk bounds can be achieved, unlike in the general case where only uniform bounds exist.

Experimental results

Research questions

• RQ1 What is the min-max prediction risk for estimating the next state's conditional distribution using KL-divergence, and how does it scale with k and n?
• RQ2 Can non-trivial risk bounds be derived for estimating the transition matrix when transition probabilities are arbitrarily small?
• RQ3 What is the optimal rate of convergence for estimating the transition matrix under smooth f-divergences such as KL and L2?
• RQ4 How does the lower bound on the prediction risk scale with the number of states k and the sample size n?
• RQ5 Under what conditions can tight upper and lower bounds be established for the estimation risk in Markov chains?

Key findings

• The min-max prediction risk for next-state prediction under KL-divergence is bounded between Ω(k log log n / n) and O(k² log log n / n), resolving the asymptotic rate up to a linear factor in k.
• When transition probabilities are bounded away from zero, the paper establishes tight risk bounds for all sufficiently smooth f-divergences, including KL, L2, Chi-squared, Hellinger, and Alpha-divergences.
• For general transition matrices with arbitrarily small probabilities, only trivial uniform upper bounds on the risk can be derived, indicating the necessity of a lower bound assumption.
• The analysis shows that the dependence on the number of states k is tight up to a linear factor, highlighting the fundamental difficulty of learning Markov chains as k increases.
• The results demonstrate that the sequential nature of the samples introduces a non-trivial structure that must be carefully accounted for in the risk analysis.
• The paper closes a long-standing gap in statistical learning theory for Markov chains by providing the first nearly-matching upper and lower bounds for key estimation problems.