[Paper Review] Maximum Likelihood Estimation for Markov Chains

Iuliana Teodorescu|ArXiv.org|May 26, 2009

Statistical Methods and InferenceMathematics11 references20 citations

TL;DR

This paper proposes a smoothed maximum likelihood estimation method for Markov chains with sparse transition matrices, improving confidence interval coverage in finite samples. By applying a smoothing parameter to the MLE, the method reduces bias and enhances bootstrap inference, with simulations showing superior coverage (up to 99.6%) at low smoothing values (u=0.5) compared to standard bootstrap or higher u values.

ABSTRACT

A new approach for optimal estimation of Markov chains with sparse transition matrices is presented.

Motivation & Objective

• To address poor finite-sample performance of standard maximum likelihood estimation (MLE) for Markov chains with sparse transition matrices.
• To improve the accuracy of bootstrap-based confidence intervals for transition probabilities in small or high-dimensional Markov chains.
• To investigate the impact of smoothing parameters on coverage properties of bootstrap percentile intervals.
• To compare the performance of standard bootstrap (u=∞) with smoothed bootstrap (u<∞) under varying sample sizes and transition matrix structures.

Proposed method

• Uses a smoothed estimator $\widetilde{P}$ by adding a small smoothing parameter $u$ to the MLE $\widehat{P}$, ensuring non-zero transition probabilities even for unobserved states.
• Applies the bootstrap percentile method to both $\widehat{P}$ and $\widetilde{P}$ to construct 90% confidence intervals for individual transition probabilities.
• Employs a simulation study with 1,000 replications per condition, using two distinct true transition matrices ($P_I$ and $P_{II}$) to assess coverage across different sample sizes ($n=25,50,100$) and smoothing parameters ($u=0.5,1.0,2.0,\infty$).
• Derives asymptotic normality of the MLE via the central limit theorem, showing $\sqrt{n}(\widehat{P}_{v,n} - P_v) \xrightarrow{w} N(0, \Sigma_P)$, with $\Sigma_P$ defined by a Kronecker delta structure.
• Uses the vectorized form $P_v = \text{vec}(P)$ to analyze the joint asymptotic distribution of all $d^2$ transition probabilities.
• Validates the $O(n^{-0.5})$ convergence rate of both $\widehat{P}$ and $\widetilde{P}$ through direct computation on simulated matrices at $n=500,1000,10,000$.

Experimental results

Research questions

• RQ1 Does smoothing the MLE improve the coverage of bootstrap confidence intervals for sparse Markov chain transition probabilities?
• RQ2 How does the choice of smoothing parameter $u$ affect the coverage performance of bootstrap intervals?
• RQ3 Is the smoothed bootstrap method more robust than the standard bootstrap in small-sample or high-dimensional settings?
• RQ4 Does increasing the chain length $n$ improve the coverage performance of both standard and smoothed bootstrap intervals?
• RQ5 Are the asymptotic properties of the MLE preserved under smoothing, and does the smoothed estimator maintain consistency and asymptotic normality?

Key findings

• For $n=25$, the smoothed bootstrap with $u=0.5$ achieved 99.6% empirical coverage for $P_{11}$ and $P_{12}$ in $P_{II}$, significantly exceeding the nominal 90%.
• At $n=25$, the standard bootstrap ($u=\infty$) achieved only 53.0% coverage for $P_{11}$ in $P_{II}$, indicating severe under-coverage.
• The smoothed bootstrap with $u=0.5$ consistently achieved coverage above 90% across all tested conditions, with the highest coverage (99.6%) observed at $n=25$ and $P_{II}$.
• Increasing the smoothing parameter $u$ from 0.5 to 2.0 led to a drop in coverage from ~90% to ~80%, indicating that higher $u$ values reduce interval precision.
• For $n=100$, the smoothed bootstrap with $u=0.5$ achieved 92.0% and 92.9% coverage for $P_{11}$ and $P_{12}$ in $P_I$, while the standard bootstrap achieved 87.0% and 89.1%.
• Simulations up to $n=1,000,000$ confirmed that both $\widehat{P}$ and $\widetilde{P}$ converge at rate $O(n^{-0.5})$, supporting the theoretical asymptotic normality.