[Paper Review] Approximating the Stationary Probability of a Single State in a Markov chain

Christina Lee, Asuman Ozdaglar+1 more|arXiv (Cornell University)|Dec 6, 2013

Markov Chains and Monte Carlo Methods30 references3 citations

TL;DR

This paper presents a novel iterative Monte Carlo algorithm that estimates the stationary probability of a single state in a positive recurrent Markov chain by simulating return times to that state. It achieves an $\epsilon$-multiplicative approximation with high probability using $\tilde{O}(t_{\text{mix}}\ln(1/\alpha)/(\pi_i\epsilon^2))$ steps, with decreasing upper-bound estimates and a termination criterion ensuring bounded error, extending to countably infinite chains via Lyapunov functions.

ABSTRACT

In this paper, we present a novel iterative Monte Carlo method for approximating the stationary probability of a single state of a positive recurrent Markov chain. We utilize the characterization that the stationary probability of a state $i$ is inversely proportional to the expected return time of a random walk beginning at $i$. Our method obtains an $ε$-multiplicative close estimate with probability greater than $1 - α$ using at most $ ilde{O}\left(t_{ ext{mix}} \ln(1/α) / π_i ε^2 ight)$ simulated random walk steps on the Markov chain across all iterations, where $t_{ ext{mix}}$ is the standard mixing time and $π_i$ is the stationary probability. In addition, the estimate at each iteration is guaranteed to be an upper bound with high probability, and is decreasing in expectation with the iteration count, allowing us to monitor the progress of the algorithm and design effective termination criteria. We propose a termination criteria which guarantees a $ε(1 + 4 \ln(2) t_{ ext{mix}})$ multiplicative error performance for states with stationary probability larger than $Δ$, while providing an additive error for states with stationary probability less than $Δ\in (0,1)$. The algorithm along with this termination criteria uses at most $ ilde{O}\left(\frac{\ln(1/α)}{ε^2} \min\left(\frac{t_{ ext{mix}}}{π_i}, \frac{1}{εΔ} ight) ight)$ simulated random walk steps, which is bounded by a constant with respect to the Markov Chain. We provide a tight analysis of our algorithm based on a locally weighted variant of the mixing time. Our results naturally extend for countably infinite state space Markov chains via Lyapunov function analysis.

Motivation & Objective

• To develop an efficient algorithm for estimating the stationary probability of a single state in large or infinite Markov chains without computing the full stationary distribution.
• To focus computation locally around the target state by leveraging return time sampling, reducing cost in sparse, high-dimensional systems.
• To provide a termination criterion that ensures multiplicative or additive error bounds depending on the target state's stationary probability.
• To extend the method to countably infinite state spaces using Lyapunov function analysis for positive recurrence.

Proposed method

• The method uses the inverse relationship $\pi_i = 1/\mathbb{E}_i[T_i]$, where $T_i$ is the return time to state $i$, to estimate $\pi_i$ via simulation of return walks.
• It iteratively increases a truncation threshold $\theta^{(k)}$ and the number of samples to refine estimates, ensuring convergence.
• At each iteration, the algorithm computes an upper-bound estimate of $\pi_i$ that decreases in expectation, enabling progress monitoring.
• A locally weighted mixing time is introduced to tightly analyze the convergence rate, replacing standard mixing time in the complexity bound.
• The algorithm uses Chernoff bounds to ensure high-probability concentration of sample averages around their expectations.
• For infinite state spaces, a Lyapunov function is constructed with $V(j) = \mathbb{E}_j[T_i]$ to establish positive recurrence and enable analysis.

Experimental results

Research questions

• RQ1 Can we estimate the stationary probability of a single state in a Markov chain more efficiently than computing the full stationary distribution?
• RQ2 How can we design a local, iterative Monte Carlo method that converges to the correct stationary probability with high probability and bounded error?
• RQ3 What termination criterion ensures multiplicative or additive error guarantees depending on the magnitude of $\pi_i$?
• RQ4 How can the method be extended to countably infinite state spaces while maintaining theoretical guarantees?
• RQ5 What is the optimal trade-off between computational cost and accuracy in terms of $\epsilon$, $\alpha$, $t_{\text{mix}}$, and $\pi_i$?

Key findings

• The algorithm achieves an $\epsilon$-multiplicative approximation of $\pi_i$ with probability greater than $1-\alpha$ using $\tilde{O}(t_{\text{mix}}\ln(1/\alpha)/(\pi_i\epsilon^2))$ simulated steps.
• The estimate at each iteration is an upper bound with high probability and decreases in expectation, enabling effective monitoring and termination.
• A termination criterion guarantees $\epsilon(1+4\ln(2)t_{\text{mix}})$ multiplicative error for states with $\pi_i > \Delta$, and additive error for $\pi_i < \Delta$, with total cost bounded by $\tilde{O}(\ln(1/\alpha)/\epsilon^2 \cdot \min(t_{\text{mix}}/\pi_i, 1/(\epsilon\Delta)))$.
• The method is extended to countably infinite state spaces via Lyapunov function analysis, ensuring validity under positive recurrence.
• The analysis introduces a locally weighted mixing time, which provides tighter bounds than standard mixing time in the complexity analysis.
• The algorithm’s complexity is independent of the overall state space size, making it suitable for large-scale and sparse Markov chains.