[Paper Review] Stationary distributions of continuous-time Markov chains: a review of theory and truncation-based approximations

Juan Kuntz, Philipp Thomas+2 more|arXiv (Cornell University)|Sep 12, 2019

Gene Regulatory Network Analysis162 references4 citations

TL;DR

This paper reviews the theory and approximation methods for stationary distributions in continuous-time Markov chains (CTMCs), focusing on truncation-based schemes for infinite state spaces. It presents convergence analysis, error bounds, and computational trade-offs, with key contributions including computable error bounds via Lyapunov functions and a framework for a priori error estimation.

ABSTRACT

Computing the stationary distributions of a continuous-time Markov chain (CTMC) involves solving a set of linear equations. In most cases of interest, the number of equations is infinite or too large, and the equations cannot be solved analytically or numerically. Several approximation schemes overcome this issue by truncating the state space to a manageable size. In this review, we first give a comprehensive theoretical account of the stationary distributions and their relation to the long-term behaviour of CTMCs that is readily accessible to non-experts and free of irreducibility assumptions made in standard texts. We then review truncation-based approximation schemes for CTMCs with infinite state spaces paying particular attention to the schemes' convergence and the errors they introduce, and we illustrate their performance with an example of a stochastic reaction network of relevance in biology and chemistry. We conclude by discussing computational trade-offs associated with error control and several open questions.

Motivation & Objective

• To provide a comprehensive, accessible review of stationary distributions in CTMCs without requiring irreducibility assumptions, making advanced theory available to non-experts.
• To analyze and compare truncation-based approximation schemes for CTMCs with infinite state spaces, focusing on convergence and error control.
• To evaluate the performance of methods such as finite state projection, truncation-and-augmentation, and linear programming in approximating stationary distributions.
• To investigate computational trade-offs between accuracy, truncation size, and numerical stability in practical implementations.
• To identify open theoretical questions, particularly regarding a priori error prediction and optimal parameter selection in approximation schemes.

Proposed method

• Theoretical analysis of stationary distributions using closed communicating classes and ergodic distributions, without assuming irreducibility.
• Application of Foster-Lyapunov criteria to establish existence and uniqueness of stationary distributions in general CTMCs.
• Use of truncation-based schemes—such as finite state projection, truncation-and-augmentation (TA), iterated TA (ITA), and level-dependent QBD processes (LDQBDP)—to approximate stationary distributions on finite subsets.
• Employment of linear programming (LP) and its variants (ILP) to compute computable lower and upper bounds on the total variation error of approximations.
• Introduction of Lyapunov-function-based error bounds that scale proportionally with actual error for large truncations, offering tighter and more reliable error control.
• Numerical evaluation on a stochastic toggle switch model to compare convergence, error behavior, and computational cost across schemes.

Experimental results

Research questions

• RQ1 How can stationary distributions be characterized in CTMCs without assuming irreducibility, and what conditions ensure their existence and uniqueness?
• RQ2 Under what conditions do truncation-based approximation schemes such as TA, ITA, and LDQBDP converge to the true stationary distribution?
• RQ3 How can computable error bounds be derived for these approximations, and can they be made tight and reliable without relying on conservative tail bounds?
• RQ4 What is the relationship between the truncation error and the actual approximation error, and can this be used for a priori error control?
• RQ5 How should free parameters (e.g., β in the Lyapunov-based bound) be selected to minimize error bounds and improve approximation quality?

Key findings

• The set of stationary distributions in a CTMC corresponds exactly to the convex hull of the ergodic distributions on closed communicating classes, even without irreducibility.
• Truncation-based schemes such as TA, ITA, and LDQBDP converge to the true stationary distribution under mild conditions, but convergence depends on the choice of re-entry matrices and truncation structure.
• Lyapunov-function-based error bounds for the TA scheme become proportional to the actual error for large truncations, offering a significant improvement over conservative tail-bound estimates.
• For large truncations, the errors of the ITA and ILP lower bounds were observed to equal the tail bound, while upper bounds were proportional to the truncation error, suggesting potential for a priori error estimation.
• Numerical instability due to large condition numbers in truncated rate matrices is a major challenge, mitigated by scaling or higher-precision arithmetic.
• The ILP scheme provides an automatic test for uniqueness of the stationary distribution: if any state-wise lower bound is non-zero, then at most one stationary distribution exists.