[Paper Review] Fast MCMC sampling for Markov jump processes and extensions

Vinayak Rao, Yee Whye Teh|arXiv (Cornell University)|Aug 23, 2012

Markov Chains and Monte Carlo Methods33 references98 citations

TL;DR

This paper proposes an exact, efficient Gibbs sampler for posterior inference in continuous-time Markov jump processes (MJPs) using uniformization-based auxiliary variables. By alternating between sampling virtual jump times and resampling paths via forward-filtering backward-sampling, the method avoids time-discretization and matrix exponentiation, enabling exact inference with significant computational gains over state-of-the-art methods for MJPs, Markov-modulated Poisson processes, and continuous-time Bayesian networks.

ABSTRACT

Markov jump processes (or continuous-time Markov chains) are a simple and important class of continuous-time dynamical systems. In this paper, we tackle the problem of simulating from the posterior distribution over paths in these models, given partial and noisy observations. Our approach is an auxiliary variable Gibbs sampler, and is based on the idea of uniformization. This sets up a Markov chain over paths by alternately sampling a finite set of virtual jump times given the current path and then sampling a new path given the set of extant and virtual jump times using a standard hidden Markov model forward filtering-backward sampling algorithm. Our method is exact and does not involve approximations like time-discretization. We demonstrate how our sampler extends naturally to MJP-based models like Markov-modulated Poisson processes and continuous-time Bayesian networks and show significant computational benefits over state-of-the-art MCMC samplers for these models.

Motivation & Objective

• To develop an exact MCMC sampler for posterior inference over paths in Markov jump processes given noisy, partial observations.
• To avoid approximations such as time-discretization or matrix exponentiation that are computationally expensive or biased.
• To extend the sampler efficiently to complex models like Markov-modulated Poisson processes and continuous-time Bayesian networks.
• To design a method that leverages auxiliary variables in a way that decouples observation dependence from the auxiliary variable distribution, simplifying sampling.
• To achieve significant computational speedups compared to existing MCMC samplers for MJP-based models.

Proposed method

• The method uses uniformization to construct a dominating Poisson process with rate Ω ≥ max_i q_ii, enabling exact simulation of jump times.
• It introduces auxiliary virtual jump times as latent variables, which are sampled conditionally on the current path using a piecewise-constant inhomogeneous Poisson process.
• The path is resampled using a forward-filtering backward-sampling (FFBS) algorithm on a discrete-time hidden Markov model constructed from the real and virtual jump times.
• The Gibbs sampler alternates between: (1) sampling virtual jump times given the current path, and (2) resampling the full path given the set of real and virtual jump times.
• The forward pass computes filtering distributions α^t(s) = p(O^0,…,O^{t-1}, S^t = s), and the backward pass samples S_t from p(S_t = s | S_{t+1} = s', O) ∝ α^t(s) B^t_{s's} L^t(s).
• The approach is extended to MJPs with unbounded rates by treating Ω as a trajectory-dependent random variable, and to semi-Markov processes via further generalization of uniformization.

Experimental results

Research questions

• RQ1 Can an exact MCMC sampler be designed for posterior inference in continuous-time Markov jump processes without time-discretization or matrix exponentiation?
• RQ2 How can auxiliary variables be used in a way that decouples the observation likelihood from the auxiliary variable distribution, enabling efficient sampling?
• RQ3 Can the proposed method be efficiently extended to more complex models such as Markov-modulated Poisson processes and continuous-time Bayesian networks?
• RQ4 What are the computational and mixing advantages of this method compared to existing state-of-the-art samplers for MJP-based models?
• RQ5 Can the method be generalized to inhomogeneous MJPs, semi-Markov processes, or infinite-state MJPs?

Key findings

• The proposed MCMC sampler achieves exact posterior inference over MJP paths without time-discretization or matrix exponentiation, ensuring convergence to the true posterior.
• The method significantly outperforms state-of-the-art samplers in terms of computational efficiency for both standard MJPs and extended models like MMPPs and CTBNs.
• The use of auxiliary variables via uniformization allows for a tractable distribution over virtual jump times that is independent of observations, simplifying the sampling process.
• The forward-filtering backward-sampling step enables efficient resampling of the full path given the jump times, maintaining exactness and scalability.
• Experiments show that the method scales well even for models with widely varying transition rates, though tuning the dominating rate Ω remains a practical consideration.
• The framework naturally extends to inhomogeneous MJPs, semi-Markov processes, and infinite-state MJPs, with potential for adaptive tuning of Ω to improve mixing.