[Paper Review] Lyapunov Conditions for Differentiability of Markov Chain Expectations: the Absolutely Continuous Case

Chang Han Rhee, Peter W. Glynn|arXiv (Cornell University)|Jul 12, 2017

Markov Chains and Monte Carlo Methods10 references3 citations

TL;DR

This paper establishes easily verifiable Lyapunov-type conditions that guarantee the differentiability of performance measures—such as stationary and random horizon discounted expectations—of general state space Markov chains with respect to underlying parameters. It provides probabilistic representations for the derivatives, enabling practical computation and sensitivity analysis without requiring restrictive moment or geometric ergodicity assumptions.

ABSTRACT

We consider a family of Markov chains whose transition dynamics are affected by model parameters. Understanding the parametric dependence of (complex) performance measures of such Markov chains is often of significant interest. The derivatives of the performance measures w.r.t. the parameters play important roles, for example, in numerical optimization of the performance measures, and quantification of the uncertainties in the performance measures when there are uncertainties in the parameters from the statistical estimation procedures. In this paper, we establish conditions that guarantee the differentiability of various types of intractable performance measures---such as the stationary and random horizon discounted performance measures---of general state space Markov chains and provide probabilistic representations for the derivatives.

Motivation & Objective

• To address the challenge of differentiating complex performance measures of Markov chains that depend on model parameters, especially in settings where traditional methods fail due to restrictive assumptions.
• To provide sufficient conditions for the differentiability of stationary and random horizon discounted performance measures in general state space Markov chains.
• To derive probabilistic representations of the derivatives that are amenable to simulation and numerical computation.
• To weaken prior assumptions—particularly on moment conditions and geometric ergodicity—thereby broadening applicability to a wider class of models.
• To support uncertainty quantification and optimization by enabling the computation of gradients under statistical estimation uncertainty.

Proposed method

• Derives sufficient Lyapunov-type conditions on the transition kernel and performance function to ensure differentiability of the expectation with respect to the parameter θ.
• Uses a coupling argument involving a reference Markov chain and a perturbed version to control the difference in expectations under parameter shifts.
• Applies a martingale representation and truncation techniques to bound the difference between the performance measures under θ and θ+h.
• Introduces weighted Lp and operator norm spaces (Lw, Lw, Mw) to ensure completeness and convergence of operators in the derivative representation.
• Employs a measure-valued differentiation framework to derive explicit representations of the derivative as an expectation involving the score function and the performance functional.
• Establishes that the derivative can be represented as an expectation involving the likelihood ratio and the performance increment, enabling simulation-based estimation.

Experimental results

Research questions

• RQ1 Under what conditions is the performance measure of a general state space Markov chain differentiable with respect to its underlying parameters?
• RQ2 Can the derivative of a stationary or random horizon discounted performance measure be represented in a form suitable for simulation and computation?
• RQ3 What minimal moment or ergodicity assumptions are required to guarantee differentiability in the absence of geometric ergodicity?
• RQ4 How can the derivative be represented probabilistically to enable uncertainty quantification and optimization under parameter estimation error?
• RQ5 Can the conditions for differentiability be verified using model-building blocks without requiring complex verification of long-term pathwise behavior?

Key findings

• The paper establishes that differentiability of performance measures holds under easily verifiable Lyapunov-type conditions, even without requiring finite exponential moments or geometric ergodicity.
• The derivative of the random horizon discounted performance measure is shown to be O(hx^p) in expectation, with bounds derived via moment control and truncation techniques.
• The derivative of the stationary measure is represented as an expectation involving the score function and the performance increment, enabling simulation-based estimation.
• The completeness of the weighted operator spaces Lw, Lw, and Mw is proven, ensuring convergence of approximating sequences and validity of limit operations.
• The method avoids reliance on strong assumptions such as finite exponential moments for stopping times or geometric ergodicity, significantly broadening applicability.
• The results support the construction of asymptotic confidence intervals for performance measures under parameter uncertainty, as formalized in the delta method.