[Paper Review] Ergodic BSDEs driven by Markov Chains

Samuel N. Cohen, Ying Hu|arXiv (Cornell University)|Jul 24, 2012

Stochastic processes and financial applications15 references4 citations

TL;DR

This paper establishes the existence and uniqueness of bounded Markovian solutions to ergodic backward stochastic differential equations (EBSDEs) driven by uniformly ergodic countable-state Markov chains with Lipschitz drivers satisfying a comparison theorem. The key contribution is proving that uniform ergodicity of the underlying chain is preserved under perturbations of the rate matrix, enabling the construction of EBSDE solutions via coupling and splitting techniques, with applications to risk-averse ergodic control.

ABSTRACT

We consider ergodic backward stochastic differential equations, in a setting where noise is generated by a countable state uniformly ergodic Markov chain. We show that for Lipschitz drivers such that a comparison theorem holds, these equations admit unique solutions. To obtain this result, we show by coupling and splitting techniques that uniform ergodicity estimates of Markov chains are robust to perturbations of the rate matrix, and that these perturbations correspond in a natural way to EBSDEs. We then consider applications of this theory to Markov decision problems with a risk-averse average reward criterion.

Motivation & Objective

• To establish the existence and uniqueness of bounded Markovian solutions to ergodic BSDEs (EBSDEs) when the noise is generated by a uniformly ergodic countable-state Markov chain.
• To demonstrate that uniform ergodicity estimates of Markov chains are robust under perturbations of the rate matrix, which naturally arise in EBSDEs.
• To extend the theory of EBSDEs to nonlinear, risk-averse control problems by leveraging the connection between EBSDEs and nonlinear expectations.
• To provide a framework for solving infinite-horizon optimal control problems with average reward criteria under risk aversion, using EBSDEs with balanced drivers.

Proposed method

• Uses coupling and splitting techniques (Nummelin splitting) to analyze the robustness of uniform ergodicity under perturbations of the rate matrix of the Markov chain.
• Introduces a novel partial ordering on rate matrices to quantify how small perturbations preserve uniform ergodicity, ensuring stability of ergodicity estimates.
• Applies the theory of discounted BSDEs to derive time-invariant, Markovian solutions that converge to the ergodic limit as the time horizon tends to infinity.
• Constructs a driver function f(x,z) that captures risk-averse dynamics by combining state-dependent costs and perturbations of the rate matrix, enabling nonlinear expectation modeling.
• Employs the comparison theorem for BSDEs to ensure uniqueness of solutions under Lipschitz and balanced driver conditions.
• Applies the EBSDE solution to optimal control by identifying the long-run average cost λ as the solution component, with the value function derived from the driver f.

Experimental results

Research questions

• RQ1 Under what conditions does an ergodic BSDE driven by a uniformly ergodic Markov chain admit a unique bounded Markovian solution?
• RQ2 How do perturbations of the rate matrix of a Markov chain affect its uniform ergodicity, and can this robustness be quantified?
• RQ3 Can EBSDEs be used to model risk-averse ergodic control problems where the cost functional involves a nonlinear expectation over multiple possible rate matrices?
• RQ4 What is the relationship between the ergodic cost λ in the EBSDE and the optimal long-run average reward in a controlled Markov decision process?

Key findings

• Ergodic BSDEs with Lipschitz drivers satisfying a comparison theorem admit unique bounded Markovian solutions when driven by a uniformly ergodic countable-state Markov chain.
• Uniform ergodicity of the underlying Markov chain is preserved under small perturbations of the rate matrix, as quantified by a novel partial ordering of rate matrices.
• The ergodic cost λ in the EBSDE corresponds exactly to the optimal long-run average cost in the associated Markov decision problem, enabling solution of risk-averse control problems.
• The solution to the EBSDE can be computed via a single nonlinear vector equation, offering explicit computability in certain cases.
• The framework naturally accommodates risk-averse control by modeling uncertainty in the rate matrix through a family of matrices, with the driver f capturing the worst-case or nonlinear expectation over these.
• The results extend to nonlinear expectations via balanced drivers, and the theory is robust to weakening of the uniform ergodicity assumption to geometric ergodicity under additional structural constraints.