[Paper Review] Probabilistic Contraction Analysis of Iterated Random Operators

Abhishek Gupta, Rahul Jain+1 more|arXiv (Cornell University)|Apr 4, 2018

Markov Chains and Monte Carlo Methods4 citations

TL;DR

This paper introduces a novel probabilistic contraction analysis framework to establish convergence in probability of Markov chains generated by iterated random operators in complete metric spaces. By leveraging stochastic dominance and contraction properties under i.i.d. sampling, the method proves that the fixed point of the deterministic contraction operator becomes a probabilistic fixed point in the limit, enabling convergence guarantees for Monte Carlo methods like fitted value iteration in continuous-state MDPs.

ABSTRACT

In many branches of engineering, Banach contraction mapping theorem is employed to establish the convergence of certain deterministic algorithms. Randomized versions of these algorithms have been developed that have proved useful in data-driven problems. In a class of randomized algorithms, in each iteration, the contraction map is approximated with an operator that uses independent and identically distributed samples of certain random variables. This leads to iterated random operators acting on an initial point in a complete metric space, and it generates a Markov chain. In this paper, we develop a new stochastic dominance based proof technique, called probabilistic contraction analysis, for establishing the convergence in probability of Markov chains generated by such iterated random operators in certain limiting regime. The methods developed in this paper provides a general framework for understanding convergence of a wide variety of Monte Carlo methods in which contractive property is present. We apply the convergence result to conclude the convergence of fitted value iteration and fitted relative value iteration in continuous state and continuous action Markov decision problems as representative applications of the general framework developed here.

Motivation & Objective

• To develop a general framework for analyzing convergence of Monte Carlo algorithms based on iterated random operators.
• To establish conditions under which the fixed point of a deterministic contraction operator becomes a probabilistic fixed point under random sampling.
• To provide a stochastic dominance-based proof technique applicable to a wide class of randomized algorithms in optimization and reinforcement learning.
• To demonstrate convergence of fitted value iteration and fitted relative value iteration in continuous-state and continuous-action Markov decision problems.

Proposed method

• Proposes a new proof technique based on stochastic dominance to analyze the asymptotic behavior of iterated random operators in complete metric spaces.
• Defines a probabilistic fixed point as a limit point toward which the random iterates converge in probability as sample size increases.
• Uses contraction mapping properties combined with i.i.d. sampling to model each operator approximation as a random perturbation of the deterministic map.
• Applies the framework to prove convergence of empirical value iteration in discounted and average-cost MDPs with continuous states and actions.
• Employs backward iteration arguments and invariant distribution analysis to characterize the limiting behavior of the Markov chain generated by the random operators.
• Establishes conditions under which the invariant distribution of the random operator chain concentrates around the true fixed point as the number of samples grows.

Experimental results

Research questions

• RQ1 Under what conditions does the fixed point of a deterministic contraction operator remain a probabilistic fixed point under iterated random operators?
• RQ2 How can stochastic dominance be used to prove convergence in probability for Markov chains generated by random operators?
• RQ3 What are the sufficient conditions for the convergence of fitted value iteration in continuous-state MDPs using this probabilistic contraction framework?
• RQ4 How does the number of samples per iteration affect the concentration of the random iterates around the true fixed point?

Key findings

• The fixed point of the deterministic contraction operator is shown to be a probabilistic fixed point under the proposed framework, meaning the iterates converge in probability to it as the number of samples per iteration increases.
• The framework proves that fitted value iteration converges in probability for both discounted-cost and average-cost continuous-state MDPs under mild regularity conditions.
• For a specific birth-death chain model, the invariant distribution is explicitly computed, and it is shown that π^Q(0) = (2p^β - 1)/p^β, which is non-negative when p^β ≥ 0.5.
• It is established that if p > 2β/(2β + 1), then p^β > 0.606, ensuring the invariant distribution is well-defined and positive.
• The proof technique relies on mathematical induction to derive the form of the invariant distribution across blocks of states, showing exponential decay in the tail probabilities.
• The framework provides a general tool for analyzing sample complexity and consistency of recursive stochastic algorithms based on contractive random operators.