[Paper Review] Lambda-Policy Iteration with Randomization for Contractive Models with Infinite Policies: Well-Posedness and Convergence (Extended Version)

Yuchao Li, Karl Henrik Johansson+1 more|arXiv (Cornell University)|Dec 18, 2019

Reinforcement Learning in Robotics14 references4 citations

TL;DR

This paper extends lambda-policy iteration with randomization (λ-PIR) to contractive Markov decision processes with infinite policies, proving the well-posedness of the λ-operator and establishing almost-sure convergence under mild conditions. It demonstrates that λ-PIR maintains convergence even in infinite policy spaces when the model is contractive, and provides a data-driven implementation for approximating optimal cost functions in constrained linear and nonlinear control problems with strong empirical performance.

ABSTRACT

Abstract dynamic programming models are used to analyze $λ$-policy iteration with randomization algorithms. Particularly, contractive models with infinite policies are considered and it is shown that well-posedness of the $λ$-operator plays a central role in the algorithm. The operator is known to be well-posed for problems with finite states, but our analysis shows that it is also well-defined for the contractive models with infinite states studied. Similarly, the algorithm we analyze is known to converge for problems with finite policies, but we identify the conditions required to guarantee convergence with probability one when the policy space is infinite regardless of the number of states. Guided by the analysis, we exemplify a data-driven approximated implementation of the algorithm for estimation of optimal costs of constrained linear and nonlinear control problems. Numerical results indicate potentials of this method in practice.

Motivation & Objective

• To extend λ-policy iteration with randomization (λ-PIR) from finite to infinite-policy Markov decision processes.
• To establish the well-posedness of the λ-operator in contractive models with infinite states and policies.
• To identify sufficient conditions for almost-sure convergence of λ-PIR when the policy space is infinite.
• To develop a data-driven, approximated implementation of λ-PIR within the approximate dynamic programming (ADP) framework for real-time control.
• To validate the method on constrained linear and nonlinear control problems through numerical experiments.

Proposed method

• Uses abstract dynamic programming models to analyze λ-PIR, focusing on the λ-operator’s properties in infinite-state, infinite-policy settings.
• Proves the λ-operator is well-posed under the contraction property of the model, generalizing results from finite-state cases.
• Introduces a convergence condition for λ-PIR in infinite-policy problems, which can be relaxed if the problem exhibits a linear structure.
• Proposes a data-driven approximation of λ-PIR using function approximation (e.g., parameterized cost functions) for online learning.
• Employs temporal-difference learning principles and proximal algorithm insights to maintain convergence and stability.
• Embeds the algorithm in an ADP framework, using iterative policy evaluation and improvement with λ-regularized updates.

Experimental results

Research questions

• RQ1 Is the λ-operator well-posed in contractive Markov decision processes with infinite states and infinite policies?
• RQ2 What conditions ensure almost-sure convergence of λ-PIR when the policy space is uncountably infinite?
• RQ3 Can λ-PIR be effectively adapted to infinite-policy problems without losing convergence guarantees?
• RQ4 How can λ-PIR be implemented in a data-driven, approximated form suitable for real-time control of constrained systems?
• RQ5 Does the proposed method outperform standard VI and OPI in terms of convergence speed and sample efficiency?

Key findings

• The λ-operator is well-posed for contractive models with infinite states and policies, relying solely on the contraction property of the underlying model.
• Convergence of λ-PIR to the optimal cost function is guaranteed with probability one under a mild additional condition, even when the policy space is infinite.
• If the problem exhibits a linear structure, the convergence condition can be removed, simplifying the theoretical requirements.
• Numerical results show that the data-driven implementation of λ-PIR achieves faster convergence and better control performance than standard value iteration and optimistic policy iteration.
• In both linear and nonlinear control examples, the estimated cost functions converged after just 5 iterations, and the resulting closed-loop systems showed significant performance improvements.
• The method effectively combines the fast convergence of proximal algorithms with the stability of value iteration, outperforming OPI in sample efficiency.