[Paper Review] Proximal Reinforcement Learning: A New Theory of Sequential Decision Making in Primal-Dual Spaces

Sridhar Mahadevan, Bo Liu+6 more|arXiv (Cornell University)|May 26, 2014

Stochastic Gradient Optimization Techniques113 references45 citations

TL;DR

This paper introduces a novel proximal reinforcement learning framework that unifies temporal difference learning and stochastic optimization using primal-dual spaces via Legendre transforms and proximal operators. It enables provably convergent, stable, and safe off-policy learning with improved convergence rates, including accelerated $O(1/N)$ rates for GTD2-MP, and provides a systematic foundation for mirror descent, natural gradient, and sparse learning in RL.

ABSTRACT

In this paper, we set forth a new vision of reinforcement learning developed by us over the past few years, one that yields mathematically rigorous solutions to longstanding important questions that have remained unresolved: (i) how to design reliable, convergent, and robust reinforcement learning algorithms (ii) how to guarantee that reinforcement learning satisfies pre-specified "safety" guarantees, and remains in a stable region of the parameter space (iii) how to design "off-policy" temporal difference learning algorithms in a reliable and stable manner, and finally (iv) how to integrate the study of reinforcement learning into the rich theory of stochastic optimization. In this paper, we provide detailed answers to all these questions using the powerful framework of proximal operators. The key idea that emerges is the use of primal dual spaces connected through the use of a Legendre transform. This allows temporal difference updates to occur in dual spaces, allowing a variety of important technical advantages. The Legendre transform elegantly generalizes past algorithms for solving reinforcement learning problems, such as natural gradient methods, which we show relate closely to the previously unconnected framework of mirror descent methods. Equally importantly, proximal operator theory enables the systematic development of operator splitting methods that show how to safely and reliably decompose complex products of gradients that occur in recent variants of gradient-based temporal difference learning. This key technical innovation makes it possible to finally design "true" stochastic gradient methods for reinforcement learning. Finally, Legendre transforms enable a variety of other benefits, including modeling sparsity and domain geometry. Our work builds extensively on recent work on the convergence of saddle-point algorithms, and on the theory of monotone operators.

Motivation & Objective

• To develop a mathematically rigorous theory of reinforcement learning that ensures convergence, stability, and safety in sequential decision-making.
• To resolve longstanding challenges in off-policy temporal difference learning by enabling reliable, stable, and convergent algorithms.
• To unify natural gradient methods and mirror descent under a common proximal operator framework.
• To enable true stochastic gradient methods in RL through operator splitting and proximal updates.
• To integrate RL into the broader theory of stochastic composite optimization with guarantees on convergence and sparsity.

Proposed method

• Uses Legendre transforms to map between primal and dual spaces, enabling updates in dual spaces for improved stability and convergence.
• Applies proximal operators to handle non-smooth regularizers and composite objectives, particularly in value function approximation.
• Employs operator splitting strategies—especially forward-backward and primal-dual splitting—to decompose complex gradient products in off-policy TD learning.
• Introduces the GTD2-MP algorithm as a mirror-prox variant that achieves accelerated convergence via extragradient-style updates.
• Leverages monotone operator theory and saddle-point formulations to analyze convergence and derive optimal rates.
• Uses Bregman divergences and mirror descent to enable sparse learning and geometry-aware value function approximation.

Experimental results

Research questions

• RQ1 How can we design provably convergent and stable reinforcement learning algorithms under off-policy settings?
• RQ2 How can we guarantee safety and stability by keeping parameters within a stable region of the parameter space?
• RQ3 How can we systematically derive true stochastic gradient methods for value function learning in RL?
• RQ4 How can we unify natural gradient and mirror descent methods under a common theoretical framework?
• RQ5 How can we achieve accelerated convergence rates in off-policy temporal difference learning?

Key findings

• The GTD2-MP algorithm achieves an accelerated convergence rate of $Oig(rac{L_{F^*} + L_K}{N} + rac{ heta}{ u}ig)$, improving upon the $Oig(rac{L_{F^*} + L_K + heta}{ u}ig)$ rate of standard GTD/GTD2.
• The value approximation error $||V - V_ heta||_ ext{infty}$ is bounded by $\frac{L_ ext{phi}^\Xi}{1 - \gamma} \cdot O\big(\frac{L_{F^*} + L_K}{N} + \frac{\sigma}{\sqrt{N}}\big)$ for GTD2-MP, with improved sample efficiency.
• The framework establishes equivalence between natural gradient descent and mirror descent via Legendre transforms, unifying two major optimization paradigms in RL.
• Proximal operators enable systematic decomposition of complex gradient products, making true stochastic gradient methods feasible in RL.
• The use of Bregman divergences allows for sparse learning and modeling of domain geometry, enabling efficient representation in high-dimensional spaces.
• Theoretical analysis confirms that adding a primal average step to GTD/GTD2 transforms them into standard Polyak-type algorithms with $O(1/\sqrt{N})$ convergence rates.