[Paper Review] On Connections between Constrained Optimization and Reinforcement Learning

Nino Vieillard, Olivier Pietquin+1 more|arXiv (Cornell University)|Oct 18, 2019

Reinforcement Learning in Robotics20 references7 citations

TL;DR

This paper establishes formal algorithmic connections between constrained optimization and reinforcement learning by linking three dynamic programming schemes—Conservative Policy Iteration, Mirror Descent Modified Policy Iteration, and Politex—to fundamental convex optimization algorithms: Frank-Wolfe, Mirror Descent, and Dual Averaging, respectively. The key contribution is a unified framework that reveals shared structural principles, enabling cross-fertilization between optimization theory and RL for designing more efficient, analyzable algorithms.

ABSTRACT

Dynamic Programming (DP) provides standard algorithms to solve Markov Decision Processes. However, these algorithms generally do not optimize a scalar objective function. In this paper, we draw connections between DP and (constrained) convex optimization. Specifically, we show clear links in the algorithmic structure between three DP schemes and optimization algorithms. We link Conservative Policy Iteration to Frank-Wolfe, Mirror-Descent Modified Policy Iteration to Mirror Descent, and Politex (Policy Iteration Using Expert Prediction) to Dual Averaging. These abstract DP schemes are representative of a number of (deep) Reinforcement Learning (RL) algorithms. By highlighting these connections (most of which have been noticed earlier, but in a scattered way), we would like to encourage further studies linking RL and convex optimization, that could lead to the design of new, more efficient, and better understood RL algorithms.

Motivation & Objective

• To formalize algorithmic parallels between dynamic programming schemes in reinforcement learning and convex optimization algorithms.
• To unify disparate connections between RL and optimization that have previously been observed in isolation or informally.
• To enable the transfer of optimization techniques—such as regularization and convergence analysis—into RL algorithm design.
• To provide a clearer theoretical foundation for understanding and improving modern RL algorithms through optimization lenses.
• To inspire new, more efficient, and better-understood RL algorithms by leveraging insights from convex optimization.

Proposed method

• Mapping the iterative structure of Conservative Policy Iteration to the Frank-Wolfe algorithm via the use of state-action value functions as gradients.
• Establishing equivalence between Mirror Descent Modified Policy Iteration and Mirror Descent by aligning policy updates with Bregman divergence regularization.
• Reinterpreting Politex as a Dual Averaging scheme where policy updates are based on cumulative state-action value estimates, regularized by entropy.
• Generalizing these connections to Modified Policy Iteration frameworks, allowing partial policy evaluation while preserving algorithmic analogies.
• Using the Bellman operator and value function dynamics to derive analogies between optimization steps and RL policy improvement.
• Formalizing the correspondence through shared update rules: policy update as a maximization of a linear term minus a regularizer, with gradients replaced by estimated Q-values.

Experimental results

Research questions

• RQ1 How do the algorithmic structures of Conservative Policy Iteration and Frank-Wolfe compare in terms of policy improvement and convergence?
• RQ2 In what way does Mirror Descent Modified Policy Iteration mirror the behavior of Mirror Descent in convex optimization?
• RQ3 What is the precise connection between the Dual Averaging scheme and the Politex algorithm in terms of policy update rules?
• RQ4 Can the structural similarities between these RL and optimization algorithms be leveraged to improve convergence or sample efficiency in RL?
• RQ5 What theoretical insights from convex optimization can be transferred to better understand and analyze modern RL algorithms?

Key findings

• Conservative Policy Iteration is algorithmically equivalent to the Frank-Wolfe method when the state-action value function is used as a gradient estimate.
• Mirror Descent Modified Policy Iteration corresponds exactly to the Mirror Descent algorithm, with policy updates regularized by a Bregman divergence.
• Politex is formally equivalent to the Dual Averaging scheme, where the cumulative sum of Q-values acts as the dual averaging term.
• These connections are preserved in Modified Policy Iteration variants, where partial policy evaluation is used, showing robustness of the analogies.
• The results suggest that optimization techniques such as regularization and convergence analysis can be directly applied to improve and analyze RL algorithms.
• The framework provides a unified lens to understand and design RL algorithms by drawing on well-established convex optimization theory.