[Paper Review] Deep Primal-Dual Reinforcement Learning: Accelerating Actor-Critic using Bellman Duality

Woon Sang Cho, Mengdi Wang|arXiv (Cornell University)|Dec 7, 2017

Reinforcement Learning in Robotics14 references7 citations

TL;DR

This paper proposes Deep Primal-Dual Reinforcement Learning, a novel actor-critic method that leverages Bellman duality to couple value and policy updates via a regularized Lagrangian formulation. By incorporating the advantage function as a penalty term, the method stabilizes training and achieves faster convergence than standard one-step TD actor-critic, demonstrating reduced sample complexity in cart-pole experiments.

ABSTRACT

We develop a parameterized Primal-Dual $π$ Learning method based on deep neural networks for Markov decision process with large state space and off-policy reinforcement learning. In contrast to the popular Q-learning and actor-critic methods that are based on successive approximations to the nonlinear Bellman equation, our method makes primal-dual updates to the policy and value functions utilizing the fundamental linear Bellman duality. Naive parametrization of the primal-dual $π$ learning method using deep neural networks would encounter two major challenges: (1) each update requires computing a probability distribution over the state space and is intractable; (2) the iterates are unstable since the parameterized Lagrangian function is no longer linear. We address these challenges by proposing a relaxed Lagrangian formulation with a regularization penalty using the advantage function. We show that the dual policy update step in our method is equivalent to the policy gradient update in the actor-critic method in some special case, while the value updates differ substantially. The main advantage of the primal-dual $π$ learning method lies in that the value and policy updates are closely coupled together using the Bellman duality and therefore more informative. Experiments on a simple cart-pole problem show that the algorithm significantly outperforms the one-step temporal-difference actor-critic method, which is the most relevant benchmark method to compare with. We believe that the primal-dual updates to the value and policy functions would expedite the learning process. The proposed methods might open a door to more efficient algorithms and sharper theoretical analysis.

Motivation & Objective

• To address the instability and intractability of direct deep primal-dual learning in large MDPs with function approximation.
• To leverage Bellman duality for tighter coupling between value and policy updates, enhancing learning efficiency.
• To stabilize parameterized primal-dual updates by introducing a regularization penalty using the advantage function.
• To show that the dual policy update is equivalent to standard policy gradient under certain conditions, enabling practical implementation.
• To improve sample efficiency by conditioning value updates on the current policy’s behavior, rather than independent estimation.

Proposed method

• Proposes a relaxed Lagrangian formulation with a regularization penalty term proportional to the square of the advantage function to stabilize training.
• Uses a minimax optimization framework where value and policy parameters are updated jointly via primal-dual steps based on Bellman duality.
• Employs the advantage function $ A_{\theta_v}(s,a) $ as a key component in the regularized Lagrangian to maintain stability and improve convergence.
• Derives policy updates that are mathematically equivalent to standard policy gradient updates under the stationary distribution, enabling online learning without estimating $ \alpha $.
• Applies stochastic gradient descent to the regularized Lagrangian, with value and policy networks updated in a coupled manner using on-policy experience.
• Uses the one-step temporal difference error $ \delta_{\theta_v}(s,a) $ as a biased estimate of the advantage function during training.

Experimental results

Research questions

• RQ1 Can primal-dual methods based on Bellman duality be effectively adapted to deep reinforcement learning with large state spaces?
• RQ2 How can the non-convexity and instability of parameterized primal-dual updates be mitigated in deep RL?
• RQ3 Does coupling value and policy updates via Bellman duality lead to faster convergence than standard actor-critic methods?
• RQ4 Is the dual policy update in the proposed method equivalent to standard policy gradient under appropriate conditions?
• RQ5 Can conditioning value updates on the current policy’s behavior improve sample efficiency?

Key findings

• The proposed method significantly reduces sample complexity compared to the one-step TD actor-critic baseline, despite slightly higher variance in cumulative rewards.
• The policy gradient update in the dual formulation is mathematically equivalent to the standard policy gradient theorem when the stationary distribution is used, validating its theoretical foundation.
• The value and policy updates are more informative and tightly coupled due to Bellman duality, leading to faster learning dynamics.
• The regularization using the advantage function stabilizes training and prevents divergence in parameterized primal-dual updates.
• Experiments on the cart-pole environment show that the algorithm converges faster and with fewer samples than the benchmark actor-critic method.
• The method avoids direct estimation of the stationary distribution $ \alpha $, enabling online, on-policy learning without costly computation.