[Paper Review] Stein Variational Policy Gradient

Yang Liu, Prajit Ramachandran+2 more|arXiv (Cornell University)|Apr 7, 2017

Reinforcement Learning in Robotics25 references65 citations

TL;DR

SVPG combines policy gradients with Stein variational gradient descent to maintain a diverse set of high-performing policies, improving data efficiency and robustness over standard REINFORCE and A2C in continuous control tasks.

ABSTRACT

Policy gradient methods have been successfully applied to many complex reinforcement learning problems. However, policy gradient methods suffer from high variance, slow convergence, and inefficient exploration. In this work, we introduce a maximum entropy policy optimization framework which explicitly encourages parameter exploration, and show that this framework can be reduced to a Bayesian inference problem. We then propose a novel Stein variational policy gradient method (SVPG) which combines existing policy gradient methods and a repulsive functional to generate a set of diverse but well-behaved policies. SVPG is robust to initialization and can easily be implemented in a parallel manner. On continuous control problems, we find that implementing SVPG on top of REINFORCE and advantage actor-critic algorithms improves both average return and data efficiency.

Motivation & Objective

• Motivate the need for more robust and data-efficient policy optimization in reinforcement learning.
• Propose a maximum entropy framework over policy parameters to encourage exploration.
• Introduce SVPG to jointly exploit gradient information and repulse particles for diversity.
• Demonstrate improved performance and robustness of SVPG on continuous control benchmarks.

Proposed method

• Model policy parameters as a distribution q(θ) with a prior q0(θ) to maximize E_q[J(θ)] + α H(q).
• Derive posterior form q(θ) ∝ exp(J(θ)/α) q0(θ) and interpret α as a temperature control.
• Apply Stein Variational Gradient Descent (SVGD) to iteratively transport a set of particles θi to approximate q(θ).
• Use SVGD updates that combine the gradient of log q with a repulsive kernel term to maintain diversity: φ*(θ) = E_θ~ρ[∇log q(θ) k(θ,·) + ∇_θ k(θ,·)].
• Instantiate SVPG by replacing ∇log q with ∇θ[(1/α)J(θ) + log q0(θ)] in the SVGD update, yielding coupled policy-gradient and repulsion terms.
• Provide annealing of α to balance exploration and exploitation.
• Demonstrate SVPG on top of REINFORCE and A2C to improve data efficiency and robustness on continuous control tasks.

Experimental results

Research questions

• RQ1 Can a distributional, entropy-regularized view of policy parameters improve exploration and learning stability?
• RQ2 Does SVGD-based policy parameter optimization yield diverse yet high-performing policies compared to single-policy updates?
• RQ3 How does SVPG perform when combined with standard policy gradient methods (REINFORCE, A2C) on continuous control benchmarks?
• RQ4 What is the impact of the temperature parameter α on exploration-exploitation trade-offs in SVPG?

Key findings

• SVPG improves average return and data efficiency when added on top of REINFORCE and A2C in continuous control tasks.
• SVPG yields robust performance across different initializations and shows smaller variance than Joint updates and competitive diversity with Independent updates.
• SVPG discovers multiple strong policies with diverse state-visitation patterns, indicating effective parameter-space exploration.
• An intermediate α balances exploration and exploitation and supports better performance than too high or too low temperatures.
• SVPG’s parallel, nonparametric particle updates allow scalable integration with existing policy optimization frameworks.