[Paper Review] Supervised Policy Update for Deep Reinforcement Learning

Quan Vuong, Yiming Zhang+1 more|arXiv (Cornell University)|May 29, 2018

Reinforcement Learning in Robotics3 citations

TL;DR

This paper introduces Supervised Policy Update (SPU), a sample-efficient deep reinforcement learning method that first optimizes a non-parameterized proximal policy using constrained optimization, then distills it into a parameterized policy via supervised regression. SPU outperforms TRPO in Mujoco tasks and PPO in Atari games, demonstrating superior sample efficiency and robustness across diverse environments with simpler implementation than TRPO.

ABSTRACT

We propose a new sample-efficient methodology, called Supervised Policy Update (SPU), for deep reinforcement learning. Starting with data generated by the current policy, SPU formulates and solves a constrained optimization problem in the non-parameterized proximal policy space. Using supervised regression, it then converts the optimal non-parameterized policy to a parameterized policy, from which it draws new samples. The methodology is general in that it applies to both discrete and continuous action spaces, and can handle a wide variety of proximity constraints for the non-parameterized optimization problem. We show how the Natural Policy Gradient and Trust Region Policy Optimization (NPG/TRPO) problems, and the Proximal Policy Optimization (PPO) problem can be addressed by this methodology. The SPU implementation is much simpler than TRPO. In terms of sample efficiency, our extensive experiments show SPU outperforms TRPO in Mujoco simulated robotic tasks and outperforms PPO in Atari video game tasks.

Motivation & Objective

• Address the critical challenge of sample efficiency in on-policy deep reinforcement learning, where environment interactions are costly.
• Develop a general framework applicable to both discrete and continuous action spaces that maintains policy stability through proximity constraints.
• Simplify the implementation of trust-region methods like TRPO while improving performance, avoiding complex second-order optimization.
• Enable high performance with minimal hyper-parameter sensitivity, enhancing practical usability in real-world RL applications.
• Separate policy optimization into two stages—non-parameterized optimal policy search followed by parameterization via supervised learning—offering a new paradigm in policy update design.

Proposed method

• Formulate a constrained optimization problem in the non-parameterized proximal policy space using a distance metric such as KL divergence to ensure policy updates remain close to the current policy.
• Solve the constrained optimization problem to find an optimal non-parameterized policy that maximizes expected return based on newly collected data.
• Use supervised regression to map the optimal non-parameterized policy to a parameterized policy, enabling efficient sampling and training.
• Introduce per-state acceptance and dynamic stopping mechanisms to improve training stability and convergence, based on state-wise KL divergence and gradient norms.
• Support multiple proximity constraints, including those used in NPG/TRPO and PPO, by generalizing the optimization framework.
• Retrain the parameterized policy using the distilled policy as a target, with adaptive learning rates and early stopping based on gradient magnitude and KL divergence.

Experimental results

Research questions

• RQ1 Can a two-stage policy update framework—optimizing a non-parameterized policy followed by supervised parameterization—achieve better sample efficiency than existing on-policy methods?
• RQ2 How does the performance of SPU compare to TRPO and PPO across diverse environments like Mujoco and Atari, especially in terms of sample efficiency and final performance?
• RQ3 To what extent is SPU’s performance robust to hyper-parameter choices, particularly in high-dimensional and diverse RL environments?
• RQ4 What components of the SPU algorithm (e.g., gradient-based KL regularization, per-state acceptance) are most critical for achieving high performance?
• RQ5 Can a simpler implementation than TRPO achieve superior performance by decoupling optimal policy search from parameterization?

Key findings

• SPU outperforms TRPO by 28% in final performance after 3 million timesteps on 10 Mujoco environments, demonstrating superior sample efficiency.
• SPU achieves a 55% improvement over PPO in average final performance across 60 Atari games, with higher performance gains in 15 environments compared to PPO’s 9.
• The component involving the gradient of the KL divergence ($\nabla_{\theta}D_{\text{KL}}$) contributes most significantly to performance, with removal reducing SPU’s gain by 85%.
• Per-state acceptance and dynamic stopping are both essential, with per-state acceptance having the largest impact—removal reduces performance by 67%.
• SPU’s performance is highly robust: in 100 random hyper-parameter trials, all SPU variants outperformed TRPO, with 75% exceeding TRPO by at least 18%.
• SPU achieves state-of-the-art performance in sample efficiency on Mujoco and Atari, despite being simpler than TRPO and outperforming PPO in the latter domain.