[Paper Review] Supervised Policy Update.

Quan Vuong, Yiming Zhang+1 more|arXiv (Cornell University)|May 29, 2018

Reinforcement Learning in Robotics12 references3 citations

TL;DR

This paper introduces Supervised Policy Update (SPU), a sample-efficient deep reinforcement learning method that first optimizes a non-parameterized policy in a proximal space, then uses supervised regression to distill it into a parameterized policy for sample generation. SPU achieves better sample efficiency than PPO on robotic locomotion tasks by unifying TRPO and PPO under a single optimization framework.

ABSTRACT

We propose a new sample-efficient methodology, called Supervised Policy Update (SPU), for deep reinforcement learning. Starting with data generated by the current policy, SPU optimizes over the proximal policy space to find a non-parameterized policy. It then solves a supervised regression problem to convert the non-parameterized policy to a parameterized policy, from which it draws new samples. There is significant flexibility in setting the labels in the supervised regression problem, with different settings corresponding to different underlying optimization problems. We develop a methodology for finding an optimal policy in the non-parameterized policy space, and show how Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) can be addressed by this methodology. In terms of sample efficiency, our experiments show SPU can outperform PPO for simulated robotic locomotion tasks.

Motivation & Objective

• To address the challenge of sample inefficiency in deep reinforcement learning, especially in high-dimensional continuous control tasks.
• To unify and generalize existing proximal policy optimization methods like TRPO and PPO under a single, flexible framework.
• To improve policy learning efficiency by decoupling policy optimization from parameterization through a supervised regression step.
• To enable the use of non-parameterized policies for stable optimization before distillation into parameterized policies for deployment.

Proposed method

• SPU begins by generating off-policy rollouts using the current policy, forming a dataset of state-action pairs.
• It then optimizes a non-parameterized policy within a proximal neighborhood of the current policy to ensure stability and improvement.
• A supervised regression problem is formulated to map states to actions, where labels are derived from the optimized non-parameterized policy.
• The regression model is trained to produce a parameterized policy that mimics the behavior of the non-parameterized policy.
• The method allows flexible label design, enabling it to recover or generalize TRPO and PPO as special cases.
• New rollouts are collected from the updated parameterized policy, closing the loop for iterative improvement.

Experimental results

Research questions

• RQ1 Can a two-stage approach combining non-parameterized policy optimization and supervised distillation improve sample efficiency in deep reinforcement learning?
• RQ2 How can TRPO and PPO be unified under a single optimization framework using a proximal policy update and supervised regression?
• RQ3 What label configurations in the supervised regression step lead to optimal policy performance and stability?
• RQ4 Does the proposed method outperform standard PPO in terms of sample efficiency on continuous control benchmarks?

Key findings

• SPU achieves superior sample efficiency compared to PPO on simulated robotic locomotion tasks, demonstrating faster learning with fewer environment interactions.
• The method successfully generalizes both TRPO and PPO by adjusting the label design in the supervised regression step.
• Optimizing over a non-parameterized policy space enables stable and effective policy updates without relying on neural network parameterization during optimization.
• The distillation step from non-parameterized to parameterized policy maintains performance while enabling efficient rollouts for training.