[Paper Review] V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control

Hao Song, Abbas Abdolmaleki+12 more|arXiv (Cornell University)|Sep 26, 2019

Reinforcement Learning in Robotics33 references39 citations

TL;DR

V-MPO is an on-policy adaptation of MPO that uses a learned state-value function to perform policy iteration, achieving strong results in both discrete and continuous control without entropy regularization or population-based tuning.

ABSTRACT

Some of the most successful applications of deep reinforcement learning to challenging domains in discrete and continuous control have used policy gradient methods in the on-policy setting. However, policy gradients can suffer from large variance that may limit performance, and in practice require carefully tuned entropy regularization to prevent policy collapse. As an alternative to policy gradient algorithms, we introduce V-MPO, an on-policy adaptation of Maximum a Posteriori Policy Optimization (MPO) that performs policy iteration based on a learned state-value function. We show that V-MPO surpasses previously reported scores for both the Atari-57 and DMLab-30 benchmark suites in the multi-task setting, and does so reliably without importance weighting, entropy regularization, or population-based tuning of hyperparameters. On individual DMLab and Atari levels, the proposed algorithm can achieve scores that are substantially higher than has previously been reported. V-MPO is also applicable to problems with high-dimensional, continuous action spaces, which we demonstrate in the context of learning to control simulated humanoids with 22 degrees of freedom from full state observations and 56 degrees of freedom from pixel observations, as well as example OpenAI Gym tasks where V-MPO achieves substantially higher asymptotic scores than previously reported.

Motivation & Objective

• Motivate reducing variance and instability associated with policy gradient methods in on-policy reinforcement learning.
• Develop an on-policy MPO-based algorithm that leverages a learned state-value function for policy iteration.
• Demonstrate strong performance across discrete and continuous control benchmarks without extra regularization or population-based tuning.

Proposed method

• Propose V-MPO, an on-policy adaptation of Maximum a Posteriori Policy Optimization.
• Use policy iteration guided by a learned state-value function.
• Avoid entropy regularization and importance weighting while maintaining stable learning.
• Show that the method works across discrete and continuous action spaces, including high-dimensional tasks.

Experimental results

Research questions

• RQ1 How does V-MPO perform on discrete and continuous control benchmarks relative to previous on-policy methods?
• RQ2 Can V-MPO achieve strong performance without entropy regularization, importance weighting, or population-based hyperparameter tuning?
• RQ3 How well does V-MPO scale to high-dimensional action spaces and pixel-based observations?
• RQ4 What are the empirical gains on Atari-57, DMLab-30, and OpenAI Gym tasks in both multi-task and single-task settings?

Key findings

• V-MPO surpasses previously reported scores on Atari-57 and DMLab-30 in the multi-task setting.
• The method achieves these results without importance weighting, entropy regularization, or population-based hyperparameter tuning.
• On individual DMLab and Atari levels, the scores are substantially higher than previously reported.
• V-MPO is applicable to high-dimensional continuous action spaces, demonstrated with humanoids from full state observations and pixel observations.
• OpenAI Gym tasks show substantially higher asymptotic scores than previously reported.