[Paper Review] P3O: Policy-on Policy-off Policy Optimization

Rasool Fakoor, Pratik Chaudhari+1 more|arXiv (Cornell University)|May 5, 2019

Reinforcement Learning in Robotics44 references4 citations

TL;DR

P3O is a novel reinforcement learning algorithm that combines on-policy and off-policy updates by using the effective sample size (ESS) to automatically balance the two, eliminating the need for manual hyperparameter tuning. It achieves state-of-the-art sample efficiency and performance on Atari-2600 and MuJoCo benchmarks with fixed hyperparameters across environments.

ABSTRACT

On-policy reinforcement learning (RL) algorithms have high sample complexity while off-policy algorithms are difficult to tune. Merging the two holds the promise to develop efficient algorithms that generalize across diverse environments. It is however challenging in practice to find suitable hyper-parameters that govern this trade off. This paper develops a simple algorithm named P3O that interleaves off-policy updates with on-policy updates. P3O uses the effective sample size between the behavior policy and the target policy to control how far they can be from each other and does not introduce any additional hyper-parameters. Extensive experiments on the Atari-2600 and MuJoCo benchmark suites show that this simple technique is effective in reducing the sample complexity of state-of-the-art algorithms. Code to reproduce experiments in this paper is at https://github.com/rasoolfa/P3O.

Motivation & Objective

• To address the trade-off between sample efficiency in off-policy RL and training stability in on-policy RL.
• To eliminate the need for manual hyperparameter tuning in combining on-policy and off-policy updates.
• To develop a unified, robust algorithm that generalizes across diverse environments without re-tuning.
• To use effective sample size (ESS) as a principled, automatic mechanism to control policy update behavior.

Proposed method

• P3O interleaves on-policy and off-policy policy gradient updates using a single objective function.
• It employs importance sampling with a clipping threshold c, where c is set to the normalized effective sample size (ESS).
• A KL divergence penalty is applied between the behavior policy β and target policy πθ, with λ = 1 - ESS to control policy deviation.
• The algorithm uses a replay buffer to store off-policy transitions and computes ESS to measure the quality of off-policy data.
• The ESS is used to dynamically adjust both the IS clipping threshold and the KL regularization coefficient, making the method fully automatic.
• The method avoids additional hyperparameters by deriving c and λ directly from ESS, ensuring robustness across environments.

Experimental results

Research questions

• RQ1 Can effective sample size (ESS) be used to automatically balance on-policy and off-policy updates in reinforcement learning?
• RQ2 Does a hyperparameter-free method based on ESS outperform existing algorithms with manual hyperparameter tuning on standard benchmarks?
• RQ3 Can a unified algorithm achieve both sample efficiency and training stability across diverse RL environments?
• RQ4 How does ESS-based policy update control compare to traditional hyperparameter-based blending in terms of performance and robustness?

Key findings

• P3O achieves state-of-the-art performance on the Atari-2600 benchmark, outperforming A2C, PPO, Q-Prop, and IPG across all 26 games.
• On MuJoCo continuous-control tasks, P3O significantly outperforms all baselines, achieving 5052 average return on Half-Cheetah after 3M steps.
• P3O achieves 4727 average return on Ant, surpassing IPG (3943) and Q-Prop (3374), demonstrating strong performance on complex locomotion tasks.
• The algorithm maintains consistent performance across all MuJoCo environments with fixed hyperparameters, indicating strong generalization.
• P3O reduces sample complexity compared to on-policy methods while avoiding the instability of standard off-policy methods.
• The use of ESS as a dynamic control mechanism eliminates the need for manual tuning of clipping and regularization hyperparameters.