[Paper Review] Behavior Proximal Policy Optimization

Zifeng Zhuang, Kun Lei+3 more|arXiv (Cornell University)|Feb 22, 2023

Reinforcement Learning in Robotics8 citations

TL;DR

BPPO shows that offline reinforcement learning can be solved by a simple on-policy like PPO-based method without extra constraints, achieving strong results on D4RL by monotonically improving the behavior policy through offline data.

ABSTRACT

Offline reinforcement learning (RL) is a challenging setting where existing off-policy actor-critic methods perform poorly due to the overestimation of out-of-distribution state-action pairs. Thus, various additional augmentations are proposed to keep the learned policy close to the offline dataset (or the behavior policy). In this work, starting from the analysis of offline monotonic policy improvement, we get a surprising finding that some online on-policy algorithms are naturally able to solve offline RL. Specifically, the inherent conservatism of these on-policy algorithms is exactly what the offline RL method needs to overcome the overestimation. Based on this, we propose Behavior Proximal Policy Optimization (BPPO), which solves offline RL without any extra constraint or regularization introduced compared to PPO. Extensive experiments on the D4RL benchmark indicate this extremely succinct method outperforms state-of-the-art offline RL algorithms. Our implementation is available at https://github.com/Dragon-Zhuang/BPPO.

Motivation & Objective

• Motivate offline RL as monotonic improvement of the behavior policy using offline data.
• Show that online on-policy algorithms (like PPO) can naturally solve offline RL without extra constraints.
• Propose BPPO, a simple offline algorithm that mirrors PPO while relying on offline data.
• Demonstrate strong empirical performance on D4RL benchmarks across Gym, Adroit, Kitchen, and Antmaze.

Proposed method

• Formulate offline monotonic policy improvement using the Performance Difference Theorem.
• Derive a practical BPPO objective that mirrors PPO but replaces online state distributions with offline dataset distributions.
• Impose a divergence constraint between the updated policy and the current policy to ensure monotonic improvement, implemented via a clipped surrogate loss.
• Use importance sampling to reweight the offline-data-based advantage with the current policy.
• Approximate and compute the advantage A_pi_k using off-policy Q and V estimates tied to the behavior policy.
• Incorporate clip ratio decay to keep the learned policy tied to the behavior policy while allowing controlled updates.

Experimental results

Research questions

• RQ1 Can online on-policy algorithms achieve monotonic improvement in offline RL without explicit regularization?
• RQ2 Does a PPO-like BPPO approach yield superior or competitive performance on standard offline RL benchmarks?
• RQ3 How does BPPO compare to one-step and iterative/off-policy offline methods in practice?
• RQ4 What implementation choices (advantage estimation, clip scheduling) influence BPPO’s effectiveness in offline settings.

Key findings

• BPPO achieves competitive or superior performance compared with state-of-the-art offline RL methods on D4RL benchmarks.
• BPPO substantially improves over Behavior Cloning baselines and shows strong results on Adroit and Kitchen tasks.
• Empirical results indicate BPPO often outperforms Onestep RL and is competitive with or better than iterative/off-policy methods on several tasks.
• Introducing monotonic improvements via a PPO-like loss in offline data yields strong performance without additional regularization terms beyond those in PPO.
• Clip ratio decay and careful advantage estimation are important for stable BPPO performance.
• BPPO demonstrates strong performance on sparse-reward tasks such as Antmaze, outperforming several baselines.