[Paper Review] Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO

Logan Engstrom, Andrew Ilyas+5 more|arXiv (Cornell University)|May 25, 2020

Reinforcement Learning in RoboticsComputer Science16 references137 citations

TL;DR

The paper performs an in-depth ablation study showing that code-level optimizations in PPO—not just its clipping mechanism—drive much of its performance over TRPO and can fundamentally alter how trust regions operate in policy gradient methods.

ABSTRACT

We study the roots of algorithmic progress in deep policy gradient algorithms through a case study on two popular algorithms: Proximal Policy Optimization (PPO) and Trust Region Policy Optimization (TRPO). Specifically, we investigate the consequences of "code-level optimizations:" algorithm augmentations found only in implementations or described as auxiliary details to the core algorithm. Seemingly of secondary importance, such optimizations turn out to have a major impact on agent behavior. Our results show that they (a) are responsible for most of PPO's gain in cumulative reward over TRPO, and (b) fundamentally change how RL methods function. These insights show the difficulty and importance of attributing performance gains in deep reinforcement learning. Code for reproducing our results is available at https://github.com/MadryLab/implementation-matters .

Motivation & Objective

• Assess how individual components of deep policy gradient methods influence agent training and performance.
• Identify which code-level optimizations in PPO are responsible for performance gains over TRPO.
• Promote modular design by understanding each component's impact on final rewards and training dynamics.

Proposed method

• Compare TRPO and PPO implementations with and without code-level optimizations through ablation studies.
• Introduce PPO-M (PPO without code-level optimizations) and TRPO+ (TRPO with PPO-like optimizations) to isolate effects.
• Evaluate on MuJoCo tasks (e.g., Walker2d-v2, Hopper-v2, Humanoid-v2) with multiple random seeds and hyperparameter grids.
• Measure final rewards, per-step metrics, and trust-region indicators (KL divergence and ratio constraints).
• Define AAI and ACLI metrics to quantify the relative impact of algorithm choice versus code-level optimizations.

Experimental results

Research questions

• RQ1 How do code-level optimizations in PPO influence final performance compared to TRPO?
• RQ2 Do code-level optimizations alter the trust-region behavior of PPO beyond what the clipping mechanism suggests?
• RQ3 What is the relative contribution of core algorithm choice versus code-level optimizations to agent performance across tasks?

Key findings

• Code-level optimizations account for a substantial portion of PPO’s performance gains over TRPO.
• PPO and PPO-M differ mainly in how optimizations shape the trust region, not solely in the clipping mechanism.
• Code-level optimizations dramatically affect reward landscapes across hyperparameters, sometimes more than changing between PPO and TRPO.
• PPO-NoClip can match PPO performance when paired with sufficient code-level optimizations, indicating clipping is not always essential.
• TRPO+ and PPO-M reveal that optimizations can outweigh core algorithm choices in most tasks.