[Paper Review] Are Deep Policy Gradient Algorithms Truly Policy Gradient Algorithms

Andrew Ilyas, Logan Engstrom+5 more|arXiv (Cornell University)|Nov 6, 2018

Reinforcement Learning in Robotics1 references34 citations

TL;DR

This paper investigates whether deep policy gradient algorithms truly follow the theoretical framework they are based on. Through a fine-grained analysis of gradient estimation, value prediction, and optimization landscapes, it reveals significant mismatches: the surrogate objective diverges from the reward landscape, value estimators fail to fit the true value function, and gradient estimates poorly correlate with the true gradient—indicating a fundamental gap between theory and practice in current deep reinforcement learning methods.

ABSTRACT

We study how the behavior of deep policy gradient algorithms reflects the conceptual framework motivating their development. To this end, we propose a fine-grained analysis of state-of-the-art methods based on key elements of this framework: gradient estimation, value prediction, and optimization landscapes. Our results show that the behavior of deep policy gradient algorithms often deviates from what their motivating framework would predict: the surrogate objective does not match the reward landscape, learned value estimators fail to fit the value function, and gradient estimates poorly correlate with the true gradient. The mismatch between predicted and empirical behavior we uncover highlights our poor understanding of current methods, and indicates the need to move beyond current benchmark-centric evaluation methods.

Motivation & Objective

• To assess whether deep policy gradient algorithms behave as predicted by their theoretical framework.
• To identify discrepancies between the conceptual motivations of policy gradient methods and their empirical behavior in practice.
• To challenge current benchmark-centric evaluation methods that may obscure fundamental flaws in algorithmic design.
• To provide a fine-grained analysis of key components: gradient estimation, value prediction, and optimization landscapes in state-of-the-art deep policy gradient algorithms.

Proposed method

• The authors conduct a detailed empirical analysis of state-of-the-art deep policy gradient algorithms using a decomposition into core framework components: gradient estimation, value prediction, and optimization landscapes.
• They evaluate how well the surrogate objective aligns with the actual reward landscape across different environments.
• They assess the fidelity of learned value estimators by measuring their fit to the true value function.
• They compute correlation metrics between estimated gradients and the true policy gradient to evaluate gradient estimation quality.
• The analysis is applied across a range of continuous control benchmarks to ensure generalizability of findings.
• The study uses quantitative metrics to compare theoretical predictions with empirical observations, highlighting systemic deviations.

Experimental results

Research questions

• RQ1 To what extent does the surrogate objective in deep policy gradient methods reflect the true reward landscape?
• RQ2 How accurately do learned value estimators approximate the true value function in practice?
• RQ3 How well do gradient estimates correlate with the true policy gradient in deep policy gradient algorithms?
• RQ4 Why do current benchmark-centric evaluations fail to detect fundamental misalignments in algorithmic behavior?
• RQ5 What are the implications of these mismatches for the theoretical understanding and design of deep reinforcement learning algorithms?

Key findings

• The surrogate objective used in deep policy gradient algorithms often fails to align with the actual shape of the reward landscape, indicating a mismatch in optimization goals.
• Learned value estimators in state-of-the-art methods do not reliably fit the true value function, undermining their role in reducing policy gradient variance.
• Gradient estimates in these algorithms show poor correlation with the true policy gradient, suggesting that optimization is not following the intended direction.
• The observed deviations are consistent across multiple environments, indicating systemic issues rather than isolated failures.
• These mismatches reveal a significant gap between theoretical assumptions and empirical behavior, challenging the validity of current evaluation paradigms.