[Paper Review] Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic

Shixiang Gu, Timothy Lillicrap+3 more|arXiv (Cornell University)|Nov 7, 2016

Reinforcement Learning in Robotics32 references100 citations

TL;DR

Q-Prop combines on-policy Monte Carlo policy gradients with an off-policy critic as a control variate to improve sample efficiency and stability over TRPO and DDPG in continuous control tasks.

ABSTRACT

Model-free deep reinforcement learning (RL) methods have been successful in a wide variety of simulated domains. However, a major obstacle facing deep RL in the real world is their high sample complexity. Batch policy gradient methods offer stable learning, but at the cost of high variance, which often requires large batches. TD-style methods, such as off-policy actor-critic and Q-learning, are more sample-efficient but biased, and often require costly hyperparameter sweeps to stabilize. In this work, we aim to develop methods that combine the stability of policy gradients with the efficiency of off-policy RL. We present Q-Prop, a policy gradient method that uses a Taylor expansion of the off-policy critic as a control variate. Q-Prop is both sample efficient and stable, and effectively combines the benefits of on-policy and off-policy methods. We analyze the connection between Q-Prop and existing model-free algorithms, and use control variate theory to derive two variants of Q-Prop with conservative and aggressive adaptation. We show that conservative Q-Prop provides substantial gains in sample efficiency over trust region policy optimization (TRPO) with generalized advantage estimation (GAE), and improves stability over deep deterministic policy gradient (DDPG), the state-of-the-art on-policy and off-policy methods, on OpenAI Gym's MuJoCo continuous control environments.

Motivation & Objective

• Motivate reducing sample complexity and improving stability in deep reinforcement learning.
• Develop a policy gradient method that leverages an off-policy critic without introducing bias.
• Bridge on-policy Monte Carlo gradients with off-policy critic updates to improve data efficiency.

Proposed method

• Derives Q-Prop as a policy gradient estimator using the first-order Taylor expansion of an off-policy critic as a control variate.
• Uses Qw as the critic and μθ(st) as the expected action to form a gradient with an analytic term through the critic plus a residual Monte Carlo term.
• Expresses the gradient with advantages and shows how off-policy data can train the critic while the actor remains on-policy.
• Introduces adaptive control variate weighting η(st) to reduce variance, including fully adaptive, conservative, and aggressive variants.
• Demonstrates that Q-Prop can be combined with TRPO, GAE, DDPG, and Retrace(λ) and can be implemented on top of existing policy gradient frameworks.

Experimental results

Research questions

• RQ1 Can an off-policy critic be used as a control variate to reduce variance in policy gradient estimates without introducing bias?
• RQ2 How does Q-Prop perform in terms of sample efficiency and stability compared to TRPO-GAE and DDPG across continuous control tasks?
• RQ3 What is the impact of adaptive control variate weighting on variance and learning performance?
• RQ4 Can Q-Prop integrate with existing on-policy and off-policy RL techniques to improve practical performance?

Key findings

• Q-Prop provides substantial gains in sample efficiency over TRPO-GAE across continuous control tasks.
• Conservative Q-Prop offers improved stability and often outperforms other baselines like TRPO and DDPG in several domains.
• Adaptive Q-Prop variants (fully adaptive, conservative, aggressive) can reduce gradient variance and maintain or improve learning performance.
• On challenging domains like Humanoid-v1, Q-Prop substantially improves sample efficiency over TRPO, with DDPG sometimes failing to find good solutions.
• Q-Prop can learn effectively with small batch sizes, and its stability makes it favorable for real-world tasks.