[Paper Review] Equivalence Between Policy Gradients and Soft Q-Learning

John Schulman, Xi Chen+1 more|arXiv (Cornell University)|Apr 21, 2017

Reinforcement Learning in Robotics11 references198 citations

TL;DR

The paper proves a precise equivalence between entropy-regularized Q-learning (soft Q-learning) and policy gradient methods, and links them to natural policy gradients, with experiments on Atari showing competitive performance.

ABSTRACT

Two of the leading approaches for model-free reinforcement learning are policy gradient methods and $Q$-learning methods. $Q$-learning methods can be effective and sample-efficient when they work, however, it is not well-understood why they work, since empirically, the $Q$-values they estimate are very inaccurate. A partial explanation may be that $Q$-learning methods are secretly implementing policy gradient updates: we show that there is a precise equivalence between $Q$-learning and policy gradient methods in the setting of entropy-regularized reinforcement learning, that "soft" (entropy-regularized) $Q$-learning is exactly equivalent to a policy gradient method. We also point out a connection between $Q$-learning methods and natural policy gradient methods. Experimentally, we explore the entropy-regularized versions of $Q$-learning and policy gradients, and we find them to perform as well as (or slightly better than) the standard variants on the Atari benchmark. We also show that the equivalence holds in practical settings by constructing a $Q$-learning method that closely matches the learning dynamics of A3C without using a target network or $ε$-greedy exploration schedule.

Motivation & Objective

• Motivate understanding of why Q-learning methods sometimes perform well despite inaccurate Q-values.
• Establish a precise gradient-level equivalence between soft Q-learning and policy gradient methods under entropy regularization.
• Bridge connections to natural policy gradients and actor-critic frameworks like A3C.
• Demonstrate practical validity by experiments on Atari and by constructing a Q-learning variant mirroring A3C dynamics without target networks or epsilon-greedy exploration.

Proposed method

• Formulate entropy-regularized RL with KL-divergence penalties and Boltzmann policies.
• Show that the gradient of n-step Q-learning loss matches the gradient of an n-step policy gradient loss plus a baseline/error term.
• Derive Boltzmann back-ups and fixed-policy backup operators under entropy regularization.
• Relate soft Q-learning to natural policy gradients via a least-squares interpretation of the gradient updates.
• Define soft Q-learning variants that optimize the entropy-augmented return using target networks and n-step backups.
• Provide practical connections to actor-critic methods and discuss incorporation of KL penalties vs. classical entropy bonuses.

Experimental results

Research questions

• RQ1 Can entropy-regularized soft Q-learning gradients be shown to be identical to policy gradient updates under appropriate conditions?
• RQ2 How does entropy regularization influence the relationship between Q-learning and policy gradients, and can this explain empirical performance similarities?
• RQ3 What is the connection between soft Q-learning and natural policy gradient methods?
• RQ4 Do entropy-regularized methods perform competitively on standard benchmarks like Atari, and can Q-learning be made to mimic A3C dynamics?
• RQ5 Can practical Q-learning variants replicate actor-critic learning dynamics without target networks or fixed exploration schedules?

Key findings

• There is a precise equivalence between soft Q-learning gradients and policy gradient gradients under entropy regularization.
• The soft Q-learning gradient decomposes into a policy-gradient term plus a baseline-error-gradient term akin to actor-critic methods like A3C.
• A connection is drawn between Q-learning with replay/batch updates and natural policy gradient methods.
• Entropy-regularized versions of Q-learning and policy gradients perform as well as or slightly better than standard variants on Atari in experiments.
• The equivalence holds in practice by constructing a Q-learning method that closely matches A3C learning dynamics without a target network or epsilon-greedy exploration.