[Paper Review] A short variational proof of equivalence between policy gradients and soft Q learning

Pierre H. Richemond, Brendan Maginnis|arXiv (Cornell University)|Dec 22, 2017

Reinforcement Learning in Robotics12 references3 citations

TL;DR

This paper presents a concise variational proof of the equivalence between soft Q-learning and softmax policy gradients using convex duality, specifically the Gibbs/Donsker-Varadhan formula. It establishes that entropic regularization and softmax relaxation are dual perspectives, enabling a streamlined derivation and introducing a new policy inequality linking the optimality gap to KL divergence from the optimal policy.

ABSTRACT

Two main families of reinforcement learning algorithms, Q-learning and policy gradients, have recently been proven to be equivalent when using a softmax relaxation on one part, and an entropic regularization on the other. We relate this result to the well-known convex duality of Shannon entropy and the softmax function. Such a result is also known as the Donsker-Varadhan formula. This provides a short proof of the equivalence. We then interpret this duality further, and use ideas of convex analysis to prove a new policy inequality relative to soft Q-learning.

Motivation & Objective

• To provide a shorter, more insightful proof of the equivalence between soft Q-learning and softmax policy gradients using convex duality.
• To clarify the fundamental role of entropy regularization and softmax relaxation in unifying these two RL paradigms.
• To derive a new policy inequality that bounds the expected optimality gap in terms of KL divergence from the optimal policy.
• To connect the theoretical equivalence to established results in convex analysis and large deviations theory.
• To lay groundwork for future work on the duality between convex optimization algorithms and reinforcement learning methods.

Proposed method

• Uses the Gibbs variational principle to represent the free energy functional as a convex conjugate of the entropy functional.
• Applies the Donsker-Varadhan formula to express the optimal policy value as a log-partition function of the reward under a Gibbs measure.
• Employs Legendre-Fenchel duality to derive the equivalence between soft Q-learning and policy gradients in the one-step bandit setting.
• Introduces a transportation inequality that relates the expected reward gap to the KL divergence between policies.
• Derives a general inequality using the inverse of the Legendre transform of a convex function, linking reward difference to KL divergence.
• Applies the duality framework to parameterized softmax policies, showing that KL divergence from the optimal policy is proportional to the difference between expected Q-values and rewards.

Experimental results

Research questions

• RQ1 How can convex duality be used to provide a shorter and more insightful proof of the equivalence between soft Q-learning and policy gradients?
• RQ2 What is the precise mathematical relationship between entropic regularization and softmax relaxation in the context of reinforcement learning?
• RQ3 Can a new policy inequality be derived that bounds the optimality gap using the KL divergence from the optimal policy?
• RQ4 How does the Donsker-Varadhan variational formula unify the treatment of entropy and expectation in RL?
• RQ5 What are the implications of this duality for designing variance-reduction techniques or new optimization algorithms in RL?

Key findings

• The equivalence between soft Q-learning and softmax policy gradients is rigorously established via the Gibbs variational principle and convex duality, with a significantly shorter proof than prior work.
• The Donsker-Varadhan formula provides a variational representation of the free energy functional, enabling a direct derivation of the equivalence in the bandit setting.
• A new policy inequality is derived: the expected reward gap between any policy and the optimal policy is bounded by the inverse Legendre transform of a convex function applied to the KL divergence from the optimal policy.
• The KL divergence between the optimal policy and a parameterized policy is shown to be proportional to the difference between the expected Q-value and the expected reward under that policy.
• The framework allows for the use of different softmax temperatures in practice, enabling flexible estimation of the bound via empirical moment generating functions.
• The results suggest a deep connection between convex optimization, large deviations, and reinforcement learning, opening avenues for improved sample efficiency and variance reduction in Monte Carlo estimators.