[Paper Review] AlgaeDICE: Policy Gradient from Arbitrary Experience

Ofir Nachum, Bo Dai+4 more|arXiv (Cornell University)|Dec 4, 2019

Reinforcement Learning in Robotics52 references82 citations

TL;DR

AlgaeDICE proposes a off-policy policy gradient method that uses density regularization and a dual function to recover on-policy gradients from arbitrary off-policy data without importance weighting.

ABSTRACT

In many real-world applications of reinforcement learning (RL), interactions with the environment are limited due to cost or feasibility. This presents a challenge to traditional RL algorithms since the max-return objective involves an expectation over on-policy samples. We introduce a new formulation of max-return optimization that allows the problem to be re-expressed by an expectation over an arbitrary behavior-agnostic and off-policy data distribution. We first derive this result by considering a regularized version of the dual max-return objective before extending our findings to unregularized objectives through the use of a Lagrangian formulation of the linear programming characterization of Q-values. We show that, if auxiliary dual variables of the objective are optimized, then the gradient of the off-policy objective is exactly the on-policy policy gradient, without any use of importance weighting. In addition to revealing the appealing theoretical properties of this approach, we also show that it delivers good practical performance.

Motivation & Objective

• Motivate learning from costly or limited environment interactions by enabling off-policy policy optimization.
• Reformulate max-return optimization as an off-policy problem using density regularization over state-action occupancies.
• Derive a saddle-point objective linking a policy (actor) with a dual function (critic) that can be optimized from arbitrary data.
• Show that optimizing the dual yields the on-policy policy gradient with a regularized reward.
• Provide theoretical guarantees and empirical validation of the approach.

Proposed method

• Start from the dual formulation of the max-return objective expressed in terms of normalized state-action occupancies.
• Introduce a regularizer using an f-divergence between the on-policy and off-policy occupancies to enable off-policy data usage.
• Apply a change of variables to obtain a purely off-policy objective J_{D,f}(π,ν) that is optimized over the policy and a dual function ν.
• Use a variational form of the f-divergence and a dual embedding to handle the double-sampling problem.
• Demonstrate that, when the dual ν is optimized, the gradient w.r.t. policy parameters matches the on-policy policy gradient with a modified reward tilde{r}(s,a) = r(s,a) - α f'(w_{π/ D}(s,a)).
• Discuss the Lagrangian/LP perspective that yields a single unified objective for policy and value learning and enables behavior-agnostic off-policy optimization.

Experimental results

Research questions

• RQ1 Can we express max-return optimization as an off-policy problem without importance weighting?
• RQ2 Does dual optimization yield the on-policy policy gradient when learning from arbitrary off-policy data?
• RQ3 How can f-divergence regularization and dual embeddings enable stable off-policy policy optimization?
• RQ4 What are the theoretical guarantees and practical implications for policy learning using AlgaeDICE under off-policy data?

Key findings

• The off-policy objective with a dual function reproduces the on-policy policy gradient when the dual is optimized.
• The regularized dual formulation yields a unified objective that trains policy and critic from off-policy data without importance weights.
• Choosing a quadratic f leads to an actor-critic-like objective, but with a principled behavior-agnostic off-policy basis.
• The Lagrangian/LP view provides strong duality and allows recovering a Fenchel AlgaeDICE objective that matches the regularized max-return objective.
• Empirical results show AlgaeDICE can perform well on benchmark tasks, including offline Four Rooms and continuous control suites.