[Paper Review] Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning

Xue Bin Peng, Aviral Kumar+2 more|arXiv (Cornell University)|Oct 1, 2019

Reinforcement Learning in Robotics29 references165 citations

TL;DR

AWR is a simple off-policy RL algorithm that uses two supervised learning steps (value regression and advantage-weighted policy regression) with experience replay, achieving competitive results on OpenAI Gym and complex motion imitation tasks.

ABSTRACT

In this paper, we aim to develop a simple and scalable reinforcement learning algorithm that uses standard supervised learning methods as subroutines. Our goal is an algorithm that utilizes only simple and convergent maximum likelihood loss functions, while also being able to leverage off-policy data. Our proposed approach, which we refer to as advantage-weighted regression (AWR), consists of two standard supervised learning steps: one to regress onto target values for a value function, and another to regress onto weighted target actions for the policy. The method is simple and general, can accommodate continuous and discrete actions, and can be implemented in just a few lines of code on top of standard supervised learning methods. We provide a theoretical motivation for AWR and analyze its properties when incorporating off-policy data from experience replay. We evaluate AWR on a suite of standard OpenAI Gym benchmark tasks, and show that it achieves competitive performance compared to a number of well-established state-of-the-art RL algorithms. AWR is also able to acquire more effective policies than most off-policy algorithms when learning from purely static datasets with no additional environmental interactions. Furthermore, we demonstrate our algorithm on challenging continuous control tasks with highly complex simulated characters.

Motivation & Objective

• Develop a simple, scalable off-policy RL algorithm based on supervised learning losses.
• Enable learning from off-policy data via experience replay with a stable, bounded update.
• Demonstrate competitive performance against established on-policy and off-policy methods on standard benchmarks and motion imitation tasks.

Proposed method

• Two-stage supervised learning update: fit a value function by regression on returns, then fit the policy via weighted regression using exp(A/β) as weights.
• Advantage A(s,a) = R(s,a) - V(s) guides the policy update.
• Derivation treats AWR as a constrained policy search optimizing expected improvement with a KL-divergence constraint.
• Extend to off-policy data by modeling the sampling policy as a mixture of past policies via a replay buffer.
• Use TD(λ) for low-variance return estimates and clip weights to stabilize training.
• Optional baseline V̄(s) computed as a weighted average of past value functions for multiple policies.

Experimental results

Research questions

• RQ1 Can a simple, regression-based off-policy RL algorithm achieve competitive performance with minimal complexity?
• RQ2 How does incorporating a baseline and experience replay affect stability and sample efficiency?
• RQ3 What is the impact of replay buffer size and off-policy data on learning quality?
• RQ4 Can AWR scale to high-dimensional continuous control and motion imitation tasks?

Key findings

• AWR achieves competitive results compared to popular on-policy and off-policy methods on OpenAI Gym benchmarks.
• AWR significantly outperforms purely on-policy methods like PPO and TRPO in both sample efficiency and asymptotic performance.
• AWR attains similar asymptotic performance to SAC and TD3 on many tasks, despite using simple supervised regression for both value and policy updates.
• The baseline V(s) and experience replay are crucial; removing them degrades performance, and larger replay buffers improve stability and final performance.
• AWR effectively handles fully off-policy learning from static datasets in motion imitation tasks, matching or surpassing RWR and PPO under certain conditions.
• On challenging Humanoid-V2, AWR still lags behind SAC, indicating room for improvement on very difficult tasks.