[Paper Review] AWAC: Accelerating Online Reinforcement Learning with Offline Datasets

Ashvin Nair, Abhishek Gupta+2 more|arXiv (Cornell University)|Jun 16, 2020

Reinforcement Learning in RoboticsComputer Science60 references71 citations

TL;DR

AWAC is an off-policy actor-critic algorithm that learns from offline datasets and efficiently fine-tunes online by using an implicit constraint on the actor, enabling rapid skill acquisition from demonstrations or suboptimal data.

ABSTRACT

Reinforcement learning (RL) provides an appealing formalism for learning control policies from experience. However, the classic active formulation of RL necessitates a lengthy active exploration process for each behavior, making it difficult to apply in real-world settings such as robotic control. If we can instead allow RL algorithms to effectively use previously collected data to aid the online learning process, such applications could be made substantially more practical: the prior data would provide a starting point that mitigates challenges due to exploration and sample complexity, while the online training enables the agent to perfect the desired skill. Such prior data could either constitute expert demonstrations or sub-optimal prior data that illustrates potentially useful transitions. While a number of prior methods have either used optimal demonstrations to bootstrap RL, or have used sub-optimal data to train purely offline, it remains exceptionally difficult to train a policy with offline data and actually continue to improve it further with online RL. In this paper we analyze why this problem is so challenging, and propose an algorithm that combines sample efficient dynamic programming with maximum likelihood policy updates, providing a simple and effective framework that is able to leverage large amounts of offline data and then quickly perform online fine-tuning of RL policies. We show that our method, advantage weighted actor critic (AWAC), enables rapid learning of skills with a combination of prior demonstration data and online experience. We demonstrate these benefits on simulated and real-world robotics domains, including dexterous manipulation with a real multi-fingered hand, drawer opening with a robotic arm, and rotating a valve. Our results show that incorporating prior data can reduce the time required to learn a range of robotic skills to practical time-scales.

Motivation & Objective

• Motivate practical RL by efficiently leveraging large offline datasets to pre-train policies for real-world robotics.
• Develop a simple, data-efficient algorithm that combines offline pre-training with online fine-tuning without requiring explicit behavior policy modeling.
• Demonstrate that incorporating prior data reduces online training time across diverse robotic tasks.
• Assess robustness to suboptimal offline data and demonstrate real-world applicability.

Proposed method

• Off-policy critic learning via TD bootstrapping to estimate Q^π( s, a ).
• Policy improvement via maximizing A^π_k(s,a) under a KL-like implicit constraint without explicit behavior model.
• Derivation of a closed-form non-parametric actor solution π*(a|s) ∝ π_β(a|s) exp(A^π_k(s,a)/λ) and projection to a parametric policy using forward KL minimization.
• Parameterization of actor and critic with neural networks and updates via supervised-like weighted maximum likelihood (Eq. 13) with advantages from the learned critic.
• Use of a replay buffer containing offline data β and online data; online data sparse introduction after offline steps.
• Comparison to AWR and ABM/MPO-like methods to show benefits of TD bootstrapping and no explicit behavior model.

Experimental results

Research questions

• RQ1 Can AWAC effectively combine offline pretraining with online fine-tuning to learn complex robotic control tasks?
• RQ2 How does AWAC perform with suboptimal or random offline data compared to demonstrations?
• RQ3 Does avoiding explicit behavior modeling improve online fine-tuning efficiency and stability?
• RQ4 How does AWAC compare to prior offline and online RL methods on high-dimensional, sparse-reward robotics tasks?

Key findings

• AWAC enables rapid learning from offline data and online fine-tuning on diverse robotics tasks, including dexterous manipulation and real-world experiments.
• AWAC outperforms purely offline or purely online baselines in fine-tuning efficiency, solving challenging tasks with limited online data (e.g., 120K timesteps for pen task).
• The method can utilize demonstrations, suboptimal data, or random exploration data without algorithmic changes and still achieve pruning online data needs.
• Avoiding explicit behavior policy modeling makes AWAC less conservative and more effective at online refinement compared to prior offline RL approaches.
• TD bootstrapping for the critic and the implicit constraint on the actor are key design choices that yield better performance than variants lacking these features.